I'm printing out all the headers and I get:
map[Cookie:[_ga=GA1.2.843429125.1462575405] User-Agent:[Mozilla/5.0
(Macintosh; Intel Mac OS X 10_11_3) AppleWebKit/601.4.4 (KHTML, like Gecko)
Version/9.0.3 Safari/601.4.4] Accept-Language:[en-us]
Accept-Encoding:[gzip, deflate] Connection:[keep-alive]
Accept:[text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8]]
which means my browser is sending "Cookie", "User-Agent", "Accept-Language", "Accept-Encoding", "Connection", and "Accept" but there is no "Host" value.
How can I get my https://en.wikipedia.org/wiki/Virtual_hosting working without this value?
I'm using https://github.com/gin-gonic/gin
It stated on Golang http docs :
For incoming requests, the Host header is promoted to the Request.Host
field and removed from the Header map.
So you can get the host by access
http.Request.Host
Check here for details : https://golang.org/pkg/net/http/
Related
I deployed my express app on Firebase Functions with cors setting. However, on the browser, the below error printed on the console.
Access to fetch at 'https://....cloudfunctions.net/users/' from origin 'http://localhost:3000' has been blocked by CORS policy: No 'Access-Control-Allow-Origin' header is present on the requested resource. If an opaque response serves your needs, set the request's mode to 'no-cors' to fetch the resource with CORS disabled.
On my code,
...
import * as cors from 'cors'
const app = express()
app.use(cors({ origin: true })) // Set CORS properly
app.use(helmet())
app.use(express.json())
app.use(express.urlencoded({ extended: true }))
app.use('/', router)
app.use(errorHandler)
export const users = functions
.region('asia-northeast3')
.https.onRequest(app)
I cannot find what is the problem. Even the official document recommends app.use(cors({ origin: true})). How can I solve this problem?
====EDIT====
My request looks like:
GET /baggle-318109/us-central1/auth/kakao HTTP/1.1
Host: localhost:5001
Connection: keep-alive
sec-ch-ua: "Chromium";v="92", " Not A;Brand";v="99", "Google Chrome";v="92"
sec-ch-ua-mobile: ?0
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/92.0.4515.131 Safari/537.36
Accept: */*
Origin: http://localhost:3000
Sec-Fetch-Site: same-site
Sec-Fetch-Mode: cors
Sec-Fetch-Dest: empty
Referer: http://localhost:3000/
Accept-Encoding: gzip, deflate, br
Accept-Language: ko-KR,ko;q=0.9,en-US;q=0.8,en;q=0.7
If-None-Match: W/"d-vfAVvdILIkmIo67j2sc7s17+9M0"
You send a request from localhost:3000 to localhost:5001. So, basically you use different URLs (differing by port number) and that throws a CORS error, that is saying to you that
you have a different origin from the target
the target did not allow you to do so
The solution is simple: make the target allowing you to do this. Before you continue, read this: https://developer.mozilla.org/en-US/docs/Web/HTTP/CORS/Errors
You will need to have a Access-Control-Allow-Origin header in your response at localhost:5001. You could specify a wildcard of
Access-Control-Allow-Origin: *
to allow any origin to send a request. You may allow a single source, by putting the source as a value instead of *. Read more here: https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Access-Control-Allow-Origin
Changing the response headers and adding the Access-Control-Allow-Origin is a topic by itself and can be done differently for different servers. You will need to know what server you use for your project that's triggered when you request on localhost:5001 and to look up how can you add this header there.
I have a firebase function which serves a website as per below:
const app = express();
app.use(express.static("my-website"));
exports.app = functions.https.onRequest(app);
Where my-website has my-website/subfolder/index.html and my-website/subfolder/index.js. I have deployed the function and it successfully serves up index.html and index.js. However, index.js is attempting to download a file from Cloud Storage but is getting the following error:
Access to XMLHttpRequest at 'https://storage.cloud.google.com/my-project.appspot.com/my-project-files/data.csv' from origin 'https://us-central1-my-project.cloudfunctions.net' has been blocked by CORS policy: No 'Access-Control-Allow-Origin' header is present on the requested resource.
I am not using the firebase cloud storage web library, because the code is already written using the d3.js library as per below, and I can't change it:
d3.queue()
.defer(
d3.csv,
"https://storage.cloud.google.com/my-project.appspot.com/my-project-files/data.csv"
)
.defer(d3.json, "data/config.json")
.defer(d3.json, "data/geog.json")
.await(ready);
This code generates the following request:
:authority: storage.cloud.google.com
:method: GET
:path: /my-project.appspot.com/my-project-files/data.csv
:scheme: https
accept: text/csv,*/*
accept-encoding: gzip, deflate, br
accept-language: en-US,en;q=0.9
cache-control: no-cache
origin: https://us-central1-my-project.cloudfunctions.net
pragma: no-cache
referer: https://us-central1-my-project.cloudfunctions.net/app/my-website/subfolder/index.html
sec-fetch-dest: empty
sec-fetch-mode: cors
sec-fetch-site: cross-site
user-agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_13_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/83.0.4103.61 Safari/537.36
I have set a CORS policy on my bucket to allow requests from my cloud functions, gsutil cors get gs://my-project.appspot.com returns:
[{"maxAgeSeconds": 3600,
"method": ["GET"],
"origin": ["https://us-central1-my-project.cloudfunctions.net"],
"responseHeader": ["Content-Type"]}]
I am not sure where to go from here. What am I doing wrong?
This is a fairly common error for people setting up webapps. To make sure that sites are secure and to prevent code injections, browsers prevent cross-origin requests. Mozilla has a great manual on how to solve this. https://developer.mozilla.org/en-US/docs/Web/HTTP/CORS/Errors
I worked out that the problem is the Cloud Storage doesn't actually let you allow CORS for the storage.cloud.google.com, instead you have to use storage.googleapis.com.
I am trying to pull a web page in my client (not a browser) with the following settings in the HTTP header
Accept: "text/html;charset=UTF-8"
Accept-Charset: "ISO-8859-1"
User-Agent: "Mozilla/5.0"
however I get an error code 406,
I also tried changing to;
Accept: "text/html"
with no success; error code and status message in the response header is
statusCode: 406
statusMessage: "Not Acceptable"
any idea waht the correct header settings should be, the page loads fine in the browser
Finally figured it out, I ran a sniffer to see which header settings worked, and here is what worked in every case
headers: {
'User-Agent': 'Mozilla/5.0 (Macintosh; U; Intel Mac OS X; de-de) AppleWebKit/523.10.3 (KHTML, like Gecko) Version/3.0.4 Safari/523.10',
'Accept-Charset': 'ISO-8859-1,UTF-8;q=0.7,*;q=0.7',
'Accept-language' : 'de,en;q=0.7,en-us;q=0.3'
}
You should add Accept-Language. See Here
Why are you sending contradictory headers? You are requesting a representation that is both UTF8 and ISO-8859-1 at the same time. I guess that you could interpret the request as being for 7-bit ASCII representation.
In this case I would omit the Accept-Charset and change the Accept header to text/html, */*;q=0.1 so that you will get something back with a strong preference for HTML. See the Content Negotiation section of RFC7231 for details about these headers.
I've been scraping data from an API using R with the httr and plyr libraries. Its pretty straight forward and works well with the following code:
library(httr)
library(plyr)
headers <- c("Accept" = "application/json, text/javascript",
"Accept-Encoding" = "gzip, deflate, sdch",
"Connection" = "keep-alive",
"Referer" = "http://www.afl.com.au/stat",
"Host" = "www.afl.com.au",
"User-Agent" = "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/41.0.2272.118 Safari/537.36",
"X-Requested-With"= "XMLHttpRequest",
"X-media-mis-token" = "f31fcfedacc75b1f1b07d5a08887f078")
query <- GET("http://www.afl.com.au/api/cfs/afl/season?seasonId=CD_S2016014", add_headers(headers))
stats <- httr::content(query)
My question is with regards to the request token required in the headers (i.e. X-media-mis-token). This is easy to get manually by inspecting the XHR elements in Chrome or Firefox, but the token is updated every 24 hrs making manual extraction a pain.
Is it possible to query the web page and extract this token automatically using R?
You can get the X-media-mis-token token, but with a disclaimer. ;)
library(httr)
token_url <- 'http://www.afl.com.au/api/cfs/afl/WMCTok'
token <- POST(token_url, encode="json")
content(token)$token
#[1] "f31fcfedacc75b1f1b07d5a08887f078"
content(token)$disclaimer
#[1] "All content and material contained within this site is protected by copyright owned by or licensed to Telstra. Unauthorised reproduction, publishing, transmission, distribution, copying or other use is prohibited.
I am writing a browser that sends http get requests constructed the same way browsers do. I want to test the "keep-alive" header so I send and receive the following package twice:
"POST /test.php HTTP1.1\r\n"
"Host: 192.168.0.1:8000\r\n"
"Connection: keep-alive\r\n"
"Content-Length: 17\r\n"
"Content-Type: application/x-www-form-urlencoded\r\n"
"Accept-Language: en-US,en;q=0.8\r\n"
"Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.3\r\n"
"User-Agent: Mozilla/5.0 (Windows NT 5.1) AppleWebKit/535.2 (KHTML, like Gecko) Chrome/15.0.874.106 Safari/535.2\r\n"
"Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8\r\n"
"Referer: http://192.168.0.1:8000/test.php\r\n"
"Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.3\r\n"
"\r\n\r\n"
"para:test_tst_tst"
However, when i receive the second time, I get error 10054 (windows error). It seems as though the server is refusing to send me data.
What this wrong with this?
I guess the error 10054 you are talking about is the Winsock error "Connection reset by peer".
Your problem is that you have 3 \r\n pairs between http header and body, rather than just 2
"Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.3\r\n"
"\r\n\r\n"
"para:test_tst_tst"
Quoting from the spec:
generic-message = start-line
*(message-header CRLF)
CRLF
[ message-body ]
So the server will read the third \r\n pair as the first 2 bytes of the body, and then what remains of the 17 bytes of body length as specified in the Content-Length header. So rather than starting with POST , your second request will be interpreted as starting with stPOST , that is what's probably confusing the server.
Remove one \r\n pair and you should be fine.
Btw, Keep-Alive is the default value for the Connection-Type in HTTP 1.1, so you don't even need to specify that explicitly.