Magnolia: Range request doesn't serve content when cache filter enabled resulting in Facebook Sharing not to work - magnolia

When sending an HTTP request with a Range header to Magnolia I get a Response with
Content-Length: 0:
curl -I -X GET \
http://localhost:8080/ \
-H 'Accept-Encoding: gzip, deflate' \
-H 'Cache-Control: no-cache' \
-H 'Range: bytes=0-2000'
HTTP/1.1 206
Set-Cookie: SID=C36D961EC92D152724BBCD0C34EC6536; Path=/; HttpOnly
X-Magnolia-Registration: Registered
Accept-Ranges: bytes
Cache-Control: no-cache, no-store, must-revalidate, max-age=0
ETag: 8B4901E7DD862E5E74287A0F538DCDDFEB78DE77
Content-Range: bytes 0-2000/23529
Content-Encoding: gzip
Vary: Accept-Encoding
Pragma: no-cache
Expires: Thu, 01 Jan 1970 00:00:00 GMT
Last-Modified: Thu, 19 Dec 2019 08:52:49 GMT
Content-Type: text/html;charset=UTF-8
Content-Length: 0
Date: Thu, 19 Dec 2019 08:52:49 GMT
However, when I disable the Magnolia Cache Module I get the expected response:
/server/filters/cache -> enabled: false
curl -I -X GET \
http://localhost:8080/ \
-H 'Accept-Encoding: gzip, deflate' \
-H 'Cache-Control: no-cache' \
-H 'Range: bytes=0-2000'
HTTP/1.1 206
Set-Cookie: SID=FF557EC1F0653E5CBD81A57D599091AE; Path=/; HttpOnly
X-Magnolia-Registration: Registered
Accept-Ranges: bytes
ETag: 2A9DE4F4B2ACDDE22BAC3C07784CD65693574B67
Content-Range: bytes 0-2000/2147483647
Content-Type: text/html;charset=UTF-8
Content-Length: 2001
Date: Thu, 19 Dec 2019 08:51:49 GMT
I got the problem that the Facebook crawler isn't able to detect any open graph meta tags when trying to crawl my website. I think the reason is the above described problem with sending range requests to Magnolia (What the Facebook crawler does).
My Open Graph tags are properly set (Working for opengraphcheck and Twitter Card Validator).
I'm using Magnolia 5.7.1.

The simplest work around is to configure request header voter to bypass cache when range header is present.
See RequestHeaderPatternSimpleVoter and/or RequestHeaderPatternRegexVoter for more details on how to set it, but I would still consider it workaround and not final solution.
It seems weird that such thing should be happening. Could you replicate it against e.g. https://demo.magnolia-cms.com?

Related

How a HTTP client can benefit from multipart ranges?

I can't get my head around multipart ranges usage. Sounds interesting but in what way it is beneficial? How a client can use such a response? and how content-length became 804??
(5−0)+(15−9)+(3587−3147)+3 = 454
$ curl -X GET -H "Range: bytes=0-5, 9-15, 3147-3587" http://dl11.f2m.io/film/The.Spy.Gone.North.2018.720p.BluRay.Film2Movie_WS.mkv
HTTP/1.1 206 Partial Content
Server: nginx
Date: Sat, 13 Apr 2019 10:48:16 GMT
Content-Type: multipart/byteranges; boundary=00000000000000000008
Content-Length: 804
Last-Modified: Sat, 13 Apr 2019 09:23:42 GMT
Connection: keep-alive
ETag: "5cb1aa9e-51539d77"
--00000000000000000008
Content-Type: application/octet-stream
Content-Range: bytes 0-5/1364434295
�Eߣ�B
--00000000000000000008
Content-Type: application/octet-stream
Content-Range: bytes 9-15/1364434295
B��B�
--00000000000000000008
Content-Type: application/octet-stream
Content-Range: bytes 3147-3587/1364434295
--00000000000000000008--
If i wanted to download the resource in parallel (threading) i could use multiple Range requests.
$ curl -X GET -H "Range: bytes=0-5" {url}
$ curl -X GET -H "Range: bytes=9-15" {url}
$ curl -X GET -H "Range: bytes=3147-3587" {url}
So what is multipart ranges usage?

Does Firebase support HTTP HEAD request?

My app has files on firebase storage that i need to serve up to another service. The files are publicly accessible but this service likes to make a HEAD request first which is denied by firebase (error 400).
An this be configured somehow? I believe that Google storage supports this.
eg: file get is ok:
$ curl https://firebasestorage.googleapis.com/v0/b/test-451f9.appspot.com/o/temp%2Fhello.txt?
alt=media -o -
Hello
but the HEAD request:
$ curl --head https://firebasestorage.googleapis.com/v0/b/test-451f9.appspot.com/o/temp%2Fhel
lo.txt?alt=media -o -
HTTP/2 400
x-guploader-uploadid: AEnB2UqWsCbhq_AKpXh29El8_aiJnZqDEUeGsn2i1j0ZPQie0-OB2AQjnKqi_ya50hIw7Yb4WmlKV19ilYQBk9KGdndj4oX9oQ
x-content-type-options: nosniff
content-type: application/json; charset=UTF-8
access-control-expose-headers: Content-Range, X-Firebase-Storage-XSRF
access-control-allow-origin: *
date: Mon, 10 Dec 2018 17:20:52 GMT
expires: Mon, 10 Dec 2018 17:20:52 GMT
cache-control: private, max-age=0
server: UploadServer
alt-svc: quic=":443"; ma=2592000; v="44,43,39,35"
fails.

Why does curl repeat headers in the output?

Options I used:
-I, --head
(HTTP/FTP/FILE) Fetch the HTTP-header only! HTTP-servers feature
the command HEAD which this uses to get nothing but the header
of a document. When used on an FTP or FILE file, curl displays
the file size and last modification time only.
-L, --location
(HTTP/HTTPS) If the server reports that the requested page has moved to a different location (indi-
cated with a Location: header and a 3XX response code), this option will make curl redo the request
on the new place. If used together with -i, --include or -I, --head, headers from all requested
pages will be shown. When authentication is used, curl only sends its credentials to the initial
host. If a redirect takes curl to a different host, it won't be able to intercept the user+password.
See also --location-trusted on how to change this. You can limit the amount of redirects to follow
by using the --max-redirs option.
When curl follows a redirect and the request is not a plain GET (for example POST or PUT), it will
do the following request with a GET if the HTTP response was 301, 302, or 303. If the response code
was any other 3xx code, curl will re-send the following request using the same unmodified method.
You can tell curl to not change the non-GET request method to GET after a 30x response by using the
dedicated options for that: --post301, --post302 and -post303.
-v, --verbose
Be more verbose/talkative during the operation. Useful for debugging and seeing what's going on
"under the hood". A line starting with '>' means "header data" sent by curl, '<' means "header data"
received by curl that is hidden in normal cases, and a line starting with '*' means additional info
provided by curl.
Note that if you only want HTTP headers in the output, -i, --include might be the option you're
looking for.
If you think this option still doesn't give you enough details, consider using --trace or --trace-
ascii instead.
This option overrides previous uses of --trace-ascii or --trace.
Use -s, --silent to make curl quiet.
Below is the output that I'm wondering about. In the response containing the redirect(301), all the headers are displayed twice, but only one of the duplicates has the < in front of it. How am I supposed to interpret that?
$ curl -ILv http://www.mail.com
* Rebuilt URL to: http://www.mail.com/
* Trying 74.208.122.4...
* Connected to www.mail.com (74.208.122.4) port 80 (#0)
> HEAD / HTTP/1.1
> Host: www.mail.com
> User-Agent: curl/7.43.0
> Accept: */*
>
< HTTP/1.1 301 Moved Permanently
HTTP/1.1 301 Moved Permanently
< Date: Sun, 28 May 2017 22:02:16 GMT
Date: Sun, 28 May 2017 22:02:16 GMT
< Server: Apache
Server: Apache
< Location: https://www.mail.com/
Location: https://www.mail.com/
< Vary: Accept-Encoding
Vary: Accept-Encoding
< Connection: close
Connection: close
< Content-Type: text/html; charset=iso-8859-1
Content-Type: text/html; charset=iso-8859-1
<
* Closing connection 0
* Issue another request to this URL: 'https://www.mail.com/'
* Trying 74.208.122.4...
* Connected to www.mail.com (74.208.122.4) port 443 (#1)
* TLS 1.2 connection using TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384
* Server certificate: *.mail.com
* Server certificate: thawte SSL CA - G2
* Server certificate: thawte Primary Root CA
> HEAD / HTTP/1.1
> Host: www.mail.com
> User-Agent: curl/7.43.0
> Accept: */*
>
< HTTP/1.1 200 OK
HTTP/1.1 200 OK
< Date: Sun, 28 May 2017 22:02:16 GMT
Date: Sun, 28 May 2017 22:02:16 GMT
< Server: Apache
Server: Apache
< Vary: X-Forwarded-Proto,Host,Accept-Encoding
Vary: X-Forwarded-Proto,Host,Accept-Encoding
< Set-Cookie: cookieKID=kid%40autoref%40mail.com; Domain=.mail.com; Expires=Tue, 27-Jun-2017 22:02:16 GMT; Path=/
Set-Cookie: cookieKID=kid%40autoref%40mail.com; Domain=.mail.com; Expires=Tue, 27-Jun-2017 22:02:16 GMT; Path=/
< Set-Cookie: cookiePartner=kid%40autoref%40mail.com; Domain=.mail.com; Expires=Tue, 27-Jun-2017 22:02:16 GMT; Path=/
Set-Cookie: cookiePartner=kid%40autoref%40mail.com; Domain=.mail.com; Expires=Tue, 27-Jun-2017 22:02:16 GMT; Path=/
< Cache-Control: no-cache, no-store, must-revalidate
Cache-Control: no-cache, no-store, must-revalidate
< Pragma: no-cache
Pragma: no-cache
< Expires: Thu, 01 Jan 1970 00:00:00 GMT
Expires: Thu, 01 Jan 1970 00:00:00 GMT
< Set-Cookie: JSESSIONID=F0BEF03C92839D69057FFB57C7FAA789; Path=/mailcom-webapp/; HttpOnly
Set-Cookie: JSESSIONID=F0BEF03C92839D69057FFB57C7FAA789; Path=/mailcom-webapp/; HttpOnly
< Content-Language: en-US
Content-Language: en-US
< Content-Length: 85237
Content-Length: 85237
< Connection: close
Connection: close
< Content-Type: text/html;charset=UTF-8
Content-Type: text/html;charset=UTF-8
<
* Closing connection 1
best guess: with -v you tell curl to be verbose (send debug info) to STDERR. with -I you tell curl to dump headers to STDOUT. and your shell, by default, combines STDOUT and STDERR. separate stdout and stderr, and you'll avoid the confusion.
curl -ILv http://www.mail.com >stdout.log 2>stderr.log ; cat stdout.log
Use:
curl -ILv http://www.mail.com 2>&1 | grep '^[<>\*].*$'
When cURL is called with the verbose command line flag, it sends the verbose output to stderr instead of stdout. The above command redirects stderr to stdout (2>&1), then we pipe the combined output to grep and use the above regex to only return the lines that begin with *, <, or >. All of the other lines in the output (including the dupes you were first concerned with) are removed from the output.

How to send POST to ThingSpeak?

How can I send a POST to ThingSpeak through www.requestmaker.com?
I'm using this:
POST /update HTTP/1.1
Host: api.thingspeak.com
Connection: close
X-THINGSPEAKAPIKEY: XXXXXXXXXXXXXXXX
Content-Type: application/x-www-form-urlencoded\
Content-Length: 24
field1=34&field2=1
But I get this response:
HTTP/1.1 404 Not Found
Content-Type: text/html; charset=utf-8
Content-Length: 728
Connection: close
Status: 404 Not Found
X-Request-Id: 36c8113c-3671-4302-85fa-f3ecdba364dc
X-Runtime: 0.019035
X-Powered-By: Phusion Passenger 4.0.57
Date: Mon, 19 Dec 2016 02:36:42 GMT
Server: nginx/1.9.3 + Phusion Passenger 4.0.57
Any solution?
Make sure you have the right API key -- it's common to get zeros and the letter 'O' incorrect. What API end point are you writing to? You can get help on the ThingSpeak forum: http://community.thingspeak.com/forum/

Getting strange http response codes, but the site is actually working

When I view the URL below or the other below in the code it's displayed fine. I don't see anything unusual in the network tab when I press F12 in the browser, but with the code below I will get response codes 403 or 400. When I use the response code checker here http://httpstatus.io/ it will come back fine with a 200 response for both URLS.
I get a 403 for http://psychsignal.com/ using my code below.
URL u = new URL("http://www.nasdaqomxnordic.com/"); //returns 400 response code
//u.toURI(); //to check the syntax
HttpURLConnection huc = (HttpURLConnection)u.openConnection();
huc.setRequestMethod("GET");
//huc.setRequestMethod("HEAD");
huc.connect();
System.out.println(huc.getResponseCode());
Thanks if anyone has any ideas! This is actually my first post!
My guess is that there's some restrictions placed on the User-Agent of the client. Some testing seems to support my theory:
If I use the curl default user agent:
# curl -I -H "User-Agent: curl/7.35.0" "http://www.nasdaqomxnordic.com/"
HTTP/1.1 400 Bad Request
Content-Type: text/html; charset=UTF-8
Cache-Control: no-cache
Pragma: no-cache
Expires: 0
Connection: close
If I use a hacked up standard browser agent string:
# curl -I -H "User-Agent: Mozilla/5.0" -0 "http://www.nasdaqomxnordic.com/"
HTTP/1.1 200 OK
Cache-Control: no-cache
Pragma: no-cache
Content-Length: 0
Content-Type: text/html;charset=UTF-8
Expires: Thu, 01 Jan 1970 00:00:00 GMT
Server: Microsoft-IIS/7.5
X-Powered-By: ASP.NET
Date: Wed, 22 Jul 2015 15:06:22 GMT
Connection: close
And then if I use a Java agent string (which is my guess as to what you're using):
# curl -I -H "User-Agent: Java/1.6.0_26" "http://www.nasdaqomxnordic.com/"
HTTP/1.1 400 Bad Request
Content-Type: text/html; charset=UTF-8
Cache-Control: no-cache
Pragma: no-cache
Expires: 0
Connection: close
Only the "browser" user agent gets through. I'd try tweaking your code to set the user agent string to something commonly found in a web browser.

Resources