I want to delete useless header to get lower latency.
Is it possible delete these useless headers?
I want to remove 'Set-Cookie: header first.
Is there any prepared request header options related to cookie?
If it isn't possible, second possible is using http2-ALPN?
help me...
'Date: Sat, 19 Mar 2022 12:38:02 GMT',
b'Expires: -1',
b'Cache-Control: private, max-age=0',
b'Content-Type: text/html; charset=ISO-8859-1',
b'P3P: CP="This is not a P3P policy! See g.co/p3phelp for more info."',
b'Server: gws',
b'X-XSS-Protection: 0',
b'X-Frame-Options: SAMEORIGIN',
b'Set-Cookie: 1P_JAR=2022-03-19-12; expires=Mon, 18-Apr-2022 12:38:02 GMT; pat'
b'h=/; domain=.google.com; Secure',
b'Set-Cookie: NID=511=j5NUUUt5vcHCkxIH0xRujZNH3plmQnZ3gA84H5CDCvAETMxHSLp5fioV'
b'lxdPbejtuP4qU1v9tctVbN0JaZ6M1ALiNKV2M35hf42KJ16KIuhk_tpesBC5hDD70Bl-1ZxhaIl6'
b'aRS55sUmbcXSIQ9BSgFpiR4OByiOEi1hRH9OzqA; expires=Sun, 18-Sep-2022 12:38:02 G'
b'MT; path=/; domain=.google.com; HttpOnly',
b'Alt-Svc: h3=":443"; ma=2592000,h3-29=":443"; ma=2592000,h3-Q050=":443"; ma=2'
b'592000,h3-Q046=":443"; ma=2592000,h3-Q043=":443"; ma=2592000,quic=":443"; ma'
b'=2592000; v="46,43"',
b'Accept-Ranges: none',
b'Vary: Accept-Encoding',
b'Transfer-Encoding: chunked',
b'',
b'4a92'
import socket
import ssl
import pprint
context = ssl.create_default_context()
conn = context.wrap_socket(socket.socket(socket.AF_INET),
server_hostname="www.google.com")
conn.connect(("www.google.com", 443))
conn.sendall(b"GET / HTTP/1.1\r\nHost: www.google.com:443\r\n\r\n")
pprint.pprint(conn.recv(1024).split(b"\r\n"))
Is it possible delete ....
A client cannot remove headers the server is sending - unless the server is specifically programmed to let the client do this by instrumenting the HTTP request. Usually this is not the case.
... these useless headers?
These headers actually have a defined meaning, i.e. they are not useless.
... to get lower latency
Omitting these headers will likely not improve latency at all. Latency primarily depends on the round trip time and on the number of data exchanges were one side needs to wait for the other to continue. Costly for latency are TCP handshake and TLS handshake, but not really if the server sends a bit more data in the HTTP response.
Related
For my research I need to cURL the fqdns and get their status codes. (For Http, Https services) But some http urls open as https although it returns 200 with cURL. (successful request, no redirect)
curl -I http://example.example.com/
HTTP/1.1 200 OK
Server: nginx
Date: Mon, 22 Nov 2021 10:43:32 GMT
Content-Type: text/html; charset=UTF-8
Content-Length: 64991
Connection: keep-alive
Keep-Alive: timeout=20
Vary: Accept-Encoding
Expires: Thu, 19 Nov 1981 08:52:00 GMT
Pragma: no-cache
Link: <https://example.example.com/>; rel=shortlink
X-Powered-By: WP Engine
X-Cacheable: SHORT
Vary: Accept-Encoding,Cookie
Cache-Control: max-age=600, must-revalidate
X-Cache: HIT: 10
X-Cache-Group: normal
Accept-Ranges: bytes
As seen above I get 200 response with curl request. But I can see the 307 code in my browser. (available in the picture below)
Request URL: http://example.example.com/
Request Method: GET
Status Code: 307 Internal Redirect
Referrer Policy: strict-origin-when-cross-origin
Can I detect 307 code with curl? (-L parameter doesn't work) Any suggestions?
curl -w '%{response_code}\n' -so /dev/null $URL
It can be tested out like this:
curl -w '%{response_code}\n' -so /dev/null httpbin.org/status/307
so what is the 307 in the question?
As Stefan explains here in a separate answer: that's an internal message from Chrome that informs you that it uses HSTS. It is not an actual response code. Which is why curl can't show it. Chrome should make that clearer.
HSTS
HSTS is a way for a HTTPS server to ask clients to not contact them over clear text HTTP again. curl also supports HSTS but then you need to use --hsts - and curl will still not confusingly claim any 307 response codes.
The 307 http status isn't actually a response that is sent by a server. It's an internal redirect, something that your browser does for you before even sending the request to the server.
That's why it won't show up in curl. It's a feature of your browser. cURL is much more reliable when it comes to sending unaltered requests.
A 307 (especially since you mention https redirects) internal redirect is usually encountered when dealing with the security feature of HSTS (HTTP strict-transport-security) where the whole purpose is to make sure that you never send unencrypted http requests to a server that wants to communicate via encrypted https.
See this.
I'm writing a tool that runs periodically and reads and processes batches of documents from a CosmosDb collection. I'd like to read all documents that have not been processed yet including changed and added documents.
List (ReadFeed) Documents states that:
ReadFeed can be used to retrieve (...) the incremental changes to documents within the collection.
and that this can be done by setting the If-None-Match request header etag received in the response header:
The logical sequence number (LSN) of last document returned in the response.
incremental ReadDocumentFeed can be resumed by resubmitting this value in If-None-Match.
but there is no such header in the response.
I used both REST API and .NET Cosmos SDK v3 and tried setting If-None-Match header to:
_etag of the last processed document
etag of the container
But I'm getting the same, full set of documents each time.
A sample request I have:
GET /dbs/myData/colls/myItems/docs HTTP/2
Host: cosmos-local.documents.azure.com
authorization: type%3dmaster%26ver%3d1.0%26sig%3d...%2f...%2f...%3d
x-ms-date: Fri, 19 Feb 2021 08:44:23 GMT
x-ms-version: 2018-12-31
if-none-match: "44000253-0000-0d00-0000-601428c90000"
accept: */*
I tried these etag formats in If-None-Match:
"\"44000253-0000-0d00-0000-601428c90000\""
"44000253-0000-0d00-0000-601428c90000"
44000253-0000-0d00-0000-601428c90000
a response:
HTTP/2 200
cache-control: no-store, no-cache
pragma: no-cache
content-type: application/json
content-location: https://cosmos-local.documents.azure.com/dbs/myData/colls/myItems/docs
server: Microsoft-HTTPAPI/2.0
strict-transport-security: max-age=31536000
x-ms-activity-id: 765ea7f5-40c6-48fb-bf90-7b3b506b0b82
x-ms-last-state-change-utc: Thu, 11 Feb 2021 08:35:47.977 GMT
x-ms-resource-quota: documentSize=51200;documentsSize=52428800;documentsCount=-1;collectionSize=52428800;
x-ms-resource-usage: documentSize=0;documentsSize=238;documentsCount=19;collectionSize=243;
x-ms-schemaversion: 1.11
lsn: 174
x-ms-item-count: 19
x-ms-request-charge: 2.53
x-ms-alt-content-path: dbs/myData/colls/myItems
x-ms-content-path: eiBDAJIWdUc=
x-ms-documentdb-partitionkeyrangeid: 0
x-ms-xp-role: 1
x-ms-global-committed-lsn: 173
x-ms-number-of-read-regions: 0
x-ms-transport-request-id: 1
x-ms-cosmos-llsn: 174
x-ms-session-token: 0:-1#174
x-ms-request-duration-ms: 1.002
x-ms-serviceversion: version=2.11.0.0
x-ms-gatewayversion: version=2.11.0
date: Fri, 19 Feb 2021 10:45:10 GMT
Other attempts I made:
Set A-IM header to Incremental feed - got an error in the response
Set value from lsn header in the If-None-Match - got full set
Use ChangeFeedProcessor. It looks like what I need, but it keeps waiting for new data and I'd like it to shut down / timeout if there are no new changes.
Check if setting x-ms-max-item-count makes any difference - it doesn't seem to
So the questions are:
How to run incremental ReadFeed in CosmosDb?
Is there a better approach to read all documents in a collection incrementally, in separate runs?
Best approach is to use Change Feed. You can shut this down if you want. Just call StopAsync() on the processor object. If you're running this on a schedule you can host in Azure Functions and run it from a timer. When it starts up it will connect to the lease collection and start processing again from the last lsn. When it's done processing, call StopAsync() and it will shut down.
When receiving a response back with a netty client object, I run into a FrameTooLongException. After taking a tcpdump, found that the response received is a large Mutlipart Mime response with about 200 parts (each with some short headers), but the actual HTTP Header for the response is quite small and are listed as;
> Host: foobar.com:20804
> Accept: */*
>
< HTTP/1.1 207 Multi-Status
< Date: Tue, 04 Aug 2015 19:44:09 GMT
< Vary: Accept
< Content-Type: multipart/mixed; boundary="63602357878446117"
< Content-Length: 33023
I couldn't find anything in the documentation about this, but are Mime part headers used in the HTTP Header size calculation, and does Netty parse it as such?
The exception I get is as follows:
io.netty.handler.codec.TooLongFrameException: HTTP header is larger than 8192 bytes.
at io.netty.handler.codec.http.HttpObjectDecoder$HeaderParser.newException(HttpObjectDecoder.java:787)
at io.netty.handler.codec.http.HttpObjectDecoder$HeaderParser.process(HttpObjectDecoder.java:779)
at io.netty.buffer.AbstractByteBuf.forEachByteAsc0(AbstractByteBuf.java:1022)
at io.netty.buffer.AbstractByteBuf.forEachByte(AbstractByteBuf.java:1000)
at io.netty.handler.codec.http.HttpObjectDecoder$HeaderParser.parse(HttpObjectDecoder.java:751)
at io.netty.handler.codec.http.HttpObjectDecoder.readHeaders(HttpObjectDecoder.java:545)
at io.netty.handler.codec.http.HttpObjectDecoder.decode(HttpObjectDecoder.java:221)
at io.netty.handler.codec.http.HttpClientCodec$Decoder.decode(HttpClientCodec.java:136)
at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:315)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:229)
at io.netty.channel.CombinedChannelDuplexHandler.channelRead(CombinedChannelDuplexHandler.java:147)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:339)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:324)
at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1044)
at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:934)
at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:315)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:229)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:339)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:324)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:847)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:131)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:511)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:468)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:382)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:354)
at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:111)
Http header terminates with 2 cr/lf (such as between Accept and HTTP in your example), and header shall start with a "start line" (HTTP/1.1...).
Therefore I see 2 issues with your example:
Your header does not start correctly : HTTP/1.1 should be the first line, followed later on by your accept and other host header params.
Probably there is something wrong in your response such that there is no 2 cr/of between your header and the body, thus leading to the decoding of the body as if it was part of the header, so the exception...
I am developing an HTTP proxy in Java. I resend all the data from client to server without touching it, but for some URLs (for example this) server returns the 404 error if I am connecting through my proxy.
The requested URL uses Varnish caching, so it might be the root of problem. I cannot reconfigure it - it is not my.
If I request that URL directly with browser, the server returns 200 and the image is shown correctly.
I am stuck because I even do not know what to read and how to compose a search request.
Thanks a lot.
Fix the Host: header of the re-issued request. The request going out from the proxy either has no Host header or it is broken (or only X-Host exists). Also take note that the proxy application will execute its own DNS lookup and that might yield a different IP address than your local computer (where you issued the original request).
This works:
> curl -s -D - -o /dev/null http://212.25.95.152/w/w-200/1902047-41.jpg -H "Host: msc.wcdn.co.il"
HTTP/1.1 200 OK
Content-Type: image/jpeg
Cache-Control: max-age = 315360000
magicmarker: 1
Content-Length: 27922
Accept-Ranges: bytes
Date: Sun, 05 Jul 2015 00:52:08 GMT
X-Varnish: 2508753650 2474246958
Age: 67952
Via: 1.1 varnish
Connection: keep-alive
X-Cache: HIT
I have an assignment where I need to determine how much cache space will be required to store the contents of a webpage, and I have to do it all in Scala, which I'm in the process of learning. I know I can get the required information with a HTTP HEAD request, but from what I've read it seems I need an external library for that.
Is it possible to download the HTTP header without using an HTTP request and extract the required information using only Scala (no calls to Java code)?
If you need not use 3rd party libraries, then the solution might be to use Source.fromURL to get the page and then compute its size.
Hope this helps ;)
Without your restriction that only Scala may be used I would have said: use Async-Http-Client's AsyncHandler and stop as soon as onHeadersReceived has been called.
Without external libraries, you could try to mimic what a HTTP client is doing. Here's a sample telnet session:
$ telnet www.google.com 80
HEAD / Trying 173.194.40.20...
Connected to www.google.com.
Escape character is '^]'.
HEAD / HTTP/1.1
Host: www.google.com
HTTP/1.1 302 Found
Location: http://www.google.ch/
Cache-Control: private
Content-Type: text/html; charset=UTF-8
Set-Cookie: PREF=ID=c2b92507b9088226:FF=0:TM=1361870408:LM=1361870408:S=mbY_Qws86Z75gPAk; expires=Thu, 26-Feb-2015 09:20:08 GMT; path=/; domain=.google.com
Set-Cookie: NID=67=dAFEWKT5vk9HWP1sTF6Oo49jv0sRV7_49ewSgD3fYRiTjHqlUasKl7Jz86SnJhtS-o9zU9raxwCLhdfvEwdwl9imRwONMBTDBKDXtJhFufLCnAoOKgDQetv0A5FTN3Da; expires=Wed, 28-Aug- 2013 09:20:08 GMT; path=/; domain=.google.com; HttpOnly
P3P: CP="This is not a P3P policy! See http://www.google.com/support/accounts/bin/answer.py?hl=en&answer=151657 for more info."
Date: Tue, 26 Feb 2013 09:20:08 GMT
Server: gws
Content-Length: 218
X-XSS-Protection: 1; mode=block
X-Frame-Options: SAMEORIGIN
(What I typed was HEAD / HTTP/1.1, Host: www.google.com, and an additional return.)
You could try to use the JVM's Socket class to open a TCP connection to your server and send, as in the example above, the HEAD request yourself.