I'm receiving a URL with whitespace in a query string, but HaProxy is marking such requests as bad requests I' trying to use reqrep parameter but nothing.
Example
http://example.com?ip=10.10.10.10, 1.0.0.0&xyz=abc
space between 10, 1 is not getting resolved by HaProxy.
Spaces in the URI are forbidden by RFC-3986, so you are essentially asking for HAProxy to accept a blatantly invalid request. It won't. Space characters are not valid and 400 Bad Request is the correct response. If you are receiving this from a client, then the client is broken.
HAProxy has a proxy directive option accept-invalid-http-request that relaxes the parser a little bit to allow certain broken client behavior to work, but the documentation points out that ASCII 0x20 (decimal 32) is never allowed even with this option enabled.
By default, HAProxy complies with RFC7230 in terms of message parsing. This means that invalid characters in header names are not permitted and cause an error to be returned to the client. This is the desired behavior as such forbidden characters are essentially used to build attacks exploiting server weaknesses, and bypass security filtering. Sometimes, a buggy browser or server will emit invalid header names for whatever reason (configuration, implementation) and the issue will not be immediately fixed. In such a case, it is possible to relax HAProxy's header name parser to accept any character even if that does not make sense, by specifying this option. Similarly, the list of characters allowed to appear in a URI is well defined by RFC3986, and chars 0-31, 32 (space), 34 ('"'), 60 ('<'), 62 ('>'), 92 ('\'), 94 ('^'), 96 ('`'), 123 ('{'), 124 ('|'), 125 ('}'), 127 (delete) and anything above are not allowed at all. HAProxy always blocks a number of them (0..32, 127). The remaining ones are blocked by default unless this option is enabled. (emphasis added)
You can't use reqrep to modify a message that is already invalid.
Related
We switched from citrix to HAProxy for load balancing recently.
The Problem is that for some requests HAProxy started giving a 400 Error (Which used to work well on citrix). So we moved to TCP based load balancing from a HTTP based load balancing for the time being.
On further investigating we found that some requests had an extra space in the HTTP header which caused the 400 error.
profileID<space>:value
vs
profileID:value
And these requests came from the android app so we are not able to change the source code.
We are trying to move back to http based load balancing.
Is there any config setting that may allow us to ignore the space.
HAProxy supports a proxy configuration directive called option accept-invalid-http-request.
It relaxes some of the strict protocol compliance that HAProxy correctly requires by default on incoming requests, so it should not be used blindly or carelessly without understanding the implications.
From the documentation:
By default, HAProxy complies with RFC7230 in terms of message parsing. This means that invalid characters in header names are not permitted and cause an error to be returned to the client. This is the desired behaviour as such forbidden characters are essentially used to build attacks exploiting server weaknesses, and bypass security filtering.
Sometimes, a buggy browser or server will emit invalid header names for whatever reason (configuration, implementation) and the issue will not be immediately fixed. In such a case, it is possible to relax HAProxy's header name parser to accept any character even if that does not make sense, by specifying this option.
Similarly, the list of characters allowed to appear in a URI is well defined by RFC3986, and chars 0-31, 32 (space), 34 ('"'), 60 ('<'), 62 ('>'), 92 ('\'), 94 ('^'), 96 ('`'), 123 ('{'), 124 ('|'), 125 ('}'), 127 (delete) and anything above are not allowed at all. Haproxy always blocks a number of them (0..32, 127). The remaining ones are blocked by default unless this option is enabled. This option also relaxes the test on the HTTP version, it allows HTTP/0.9 requests to pass through (no version specified) and multiple digits for both the major and the minor version.
This option should never be enabled by default as it hides application bugs and open security breaches. It should only be deployed after a problem has been confirmed.
http://cbonte.github.io/haproxy-dconv/1.6/configuration.html#4-option%20accept-invalid-http-request
(emphasis added)
Adding this option to the respective frontend section of your configurarion file should allow these invalid headers to be accepted.
Note that the potential security risks mentioned in the documentation are not risks inherent in HAProxy, but rather risks of exploits against vulnerabilities in your stack behind the proxy -- because normally, HAProxy shields those components from such invalid requests.
I'm implementing a minimum HTTPS layer for my embedded project where I'm using mbedTLS for TLS and hard-coding HTTP headers to talk with HTTPS servers.
It works fine with normal websites. But so far my implementation detects the end of HTTPS response by checking if the last byte read is \n.
if( ret > 0 && output[len-1] == '\n' )
{
ret = 0;
output[len] = 0;
break;
}
This, however, is not always working for obvious reason. I tried openssl s_client, and it behaves the same - if an HTTP response terminates with \n, then s_client returns immediately after fetching all data. Otherwise it blocks forever, waiting for more data.
An real browser seems to be able to handle this properly. Is there anything I can do beyond setting a timeout?
How to tell if an HTTP response terminates in C...
But so far my implementation detects the end of HTTPS response by checking if the last byte read is \n...
This, however, is not always working for obvious reason...
HTTP calls out \r\n, and not \n. See RFC 2616, Hypertext Transfer Protocol - HTTP/1.1 and page 15:
HTTP/1.1 defines the sequence CR LF as the end-of-line marker for all
protocol elements except the entity-body (see appendix 19.3 for
tolerant applications). The end-of-line marker within an entity-body
is defined by its associated media type, as described in section 3.7.
CRLF = CR LF
Now, what various servers send is a whole different ballgame. There will be duplicate end-of-line markers, missing end-of-line markers, and incorrect end-of-line markers. Its the wild, wild west.
You might want to look at a reference implementation of a HTTP parser. If so, check out libevent or cURL's parsers and how they maintain their state machine.
I want to ask some question about the following quote taken from Head First Servlets and JSP, Second Edition book:
The total amount of characters in a GET is really limited (depending
on the server). If the user types, say, a long passage into a “search”
input box, the GET might not work.
Why is the total amount of characters in a GET limited?
How can I learn about the total amount of character in a Get?
When I said a long text into any input box, and GET is not working.
How many solution do I have to fix this problem.
Why is the get method limited?
There is no specific limit to the length of a GET request. Different servers can have different limits. If you need to send more data to the server, use POST instead of GET. A recommended minimum to be supported by servers and browsers is 8,000 bytes, but this is not required.
RFC 7230's Section 3.1.1 "Request Line" says
HTTP does not place a predefined limit on the length of a request-line, as described in Section 2.5. A server that receives a method longer than any that it implements SHOULD respond with a 501 (Not Implemented) status code. A server that receives a request-target longer than any URI it wishes to parse MUST respond with a 414 (URI Too Long) status code (see Section 6.5.12 of RFC7231).
Various ad hoc limitations on request-line length are found in practice. It is RECOMMENDED that all HTTP senders and recipients support, at a minimum, request-line lengths of 8000 octets.
Section 2.5 "Conformance and Error Handling" says
HTTP does not have specific length limitations for many of its protocol elements because the lengths that might be appropriate will vary widely, depending on the deployment context and purpose of the implementation. Hence, interoperability between senders and recipients depends on shared expectations regarding what is a reasonable length for each protocol element. Furthermore, what is commonly understood to be a reasonable length for some protocol elements has changed over the course of the past two decades of HTTP use and is expected to continue changing in the future.
and RFC 7231's Section 6.5.12 "414 URI Too Long" says
The 414 (URI Too Long) status code indicates that the server is
refusing to service the request because the request-target (Section
5.3 of [RFC7230]) is longer than the server is willing to interpret.
This rare condition is only likely to occur when a client has
improperly converted a POST request to a GET request with long query
information, when the client has descended into a "black hole" of
redirection (e.g., a redirected URI prefix that points to a suffix of
itself) or when the server is under attack by a client attempting to
exploit potential security holes.
The 'get'-data is send in the query string - which also has a maximum length.
You can do all kind of things with the query string, e.g. bookmark it. Would you really like to bookmark a real huge text?
It is possible to configure moste servers to use larger length - some clients will accept them, some will throw errors.
"Note: Servers ought to be cautious about depending on URI lengths above 255 bytes, because some older client or proxy implementations might not properly support these lengths." HTTP 1.1 specification chapter 3.2.1:.
There is also a status code "414 Request-URI Too Long" - if you get this you will know that you have put to many chars in the get. (If you hit the server limit, if the client limit is lower then the server limit each browser will react in it's own way).
Generally it would be wise to set a limit for each data being send to a server - just if someones tries to make huge workload or slow down the server (e.g. send a huge file - 1 server connection is used. slow down the transmission, make additional sends - at some point the server wil have a lot of connections open. Use multiple clients and you have an attack scenario on a server).
What's the maximum length of an HTTP GET request?
Is there a response error defined that the server can/should return if it receives a GET request that exceeds this length?
This is in the context of a web service API, although it's interesting to see the browser limits as well.
The limit is dependent on both the server and the client used (and if applicable, also the proxy the server or the client is using).
Most web servers have a limit of 8192 bytes (8 KB), which is usually configurable somewhere in the server configuration. As to the client side matter, the HTTP 1.1 specification even warns about this. Here's an extract of chapter 3.2.1:
Note: Servers ought to be cautious about depending on URI lengths above 255 bytes, because some older client or proxy implementations might not properly support these lengths.
The limit in Internet Explorer and Safari is about 2 KB, in Opera about 4 KB and in Firefox about 8 KB. We may thus assume that 8 KB is the maximum possible length and that 2 KB is a more affordable length to rely on at the server side and that 255 bytes is the safest length to assume that the entire URL will come in.
If the limit is exceeded in either the browser or the server, most will just truncate the characters outside the limit without any warning. Some servers however may send an HTTP 414 error.
If you need to send large data, then better use POST instead of GET. Its limit is much higher, but more dependent on the server used than the client. Usually up to around 2 GB is allowed by the average web server.
This is also configurable somewhere in the server settings. The average server will display a server-specific error/exception when the POST limit is exceeded, usually as an HTTP 500 error.
You are asking two separate questions here:
What's the maximum length of an HTTP GET request?
As already mentioned, HTTP itself doesn't impose any hard-coded limit on request length; but browsers have limits ranging on the 2 KB - 8 KB (255 bytes if we count very old browsers).
Is there a response error defined that the server can/should return if it receives a GET request exceeds this length?
That's the one nobody has answered.
HTTP 1.1 defines status code 414 Request-URI Too Long for the cases where a server-defined limit is reached. You can see further details on RFC 2616.
For the case of client-defined limits, there isn't any sense on the server returning something, because the server won't receive the request at all.
Browser limits are:
Browser Address bar document.location
or anchor tag
---------------------------------------------------
Chrome 32779 >64k
Android 8192 >64k
Firefox >64k >64k
Safari >64k >64k
Internet Explorer 11 2047 5120
Edge 16 2047 10240
Want more? See this question on Stack Overflow.
A similar question is here: Is there a limit to the length of a GET request?
I've hit the limit and on my shared hosting account, but the browser returned a blank page before it got to the server I think.
Technically, I have seen HTTP GET will have issues if the URL length goes beyond 2000 characters. In that case, it's better to use HTTP POST or split the URL.
As already mentioned, HTTP itself doesn't impose any hard-coded limit on request length; but browsers have limits ranging on the 2048 character allowed in the GET method.
Yes. There isn't any limit on a GET request.
I am able to send ~4000 characters as part of the query string using both the Chrome browser and curl command.
I am using Tomcat 8.x server which has returned the expected 200 OK response.
Here is the screenshot of a Google Chrome HTTP request (hiding the endpoint I tried due to security reasons):
RESPONSE
is there any limitation for url?how many characters can take the maximum url? thank u.
This depends on the browser you are using.
From RFC 2616 (rfc2616) - Hypertext Transfer Protocol -- HTTP/1.1
The HTTP protocol does not place any a
priori limit on the length of a
URI. Servers MUST be able to handle
the URI of any resource they serve,
and SHOULD be able to handle URIs of
unbounded length if they provide
GET-based forms that could generate
such URIs. A server SHOULD return
414 (Request-URI Too Long) status if a
URI is longer than the server can
handle (see section 10.4.15).
Note: Servers ought to be cautious about depending on URI
lengths
above 255 bytes, because some older client or proxy
implementations might not properly support these lengths.
Maximum URL length is 2,083 characters in Internet Explorer
Microsoft Internet Explorer has a
maximum uniform resource locator (URL)
length of 2,083 characters. Internet
Explorer also has a maximum path
length of 2,048 characters. This limit
applies to both POST request and GET
request URLs.
If you are using the GET method, you
are limited to a maximum of 2,048
characters, minus the number of
characters in the actual path.
However, the POST method is not
limited by the size of the URL for
submitting name/value pairs. These
pairs are transferred in the header
and not in the URL.
From http://www.boutell.com/newfaq/misc/urllength.html
Firefox (Browser)
After 65,536 characters, the location bar no longer displays the URL in Windows Firefox
1.5.x. However, longer URLs will work. I stopped testing after 100,000 characters.
Safari (Browser)
At least 80,000 characters will work. I stopped testing after 80,000 characters.
Opera (Browser)
At least 190,000 characters will work. I stopped testing after 190,000 characters. Opera > 9 for Windows continued to display a fully editable, copyable and pasteable URL in the
location bar even at 190,000 characters.
Apache (Server)
My early attempts to measure the maximum URL length in web browsers bumped into a server > URL length limit of approximately 4,000 characters, after which Apache produces a "413
Entity Too Large" error. I used the current up to date Apache build found in Red Hat
Enterprise Linux 4. The official Apache documentation only mentions an 8,192-byte limit
on an individual field in a request.
Microsoft Internet Information Server
The default limit is 16,384 characters (yes, Microsoft's web server accepts longer URLs
than Microsoft's web browser). This is configurable.
Perl HTTP::Daemon (Server)
Up to 8,000 bytes will work. Those constructing web application servers with Perl's
HTTP::Daemon module will encounter a 16,384 byte limit on the combined size of all HTTP
request headers. This does not include POST-method form data, file uploads, etc., but it > does include the URL. In practice this resulted in a 413 error when a URL was
significantly longer than 8,000 characters. This limitation can be easily removed. Look
for all occurrences of 16x1024 in Daemon.pm and replace them with a larger value. Of
course, this does increase your exposure to denial of service attacks.
Check this thread of Stackoverflow What is the maximum length of a URL in different browsers?
Maximum URL length is 2083 characters in Internet Explorer
http://support.microsoft.com/kb/208427
Maximum URL length is 65000 characters in Mozila Firefox
http://support.mozilla.com/tiki-view_forum_thread.php?comments_offset=0&comments_threadId=0&comments_parentId=153230&comments_threshold=0&thread_sort_mode=commentDate_asc&forumId=1&time_control=3600
it depends on browser : for ie max characters is 2083 - http://support.microsoft.com/kb/208427