Managing "HTTP/1.1 502 Bad gateway error" - http

I need to interact with a remote HTTP server at the lowest possible level (i.e.: at socket level) because my target is a very small embedded system with no support for higher level libraries (it's a bare-metal uController wit no O.S. at all and talking to a GSM modem via serial line; modem has some support for sockets, but nothing above that).
Basic need is to upload a "file" using POST.
I have all needed Header/Body in place and it "usually works".
Problem is I randomly get a "HTTP/1.1 502 Bad gateway error" response and this is more likely to happen as the size of "file" increases.
I understand this means there's some problem between the reverse proxy frontend (nginx, apparently) and the backends, but I have absolutely no control on those (actually I dont't really know how the atual setup besides what can be gleamed from (light) probing).
My current strategy is to open a plain socket and send the folowing sequence (dots represent binary data):
POST /path/to/websend.php HTTP/1.0
Host: host.domain.tld
User-Agent: Mozilla/5.0 (Windows NT 5.1; rv:33.0) Gecko/20100101 Firefox/33.0
Connection: Keep-Alive
Proxy-Connection: Keep-Alive
Content-Type: multipart/form-data; boundary=AaB03x
Cache-Control: no-store, no-cache, must-revalidate, post-check=0, pre-check=0
Pragma: no-cache
Accept: */*
Content-Length: <full_length>
--AaB03x
Content-Disposition: form-data; name="IV"
Content-Type: application/data
Content-Transfer-Encoding: binary
000102030405060708090A0B0C0D0E0F
--AaB03x
Content-Disposition: form-data; name="S_TXT_FILE"; filename="FILENAME_s.txt"
Content-Type: application/data
Content-Transfer-Encoding: binary
..............................................................
..............................................................
...... several 512byte blocks ................................
..............................................................
..............................................................
--AaB03x--
Is there something I could do to enhance reliability?
I already do multiple retries and this actually works, but sometimes I need to retry six or more times to have a positive answer (200 OK).
Note I send exactly the same sequence on rety and it succeeds... eventually.
I need to send two parts because content is encrypted and first part is the neded "Initialization Vector".

Related

Uploading file gets Bad Request from the server

I am trying to send a file using HTTP from a C++ application (no HTML-boxes). The server keeps answering Code 400/ Bad Request.
To keep it simple, I have changed manually the content of the file to a simple string (later on, I will need to upload real binary files).
the POST request is the following:
POST /post.php HTTP/1.0
Host: posttestserver.com
Accept: */*
Content-Type: multipart/form-data; boundary=BOUNDARY
--BOUNDARY
Content-Disposition: form-data; name="userfile"; filename="example.txt"
Content-Type:text/plain
123ABC
--BOUNDARY--
Connection: close
Any idea what is going on?

Linking directly to audio files

I've inherited a website that contains about 100 audio files. The links to the files are relative links like this:
part 1
Back in the day those usually forced a download. Newer browsers now play the audio in browser. Except....
If the user comes to the site over https they are able to navigate the site and the html pages load, but the links to the audio files generate a 403 Forbidden error. Changing the protocol in the location http allows the mp3 to load and playback in the browser.
Why would the mp3 files be forbidden over https?
Is there a way to force the http protocol without having to make all the links absolute links? I notice the relative links "inherit" the protocol of the page they were loaded on. There isn't anything on any of these pages that need https so I wouldn't mind forcing all the parent pages to load over http....
This is a departmental site within a giant university. So I don't have access to the server, htaccess, or any of those kinds of tricks. All in browser, javascript, html solutions please.
UPDATE
I installed Firebug to view the headers and discovered that the audio plays fine in FireFox (on my mac). In Safari they load and play, but the controls don't show the progress or time, but they do play. And in Chrome they don't play at all.
I had also checked them on my PC at work and they don't play in IE9 (I know! Corporate IT, right?) or Chrome.
Here is what I get for headers in Firefox where the audio plays fine.
HTTP/1.1 200 OK
Date: Sat, 11 Apr 2015 15:39:04 GMT
Server: Apache
WWW: www3
Vary: X-Forwarded-Proto
Last-Modified: Tue, 16 Nov 2010 14:19:25 GMT
Etag: "78e935-d60ac-4952c3e68d540"
Accept-Ranges: bytes
Content-Length: 876716
Keep-Alive: timeout=5, max=100
Connection: Keep-Alive
Content-Type: audio/mpeg
GET /dept/area/language/stories/sounds/file.mp3 HTTP/1.1
Host: example.edu
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.10; rv:33.0) Gecko/20100101 Firefox/33.0
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
Accept-Language: en-US,en;q=0.5
Accept-Encoding: gzip, deflate
Referer: https://example.edu/dept/area/language/stories.html
Cookie: _ga=GA1.2.829124232.1405280613; BIGipServerWWW-HTTP=1378527424.20480.0000; _gat=1
Connection: keep-alive
And these are what I get in Chrome.
Remote Address:128.122.119.202:443
Request URL:https://example.edu/dept/area/language/stories/sounds/file.mp3
Request Method:GET
Status Code:206 Partial Content
HTTP/1.1 206 Partial Content
Date: Sat, 11 Apr 2015 15:46:12 GMT
Server: Apache
WWW: www4
Vary: X-Forwarded-Proto
Last-Modified: Tue, 16 Nov 2010 14:19:12 GMT
ETag: "78e939-158dbc-4952c3da27800"
Accept-Ranges: bytes
Content-Length: 1
Content-Range: bytes 382271-382271/1412540
Keep-Alive: timeout=5, max=100
Connection: Keep-Alive
Content-Type: audio/mpeg
GET /dept/area/language/stories/sounds/file.mp3 HTTP/1.1
Host: www.nyu.edu
Connection: keep-alive
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_3) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/41.0.2272.118 Safari/537.36
DNT: 1
Referer: https://example.edu/dept/area/language/stories.html
Accept-Encoding: gzip, deflate, sdch
Accept-Language: en-US,en;q=0.8,es;q=0.6,hi;q=0.4,pt;q=0.2
Cookie: _ap_utmz=57748789.1416681263.3.1.utmccn=(direct)|utmcsr=(direct)|utmcmd=(none); _ap_utma=57748789.722895429.1387124094.1423327171.1425612794.7; __utma=57748789.194555315.1387124094.1423327171.1425612794.7; __utmz=57748789.1416681262.3.1.utmcsr=(direct)|utmccn=(direct)|utmcmd=(none); BIGipServerWWW-HTTP=1395304640.20480.0000; _gat=1; _ga=GA1.2.194555315.1387124094
Range: bytes=382271-382271
If-Range: "78e939-158dbc-4952c3da27800"
HTTP Request and Response Headers
Make sure to read about headers, mime types and content encodings.
You could try to utilise the Content-Disposition response header
An opportunity to raise a "File Download" dialogue box for a known MIME type with binary format or suggest a filename for dynamic content. Quotes are necessary with special characters.
Source: Wikipedia
Anyway your issue seems like a http header issue, could be compression as well. Take a look at your headers and whats different and troubleshoot from there. When you understood the problem, you can think of solutions.
Troubleshooting Tools
Use firebug or chrome developer tools to investigate. Fiddler Proxy to simulate different headers, since you have no access to your server.
File Permissions
Could be that SSL runs as another user or config on your server and the mp3 files have the wrong permissions or their parent directory. You need to check those, but since you have no server access you could be out of luck.
However, if SSL is not important to you just link to the files like so:
<a href="http://yourDomain.tld/folder/anotherFolder/file.mp3">
This will enforce the http protocol being used for the links. Most likely this results in the SSL chain being broken due to the mix in of http traffic into your ssl secured traffic. Therefore there's another alternative to achieve what you want:
Meta Refreshes
<meta http-equiv="refresh" content="3; URL=http://www.yourNonSSLDomain.tld/">
This will redirect to your non-SSL website where you can make sure to not mix https and http resources in your html document.

HTTP Request\Response Header Grammar

In the header of an HTTP request or response will the header keys be constant in terms of capitalization, between servers.
I ask so I can expect in my code: (Using Fake Function names)
Safe Precise Python Code
for hdr in header.keys():
if 'content-length' == hdr.lower():
recv_more_data( header[hdr] ) # header[hdr] == Content-Length (5388) bytes
break # Exit for loop when if statement is met.
Code I Would Like To Use
recv_more_data (header['Content-Length'])
# I know to expect 'Content-Length' not 'content-Length' or some other variation
Meaning will a server ever return a header with the keys like so.
Standard Request
GET / HTTP/1.1
Host: www.example-host.com
User-Agent: Mozilla/5.0 (Windows NT 6.2; WOW64; rv:33.0) Gecko/20100101 Firefox/33.0
Accept: */*
Accept-Language: en-US
Accept-Encoding: gzip
Connection: closed
Content-Length: 0
A Bad But Possible Response?
HTTP/1.1 200 OK
Server: nginx/1.0.15
date: Thu, 23 Oct 2014 00:25:37 GMT
content-Type: text/html; charset=iso-8859-1
transfer-encoding: chunked
Connection: close
Expires: Thu, 19 Nov 1981 08:52:00 GMT
Cache-control: no-store, no-cache, must-revalidate, post-check=0, pre-check=0
Pragma: no-cache
Content-Encoding: gzip
Clarification will help my code neatness.
HTTP header names are case-insensitive, per the HTTP specification.
RFC 2616 - Hypertext Transfer Protocol -- HTTP/1.1
Section 4.2 - Message Headers
HTTP header fields, which include general-header (section 4.5),
request-header (section 5.3), response-header (section 6.2), and
entity-header (section 7.1) fields, follow the same generic format as
that given in Section 3.1 of RFC 822 [9]. Each header field consists
of a name followed by a colon (":") and the field value. Field names
are case-insensitive. The field value MAY be preceded by any amount
of LWS, though a single SP is preferred. Header fields can be
extended over multiple lines by preceding each extra line with at
least one SP or HT. Applications ought to follow "common form", where
one is known or indicated, when generating HTTP constructs, since
there might exist some implementations that fail to accept anything
beyond the common forms.
RFC 7230 - Hypertext Transfer Protocol (HTTP/1.1): Message Syntax and Routing
Section 3.2 - Header Fields:
Each header field consists of a case-insensitive field name followed
by a colon (":"), optional leading whitespace, the field value, and
optional trailing whitespace.
HTTP header names are case insensitive.
It looks like you're using python. Check out the requests library. It'll make your life much easier: http://docs.python-requests.org/en/latest/
Bear in mind that, even though most major servers will have consistent capitalization, any Joe PHP Developer can set the response headers manually in their code - and there is no way to police what that guy uses as a capitalization standard.

HTTP Content length less than File byte-size, did it fully download?

Trying to determine if a user actually downloaded an executable file from a website. I examined the pcap and I see that the Content-Length field = 784,536 but the Server->User is 430,380 bytes. This tells me that the user did not fully download the file. I also downloaded the file myself and see that it is 766 KB. Is it possible that the content-length value based on the HTTP header will not be EQUAL TO the file size of that EXE file if it is downloaded (the local file size)? Is this correct?
Packet Capture Data (I can't post screenshots)
GET /ChromasLite211Setup.exe HTTP/1.1
Host: www.technelysium.com.au
User-Agent: Mozilla/5.0 (Windows NT 5.1; rv:17.0) Firefox/17.0
Accept: text/html, application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
Accept-Language: en-us
Accept-Enconding: gzip, deflate
Connection: keep-alive
Referrer: http://technelysium.com.au/
HTTP/1.1 200 OK
Date: Thu, 01 Aug 2013 17:28:17 GMT
Server: Apache
Last-Modified: Mon, 15 Apr 2013 08:29:57 GMT
Accept-Ranges: bytes
Content-Length: 784536
Keep-Alive: timeout=5, max=100
Connection: Keep-Alive
Content-Type: application/x-msdownload
MZP........................#.............................!..L..This program must be run under Win32
Entire Conversation (430722 bytes)
Users IP -> Server IP (342 bytes)
Server IP -> Users IP (430380)
When I download the file from the site it shows as, "Binary FIle (766 KB)"
Converting Bytes to Kilobytes
784,536/1024 = 766.14
No. The user did not download all the bytes.
If a server sends a Content-Length header, that is exactly how many bytes of content it intends to send as the HTTP Response Body. If less than that number was sent, then something happened (Client terminated connection, Client timed out, etc.)

Why is Connection: keep-alive still being specified in http headers (isn't it deprecated)?

According to "HTTP: The Definitive Guide", using
Connection: keep-alive
to specify a persistent connection is deprecated in HTTP/1.1, since HTTP/1.1 specifies that connections are persistent by default and must be closed manually by sending
Connection: close
Thus, my simple assumption is that "Connection: keep-alive" shouldn't really be used anymore. However, it still seems alive and well. For example, keep-alive is being returned in the following query:
curl -I https://foursquare.com
HTTP/1.1 200 OK
Server: nginx/0.8.52
Date: Thu, 11 Aug 2011 21:15:45 GMT
Content-Type: text/html; charset=utf-8
Connection: keep-alive
Expires: Thu, 11 Aug 2011 21:15:45 UTC
Set-Cookie: XSESSIONID=w19~kqtn4bpqmfq51p8qolstpk6ti;Path=/;Secure;HttpOnly
Set-Cookie: LOCATION=49.25::-123.13330078125::Hockeytown::CA;Path=/;Secure
Set-Cookie: bbhive=OQ32XATE0OQAEVCY0IVSWUDPQ1A2GT
Content-Length: 38815
Cache-Control: no-cache, private, no-store
Pragma: no-cache
My question is: Why is Connection: keep-alive still being specified in HTTP headers?
A corollary question is: Are there still (clients, servers, proxies, etc) that still only speak HTTP/1.0 and its variants, or are most such entities on HTTP/1.1 as of 2011?
Here are my working hypotheses:
1) HTTP/1.0 is no longer in use, b/c that was "many years" ago
2) Given (1), keep-alive shouldn't be used anymore, but is purely for vestigial reasons (that is, certain technologies haven't bothered to remove it, or keep it around as voodoo code, etc.)
If (1) is incorrect, and HTTP/1.0 is still in use, then sure it seems plausible to keep using keep-alive, despite follow-up questions on HTTP 1.0-1.1 interop.
Thanks in advance for any insights shared!
HTTP/1.0 have no headers like Connection, but there is many different implementation of HTTP/1.0 and HTTP/1.1.
so Connection: keep-alive is used 'Just in case'

Resources