I'm having the most bizarre issue with nginx.
After upgrading from 1.6.3 to 1.12.2 on RHEL 7.2, requests for audio files are just hanging:
Connecting to mydomain [...] ... connected.
HTTP request sent, awaiting response...
In my nginx access.log, I'm seeing a 200 status:
"GET /media/Example.mp3 HTTP/1.1" 200 105243 "-" "Wget/1.19.4 (linux-gnu)" "-"
If I request an MP4 file in the same directory, with the same permissions, it works just fine. I've tried MP4s that are both larger and smaller than my MP3 file: they work just fine.
CSS/JS/images also work fine.
If I comment out the MP3 mime type in /etc/nginx/mime.types, and then request /media/Example.mp3, it works just fine (!!!).
I added the ogg mime type to see if this was somehow related to just audio, and indeed, OGG files fail in the same way as MP3s.
I've set up debug logging, and everything looks normal for an MP3 request.
I've disabled SELinux, checked the permissions on the files, parent folders, etc. and confirmed that there is not a problem with the actual MP3 file.
I've tried turning sendfile off.
I can't undo this YUM transaction; it looks like there was a security issue with that version of nginx, and it is no longer available.
I've searched around online, but can't find any related reports. Does anyone have any thoughts/suggestions?
EDIT
When I set the Accept header and try to connect, curl output looks like:
* Trying my.ipaddress...
* TCP_NODELAY set
* Connected to my.host (my.ipaddress) port 80 (#0)
> GET /media/Example.mp3 HTTP/1.1
> Host: my.host
> User-Agent: curl/7.58.0
> Accept: audio/mpeg
>
And then it just hangs...
Related
I've written a REST API service that requires that all responses be JSON. However, when the Go HTTP request parser encounters an error, it returns 400 as a plain text response without ever calling my handlers. Example:
> curl -i -H 'Authorization: Basic hi
there' 'http://localhost:8080/test' -v
* Trying ::1...
* TCP_NODELAY set
* Connected to localhost (::1) port 8080 (#0)
> GET /test HTTP/1.1
> Host: localhost:8080
> User-Agent: curl/7.54.0
> Accept: */*
> Authorization: Basic hi
> there
>
< HTTP/1.1 400 Bad Request
HTTP/1.1 400 Bad Request
< Content-Type: text/plain; charset=utf-8
Content-Type: text/plain; charset=utf-8
< Connection: close
Connection: close
<
* Closing connection 0
Note the invalid Authorization header. Of course 400 is the proper response, but it's text/plain, of course. Is there some way to configure the Go http parser to use custom error response media types and bodies?
You can't. You can find this in net/http source, it only happens if the request was malformed:
https://github.com/golang/go/blob/master/src/net/http/server.go#L1744
I think your problem might be a new line in the header you're adding in curl?
401, 403, 404, 500 errors you'll be able to respond with json, but bad requests or bad headers (too long, malformed) are handled within server.go.
There is at present no way to intercept such errors though it is under consideration, so your only solution in go would be to patch the stdlib source (I don't recommend this). However, since this error only presents if the client has made a mistake and the request is malformed, it's probably not a huge problem. The reason for the text response is so that a browser or similar client (like curl without -v) doesn't just see an empty response. You could put a proxy like nginx in front of your app but then you'd never see the request either as it is a bad request, your proxy would handle it.
Possibly you'd be able to do it with a proxy like nginx in front though if you set a specific static error page for it to serve for 400 errors and serve a 400.json file that you specify? That's the only solution I can think of. A directive something like this might work for nginx:
error_page 400 /400.json;
If you'd like to be able to customise these errors, perhaps add a comment to the issue linked to let them know you had this specific problem.
If you are using the standard net/http library you can use the following code. Take a look at this answer Showing custom 404 error page with standard http package #Mostafa to which I got this example from
func homeHandler(w http.ResponseWriter, r *http.Request) {
if r.URL.Path != "/" {
errorHandler(w, r, http.StatusNotFound)
return
}
fmt.Fprint(w, "welcome home")
}
func errorHandler(w http.ResponseWriter, r *http.Request, status int) {
w.WriteHeader(status)
if status == http.StatusNotFound {
// JSON Out here
}
}
So, I have to retrieve temperature from any one of the cities from http://www.rssweather.com/dir/Asia/India.
Let's assume I want to retrieve of Kanpur's.
How to make an HTTP GET request with Netcat?
I'm doing something like this.
nc -v rssweather.com 80
GET http://www.rssweather.com/wx/in/kanpur/wx.php HTTP/1.1
I don't know exactly if I'm even in the right direction or not. I am not able to find any good tutorials on how to make an HTTP get request with netcat, so I'm posting it on here.
Of course you could dig in standards searched for google, but actually if you want to get only a single URL, it isn'tââââ worth the effort.
You could also start a netcat in listening mode on a port:
nc -l 64738
(Sometimes nc -l -p 64738 is the correct argument list)
...and then do a browser request into this port with a real browser. Just type in your browser http://localhost:64738 and see.
In your actual case the problem is that HTTP/1.1 doesn't close the connection automatically, but it waits your next URL you want to retrieve. The solution is simple:
Use HTTP/1.0:
GET /this/url/you/want/to/get HTTP/1.0
Host: www.rssweather.com
<empty line>
or use a Connection: request header to say the server you want to close after that:
GET /this/url/you/want/to/get HTTP/1.1
Host: www.rssweather.com
Connection: close
<empty line>
Extension: After the GET header write only the path part of the request. The hostname from which you want to get data belongs to a Host: header as you can see in my examples. This is because multiple websites can run on the same webserver, so the browsers need to say him, from which site it wants to load the page.
This works for me:
$ nc www.rssweather.com 80
GET /wx/in/kanpur/wx.php HTTP/1.0
Host: www.rssweather.com
And then hit double <enter>, i.e. once for the remote http server and once for the nc command.
source: pentesterlabs
You don't even need to use/install netcat
Create a tcp socket via an unused file-descriptor i.e I use 88 here
Write the request into it
use the fd
exec 88<>/dev/tcp/rssweather.com/80
echo -e "GET /dir/Asia/India HTTP/1.1\nhost: www.rssweather.com\nConnection: close\n\n" >&88
sed 's/<[^>]*>/ /g' <&88
On MacOS, you need the -c flag as follows:
Little-Net:~ minfrin$ nc -c rssweather.com 80
GET /wx/in/kanpur/wx.php HTTP/1.1
Host: rssweather.com
Connection: close
[empty line]
The response then appears as follows:
HTTP/1.1 200 OK
Date: Thu, 23 Aug 2018 13:20:49 GMT
Server: Apache
Connection: close
Transfer-Encoding: chunked
Content-Type: text/html
The -c flag is described as "Send CRLF as line-ending".
To be HTTP/1.1 compliant, you need the Host header, as well as the "Connection: close" if you want to disable keepalive.
Test it out locally with python3 http.server
This is also a fun way to test it out. On one shell, launch a local file server:
python3 -m http.server 8000
Then on the second shell, make a request:
printf 'GET / HTTP/1.1\r\nHost: localhost\r\n\r\n' | nc localhost 8000
The Host: header is required in HTTP 1.1.
This shows an HTML listing of the directory, just as you would see from:
firefox http://localhost:8000
Next you can try to list files and directories and observe the response:
printf 'GET /my-subdir/ HTTP/1.1\n\n' | nc localhost 8000
printf 'GET /my-file HTTP/1.1\n\n' | nc localhost 8000
Every time you make a successful request, the server prints:
127.0.0.1 - - [05/Oct/2018 11:20:55] "GET / HTTP/1.1" 200 -
confirming that it was received.
example.com
This IANA maintained domain is another good test URL:
printf 'GET / HTTP/1.1\r\nHost: example.com\r\n\r\n' | nc example.com 80
and compare with: http://example.com/
https SSL
nc does not seem to be able to handle https URLs. Instead, you can use:
sudo apt-get install nmap
printf 'GET / HTTP/1.1\r\nHost: github.com\r\n\r\n' | ncat --ssl github.com 443
See also: https://serverfault.com/questions/102032/connecting-to-https-with-netcat-nc/650189#650189
If you try nc, it just hangs:
printf 'GET / HTTP/1.1\r\nHost: github.com\r\n\r\n' | nc github.com 443
and trying port 80:
printf 'GET / HTTP/1.1\r\nHost: github.com\r\n\r\n' | nc github.com 443
just gives a redirect response to the https version:
HTTP/1.1 301 Moved Permanently
Content-Length: 0
Location: https://github.com/
Connection: keep-alive
Tested on Ubuntu 18.04.
To check whether it is a cli or http request, in PHP this method php_sapi_namecan be used, take a look here. I am trying to replicate that in apache conf file. The underlying idea is, if the request is coming from cli a 'minimal info' is served, if the request is from browsers then the users are redirected to different location. Is this possible?
MY PSEUDO CODE:
IF (REQUEST_COMING_FROM_CLI) {
ProxyPass / http://${IP_ADDR}:5000/
ProxyPassReverse / http://${IP_ADDR}:5000/
}ELSE IF(REQUEST_COMING_FROM_WEB_BROWSERS){
ProxyPass / http://${IP_ADDR}:8585/welcome/
ProxyPassReverse / http://${IP_ADDR}:8585/welcome/
}
Addition: cURL uses host of different protocols including http, ftp & telnet. Can apache figure out if the request is from cli or browser?
For as far as I know, there is no way to find the difference using apache.
if a request from the command-line is set up properly, apache can not make a difference between command-line and browser.
When you check it in PHP (using php_sapi_name, as you suggested), it only checks where php itself was called from (cli, apache, etc.), not where the http request came from.
using telnet for the command line, you can connect to apache, set the required http-headers and send the request as if you were using a browser(only, the browser sets the headers for you)
so, i do not think apache could differentiate between console or browser
The only way to do this is to test the user agent sent in the header of the request but this information can be easily changed.
By default every php http request looks like this to the apache server:
192.168.1.15 - - [01/Oct/2008:21:52:43 +1300] "GET / HTTP/1.0" 200 5194 "-" "-"
this information can be easily changed to look like a browser, for example using this
ini_set('user_agent',
'Mozilla/5.0 (Windows; U; Windows NT 6.0; en-GB; rv:1.9.0.3) Gecko/2008092417 Firefox/3.0.3');
the http request will look like this
192.168.1.15 - - [01/Oct/2008:21:54:29 +1300] "GET / HTTP/1.0" 200 5193
"-" "Mozilla/5.0 (Windows; U; Windows NT 6.0; en-GB; rv:1.9.0.3) Gecko/2008092417 Firefox/3.0.3"
At this moment the apache will think that the received connection come from a windows firefox 3.0.3.
So there is no a exact way to get this information.
You can use a BrowserMatch directive if the cli requests are not spoofing a real browser in the User-Agent header. Else, like everyone else has said, there is no way to tell the difference.
I know HTTP keep-alive is on by default in HTTP 1.1 but I want to find a way to confirm that it is actually working.
Does anyone know of a simple way to test from a web browser (EG how to make sense of wireshark). I know I need to look for multiple HTTP requests over the same TCP connection but I don't know how to confirm that in wireshark or any other way.
Thanks!
As Ron Garrity said on ServerFault, you can use Curl like this:
curl -Iv http://www.aptivate.org 2>&1 | grep -i 'connection #0'
And it outputs these two lines if keep-alive is working:
* Connection #0 to host www.aptivate.org left intact
* Closing connection #0
And if keep-alive is not working, then it just outputs this line:
* Closing connection #0
If you're on Windows Vista or later, you can use Resource Manager. The Network tab will list all open TCP connections and the process they were started by. Open a browser with one tab, browse to your page, and test.
First, try to capture the traffic to the target website in Wireshark and limit it to what you need with a filter like:
tcp port 80 and host targetwebsite.com
Then load the page in a browser or fetch it by any tool you have. If the target web page refreshes itself or one of the values in it, leave it open until you have at least one change in it.
Now you have enough data and you can stop capturing procedure in Wireshark.
You should see dozens of records and their protocol should be TCP or HTTP. For the purpose of your quick simple check, you will not need TCP records. So, lets remove them by applying another filter. In top of the window there is a "filter" field. Type http there, and wireshark will hide all records but those which have a HTTP protocol.
Now select a record and look at the next level of details, which you can find in the 2nd box bellow all records. Just to be sure you are looking at the right place, the first line there starts with "Frame XYZ". The fourth line starts with "Transmission Control Protocol". Look for the port numbers after "SRC Port" and "DST Port:". Depending on the record, one of these numbers belongs to the webserver (typically 80) and the other one shows port number in your end.
Now check a couple of different GET records. To know if the request is a GET record, check the Info column. If the port numbers in your end are used several times, all those requests were made through HTTP keepalive.
Remember that most browsers will open multiple connections, even if the webserver supports keepalive. So, DO NOT conclude your evaluation by finding just one different port.
The most accurate way is to curl the same URL multiple times.
curl -v http://weibo.com -o /dev/null http://weibo.com -o /dev/null
If the output contains Re-using existing connection, then the HTTP keep-alive feature is working. For example,
* TCP_NODELAY set
* Connected to weibo.com (180.149.138.251) port 80 (#0)
> GET / HTTP/1.1
> Host: weibo.com
> User-Agent: curl/7.68.0
> Accept: */*
>
* Mark bundle as not supporting multiuse
< HTTP/1.1 301 Moved Permanently
< ...
< ...
<
{ [236 bytes data]
* Connection #0 to host weibo.com left intact
* Found bundle for host weibo.com: 0x56324121d9a0 [serially]
* Can not multiplex, even if we wanted to!
* Re-using existing connection! (#0) with host weibo.com
* Connected to weibo.com (180.149.138.251) port 80 (#0)
> GET / HTTP/1.1
> Host: weibo.com
> User-Agent: curl/7.68.0
> Accept: */*
>
* Mark bundle as not supporting multiuse
< HTTP/1.1 301 Moved Permanently
< ...
< ...
<
{ [236 bytes data]
* Connection #0 to host weibo.com left intact
Another quick way is to test with ab. But some HTTP servers might not return the Connection: keep-alive header even when they've already turned on keep-alive feature, such as uwsgi. In such cases, ab does NOT send keep-alive requests. That makes ab can only do "positive" detection on HTTP keep-alive.
ab -c 5 -n 50 -k https://www.google.com/
If the result shows
...
Complete requests: 50
Failed requests: 0
Keep-Alive requests: 50 # Pay attention to this line
Total transferred:
...
Then the HTTP keep-alive is enabled.
I have implemented a HttpModule in ASP.NET (framework 2.0).
From my logging on the server, I can see that the http request is being picked up by my HttpModule, and my code runs successfully (writing content back to the Response stream).
However, the web browser (IE, FF, Chrome, all the same) just give me a "connection reset" error message.
I've checked the Event Log on the server; there are no related errors or messages there.
EXTRA INFO: When I use the "Live HTTP Headers" plugin in FF, it does not even display my client request (although I am sure it reaches the server, due to the request being captured in the server log).
How can I troubleshoot what's happening here?
Answering my own question here... I found out that there is a log file at C:\WINDOWS\system32\LogFiles\HTTPERR.
It shows the following:
2010-08-26 12:07:10 xx.xx.xx.xx 2618 xx.xx.xx.xx 80 HTTP/1.1 GET /web/myapp - 232667915 Connection_Dropped DefaultAppPool
2010-08-26 12:07:10 xx.xx.xx.xx 2624 xx.xx.xx.xx 80 HTTP/1.1 GET /web/myapp - 232667915 Connection_Dropped DefaultAppPool
2010-08-26 12:07:10 xx.xx.xx.xx 2625 xx.xx.xx.xx 80 HTTP/1.1 GET /web/myapp - 232667915 Connection_Dropped DefaultAppPool
After googling a bit I found this: http://davidovitz.blogspot.com/2007/08/iis-60-httperr-logs-show.html
And followed the advice:
"aspnet_regiis -u"
Uninstall IIS
Reboot
Reinstall IIS
"aspnet_regiis -i"
... reconfigure sites, and everything worked beautifully again. :-)