Converting cURL request to http.Request - http

This cURL command works how it should:
curl -i -v http://localhost:81/hallo
* Trying ::1...
* TCP_NODELAY set
* Connected to localhost (::1) port 81 (#0)
> GET /hallo HTTP/1.1
> Host: localhost:81
> User-Agent: curl/7.55.1
> Accept: */*
>
1 -1 0* Connection #0 to host localhost left intact
Now I tried to do the same http-request in my go-service like this:
request, err := http.NewRequest("GET", "http://localhost:81/" + url.QueryEscape("Hallo"), nil)
client := &http.Client{}
resp, err := client.Do(request)
If I run the go code (I tried it with a test) it only produces this error: I only get the error net/http: HTTP/1.x transport connection broken: malformed HTTP status code "-1".
(I initially tried http.Get(myurl). this produces the same http Request. the current code was generated by https://mholt.github.io/curl-to-go/ )
Can anyone help me to understand why this two request produce different results?

Sample request and response to a really server:
[#xxxx ~]# curl -v -i 10.103.118.178:40000/ready
* About to connect() to 10.103.118.178 port 40000 (#0)
* Trying 10.103.118.178...
* Connected to 10.103.118.178 (10.103.118.178) port 40000 (#0)
> GET /ready HTTP/1.1
> User-Agent: curl/7.29.0
> Host: 10.103.118.178:40000
> Accept: */*
>
< HTTP/1.1 200 OK
HTTP/1.1 200 OK
< date: Fri, 25 Dec 2020 01:43:05 GMT
date: Fri, 25 Dec 2020 01:43:05 GMT
< content-length: 2
content-length: 2
< content-type: text/plain; charset=utf-8
content-type: text/plain; charset=utf-8
<
* Connection #0 to host 10.103.118.178 left intact
ok
You use -i to tell curl to include headers in the output. However, I cannot see any valid HTTP response header in your curl output. So probably your server is malfunctioning and did not make a valid HTTP response. So the Go program correctly reports about this, that it cannot interpret it as a valid HTTP header. (It tried to interpret it as an HTTP response header, but in where a status code should appear, it found -1 which is not a valid HTTP response code)

Related

format of data for a http POST resquest when using the curl command

If I issue a curl command to a REST api then I get the response below.
curl -i http://10.4.0.22:8088/api/clients/TEST_1/8194/1/1
Response:
HTTP/1.1 200 OK
Content-Type: application/json
Content-Length: 51
Server: Jetty(9.1.z-SNAPSHOT)
{"status":"CONTENT","content":{"id":1,"value":"0"}}
My understanding that this is equivalent to a http GET request.
What I'm trying to do is update the value field and change the value to 1.
I think this requires a POST request using the -d flag.
My question is how do I know what format the data should be in the curl command?
I tried
curl -d "{"status":"CONTENT","content":{"id":1,"value":"1"}}" http://10.4.0.22:8088/api/clients/EST_1/8194/1/1
but I get this response.
{"status":"METHOD_NOT_ALLOWED"}
I think the way that I am specifying the json after te -d flag is incorrect?
The response METHOD_NOT_ALLOWED means the server doesn't allow POST method.
You can add a -v to get a verbose output of the response. Sometimes the server responds back in the message as to what HTTP Methods (or Verbs) it allows.
I tried your parameters on my application endpoint and it worked. see below
curl -d "{"status":"CONTENT","content":{"id":1,"value":"1"}}" http://sitename.azurewebsites.net/index.php -v
* Adding handle: conn: 0x2c257d0
* Adding handle: send: 0
* Adding handle: recv: 0
* Curl_addHandleToPipeline: length: 1
* - Conn 0 (0x2c257d0) send_pipe: 1, recv_pipe: 0
* About to connect() to sitename.azurewebsites.net port 80 (#0)
* Trying 104.211.224.252...
* Connected to sitename.azurewebsites.net (104.211.224.252) port 80 (#0)
> POST /index.php HTTP/1.1
> User-Agent: curl/7.33.0
> Host: sitename.azurewebsites.net
> Accept: */*
> Content-Length: 39
> Content-Type: application/x-www-form-urlencoded
>
* upload completely sent off: 39 out of 39 bytes
< HTTP/1.1 200 OK
< Content-Length: 0
< Content-Type: text/html
< X-Powered-By: PHP/5.4.45
< Date: Fri, 04 Aug 2017 14:24:08 GMT
You can also try to specify the Content-Type header in your request and set it to corresponding Mime value.
For CURL, you can use the -H with the value set to "Content-Type: application/json"
Have you seen this thread: How to POST JSON data with Curl from Terminal/Commandline to Test Spring REST?

POST from REST console perceived as GET by server

Im using both the chrome REST API console and Postman to send a post request to my server (running nginx and symfony2)
Its a very simple request, just simply posting to a URL with an empty body. If this request runs from another server via a HTTP request, it will register as POST. Trying to POST from the api consoles registers as GET in my nginx access logs and returns a 405 Method not allowed.
If I use curl I initially get a 301 Moved Permanently, so I have to use -L to follow redirects. Im not sure if this is standard Symfony or if it is effecting the request.
I've found some problems with the curl request, but am unsure how to resolve them.
$ curl -v -L -d "1EepG1a63X" xxx.io/api/convert_mov/
* Trying xx.76.9.82...
* Connected to xxx.io (xx.76.9.82) port 80 (#0)
> POST /api/convert_mov/ HTTP/1.1
> Host: xxx.io
> User-Agent: curl/7.43.0
> Accept: */*
> Content-Length: 10
> Content-Type: application/x-www-form-urlencoded
>
* upload completely sent off: 10 out of 10 bytes
< HTTP/1.1 301 Moved Permanently
< Server: nginx/1.6.2
< Date: Tue, 15 Sep 2015 09:00:43 GMT
< Content-Type: text/html
< Content-Length: 184
< Connection: keep-alive
< Location: https://xxx.io/api/convert_mov/
<
* Ignoring the response-body
* Connection #0 to host xxx.io left intact
* Issue another request to this URL: 'https://xxx.io/api/convert_mov/'
* Switch from POST to GET
* Found bundle for host xxx.io: 0x7fcad9c14e70
* Trying xx.76.9.82...
* Connected to xxx.io (xx.76.9.82) port 443 (#1)
* TLS 1.2 connection using TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384
* Server certificate: xxx.io
* Server certificate: DigiCert SHA2 Secure Server CA
* Server certificate: DigiCert Global Root CA
> GET /api/convert_mov/ HTTP/1.1
> Host: xxx.io
> User-Agent: curl/7.43.0
> Accept: */*
>
< HTTP/1.1 404 Not Found
< Server: nginx/1.6.2
< Content-Type: text/html; charset=UTF-8
< Transfer-Encoding: chunked
< Connection: keep-alive
< X-Powered-By: PHP/5.5.25
< Cache-Control: no-cache
< Date: Tue, 15 Sep 2015 09:00:43 GMT
If you look closer, you will see your request is with HTTP. Then your server sends a redirect to your HTTPS site. And a 301 redirect does not preserve the request method. You MUST issue all your requests correctly against HTTPS.

cURL not returning after receiving 404 response

I have a webserver I have made in C (school project)
It seems to handle 200 response codes fine. However when i try to use cURL to test 404, it doesn't return/exit.
For example:
$ curl localhost:5555/q.txt
...
If I use verbose.
$ curl -v localhost:5555/q.txt
* Trying 127.0.0.1...
* Connected to localhost (127.0.0.1) port 5555 (#0)
> GET /q.txt HTTP/1.1
> User-Agent: curl/7.40.0
> Host: localhost:5555
> Accept: */*
>
* HTTP 1.0, assume close after body
< HTTP/1.0 404 Not Found
< Date: Fri, 14 Aug 2015 11:16:32 AEST
< Connection: close
< Server: myserver/jnd
<
The full message I am sending back is:
HTTP/1.0 404 Not Found\r\nDate: Fri, 14 Aug 2015 11:18:10 AEST\r\nConnection: close\r\nServer: myserver/jnd\r\n\r\n
Am I missing something here? A new-line or other formatting?
It seems cURL will just wait until you close the connection.
So if I just close(fd) afterwards, it works.

Preflight Options check options in Azure?

I'm building a simple ServiceStack app and intending to host it on AzureWebSites. That's working fine. I need CORS to make the app work. In IIS Express and IIS 7.5 locally, this works fine - but not on Azure or AppHarbor.
The actual AJAX GET and POST requests work fine, the problem is the pre-flight OPTIONS checks return an empty response; no headers, nothing.
The code is on GitHub. You can see in the git commit history some things I've tried.
I've enabled Failed Request Logging- but that doesn't help. I'm not getting a 400+ series error code - I'm getting no response at all.
EDIT: Thanks to #paaschpa I've started looking at the problem from different machines. Added one more response from a Linux machine. It shows that cURL is getting back something odd (see below) causing it to output debugging as per this SO question:
Why is cURL returning "additional stuff not fine"?
When I test Azure from my Rackspace Linux machine, I get this:
kyleh#media:~$ curl -X curl -X OPTIONS -H "Origin: http://www.example.com" -H "Access-Control-Request-Method: POST" -H "Access-Control-Request-Headers: X-Requested-With" --verbose http://sstodo.azurewebsites.net/items
* About to connect() to sstodo.azurewebsites.net port 80 (#0)
* Trying 23.101.118.145...
* connected
* Connected to sstodo.azurewebsites.net (23.101.118.145) port 80 (#0)
> OPTIONS /items HTTP/1.1
> User-Agent: curl/7.21.0 (x86_64-pc-linux-gnu) libcurl/7.26.0 OpenSSL/1.0.1e zlib/1.2.3.4 libidn/1.25 libssh2/1.4.2 librtmp/2.3
> Host: sstodo.azurewebsites.net
> Accept: */*
> Origin: http://www.example.com
> Access-Control-Request-Method: POST
> Access-Control-Request-Headers: X-Requested-With
>
* additional stuff not fine transfer.c:1037: 0 0
* additional stuff not fine transfer.c:1037: 0 0
* additional stuff not fine transfer.c:1037: 0 0
* HTTP 1.1 or later with persistent connection, pipelining supported
< HTTP/1.1 200 OK
< Cache-Control: private
< Content-Length: 0
< Vary: Accept
< Server: Microsoft-IIS/8.0
< X-Powered-By: ServiceStack/4.036 Win32NT/.NET
< Access-Control-Allow-Origin: *
< Access-Control-Allow-Methods: GET, POST, PUT, DELETE, PATCH, OPTIONS
< Access-Control-Allow-Headers: Content-Type
< X-AspNet-Version: 4.0.30319
< X-Powered-By: ASP.NET
< Set-Cookie: ARRAffinity=f1d67e2939c9eab291aa7a92c2c5cffe872dc89340409c771374fbf6bc961bd3;Path=/;Domain=sstodo.azurewebsites.net
< Date: Sun, 01 Feb 2015 02:57:55 GMT
<
* Connection #0 to host sstodo.azurewebsites.net left intact
* Closing connection #0
When I test the site locally, the result looks right. Here's the local result (VS / IIS Express):
curl -X OPTIONS
-H "Origin: http://www.example.com"
-H "Access-Control-Request-Method: POST"
-H "Access-Control-Request-Headers: X-Requested-With"
--verbose http://localhost:1061/items
* timeout on name lookup is not supported
* About to connect() to localhost port 1061 (#0)
* Trying 127.0.0.1...
* connected
* Connected to localhost (127.0.0.1) port 1061 (#0)
> OPTIONS /items HTTP/1.1
> User-Agent: curl/7.26.0
> Host: localhost:1061
> Accept: */*
> Origin: http://www.example.com
> Access-Control-Request-Method: POST
> Access-Control-Request-Headers: X-Requested-With
>
< HTTP/1.1 200 OK
< Cache-Control: private
< Vary: Accept
< Server: Microsoft-IIS/8.0
< X-Powered-By: ServiceStack/4.036 Win32NT/.NET
< Access-Control-Allow-Origin: *
< Access-Control-Allow-Methods: GET, POST, PUT, DELETE, PATCH, OPTIONS
< Access-Control-Allow-Headers: Content-Type
< X-AspNet-Version: 4.0.30319
< X-SourceFiles: =?UTF-8?B?QzpccHJvamVjdHNcc2VydmljZXN0YWNrLXRvZG8tYmFja2VuZFxUb0RvQmFja2VuZFxUb0RvQmFja2VuZFxpdGVtcw==?=
< X-Powered-By: ASP.NET
< Date: Thu, 22 Jan 2015 23:51:29 GMT
< Content-Length: 0
<
* Connection #0 to host localhost left intact
* Closing connection #0
Here's the result in IIS7.5 locally:
curl -H "Origin: http://www.example.com" -H "Access-Control-Request-Method: P
OST" -H "Access-Control-Request-Headers: X-Requested-With" -X OPTIONS --verbose
http://localhost/sstodo/items
* timeout on name lookup is not supported
* About to connect() to localhost port 80 (#0)
* Trying 127.0.0.1...
* connected
* Connected to localhost (127.0.0.1) port 80 (#0)
> OPTIONS /sstodo/items HTTP/1.1
> User-Agent: curl/7.26.0
> Host: localhost
> Accept: */*
> Origin: http://www.example.com
> Access-Control-Request-Method: POST
> Access-Control-Request-Headers: X-Requested-With
>
< HTTP/1.1 200 OK
< Cache-Control: private
< Vary: Accept
< Server: Microsoft-IIS/7.5
< X-Powered-By: ServiceStack/4.036 Win32NT/.NET
< Access-Control-Allow-Origin: *
< Access-Control-Allow-Methods: GET, POST, PUT, DELETE, PATCH, OPTIONS
< Access-Control-Allow-Headers: Content-Type
< X-AspNet-Version: 4.0.30319
< X-Powered-By: ASP.NET
< Date: Sat, 24 Jan 2015 02:44:55 GMT
< Content-Length: 0
<
* Connection #0 to host localhost left intact
* Closing connection #0
When I run the test against AzureWebSites (and AppHarbor) it does not.
curl -X OPTIONS
-H "Origin: http://www.example.com"
-H "Access-Control-Request-Method: POST"
-H "Access-Control-Request-Headers: X-Requested-With"
--verbose http://sstodo.azurewebsites.net/items
* timeout on name lookup is not supported
* About to connect() to sstodo.azurewebsites.net port 80 (#0)
* Trying 23.101.118.145...
* connected
* Connected to sstodo.azurewebsites.net (23.101.118.145) port 80 (#0)
> OPTIONS /items HTTP/1.1
> User-Agent: curl/7.26.0
> Host: sstodo.azurewebsites.net
> Accept: */*
> Origin: http://www.example.com
> Access-Control-Request-Method: POST
> Access-Control-Request-Headers: X-Requested-With
>
* Empty reply from server
* Connection #0 to host sstodo.azurewebsites.net left intact
curl: (52) Empty reply from server
* Closing connection #0
Here's AppHarbor:
curl -X OPTIONS
-H "Origin: http://www.example.com"
-H "Access-Control-Request-Method: POST"
-H "Access-Control-Request-Headers: X-Requested-With"
--verbose http://sstodo.apphb.com/items
* timeout on name lookup is not supported
* About to connect() to sstodo.apphb.com port 80 (#0)
* Trying 50.17.211.206...
* connected
* Connected to sstodo.apphb.com (50.17.211.206) port 80 (#0)
> OPTIONS /items HTTP/1.1
> User-Agent: curl/7.26.0
> Host: sstodo.apphb.com
> Accept: */*
> Origin: http://www.example.com
> Access-Control-Request-Method: POST
> Access-Control-Request-Headers: X-Requested-With
>
* Empty reply from server
* Connection #0 to host sstodo.apphb.com left intact
curl: (52) Empty reply from server
* Closing connection #0
I'm at a bit of a loss for direction. In my mental model, I should be able to repro the issue on my local environment. All the documentation I see for Azure and AppHarbor both indicate that they're not blocking the OPTIONS calls, but that doesn't appear to be the case.
I don't care so much about how the site works in cURL of course.
I'm betting you ran into the same issue I did; the "Web Security" module in the Cisco VPN client running on your OS X laptop was intercepting your HTTP requests and silently dropping OPTIONS requests made as part of the CORS preflight. What a great "feature" for this VPN software. It even does this when you're not using the VPN.
Switching to https fixes this because the proxy is not able to read your requests and so not able to drop OPTIONS.
More info about this problem here: http://www.bennadel.com/blog/2559-cisco-anyconnect-vpn-client-may-block-cors-ajax-options-requests.htm
You can uninstall the web security module with:
sudo /opt/cisco/anyconnect/bin/websecurity_uninstall.sh
Well...not sure this helps but I have a 'works from my machine' for you. Screenshot below...
If you fixed your own issue, I'd be curious to see the fix.
In the end, what worked was switching to HTTPS.
I can't say that I understand why, but using HTTPS always resolves the issue.

cURL receives empty body response from Nginx server

I try to fetch HTTP content with cURL, but I only get an empty body in the reply:
[root#www ~]# curl -v http://www.existingdomain.com/
* About to connect() to www.existingdomain.com port 80 (#0)
* Trying 95.211.256.257... connected
* Connected to www.existingdomain.com (95.211.256.257) port 80 (#0)
> GET / HTTP/1.1
> User-Agent: curl/7.21.0 (x86_64-redhat-linux-gnu) libcurl/7.21.0 NSS/3.12.8.0 zlib/1.2.5 libidn/1.18 libssh2/1.2.4
> Host: www.existingdomain.com
> Accept: */*
>
< HTTP/1.1 200 OK
< Server: nginx/0.8.53
< Date: Sat, 28 May 2011 15:56:23 GMT
< Content-Type: text/html
< Transfer-Encoding: chunked
< Connection: keep-alive
< Vary: Accept-Encoding
< X-Powered-By: PHP/5.3.3-0.dotdeb.1
<
* Connection #0 to host www.existingdomain.com left intact
* Closing connection #0
If I change the URL to another domain, like www.google.com, I get the content.
How can this be possible? And how to fetch content?
The server is free to send to the client whatever he likes, including nothing. While this is not exactly nice, there's little the client can do about this. You could
check the server logs to see if there is some problem which makes him so calm (given the server is under your control) or
try another client to see if the server does not like to talk to curl. You can then configure curl to mimic a regular web browser, if that helps

Resources