Is there a way to transform something like this:
curl https://some.api.com/$batch \
-H "curl/7.9.8 (i686-pc-linux-gnu) libcurl 7.9.8 (OpenSSL 0.9.6b) (ipv6 enabled)" \
-d '{ foo: ["bar" "baz"]}'
into something like this (handmade, there could be mistakes):
POST /$batch HTTP/1.1
Host: some.api.com
Content-Type: application/json
Accept-Encoding: gzip
User-Agent: curl/7.9.8 (i686-pc-linux-gnu) libcurl 7.9.8 (OpenSSL 0.9.6b) (ipv6 enabled)
Content-Length: 21
{"foo":["bar","baz"]}
Ideally it'd be some command line option on curl, or I could live with some nc hack. Or is --trace-ascii my best friend?
Related
A user is able to upload a file. During the upload the file is scanned. If there is an issue with the file Symfony returns a Response(400) and the rest of the file is not uploaded, saving the user and the host time and bandwidth.
This is done via \Symfony\Component\HttpFoundation\Request::getContent(true)
$resource = $request->getContent(true);
The file is scanned a line at a time using:
fgets($resource);
The resource is also closed before the response is sent to the user:
fclose($resource);
However there is unexpected and strange behaviour happening for some user clients.
For example wget:
wget -4 --no-check-certificate --method PUT --timeout=0 --header 'Authorization: Bearer xxx' --body-file='xxx' 'https://example.com/xxx' --content-on-error -d -O -
Response hangs:
---request begin---
PUT /xxx HTTP/1.1
User-Agent: Wget/1.20.3 (linux-gnu)
Accept: */*
Accept-Encoding: identity
Host: xxx
Connection: Keep-Alive
Content-Length: 37767602
Authorization: Bearer xxx
---request end---
[writing BODY file xxx ...
It appears that wget does not understand the upload does not need to be completed, is this a header that php is failing to send or a flag required in the wget command?
A similar command in curl works
curl -k --location --request PUT 'https://example.com/xxx' \
--header 'Authorization: Bearer xxx' \
--data-binary '#/xxx'
Response
< Server: Apache/2.4.38 (Debian)
< Vary: Authorization
< X-Robots-Tag: noindex
< Transfer-Encoding: chunked
* HTTP error before end of send, stop sending
<
* Closing connection 0
* TLSv1.3 (OUT), TLS alert, close notify (256):
What I want to do is, is create special fuzz based request using curl for type 1 and another type 2 request which is based on 4 requests simultaneously and separately using curl, there is a command of perl, I also want that to include in the request
This is what I came up with.
My Curl Code Type 1:
curl -X GET -d '
Content-Length: 22
Foo: HTTP/1.1 200 OK
Content-Length: 0
Content-Type: text/html
Last-Modified: ...
Refresh: 0; URL=http://www.google.com/
Bar:
' -i $Local'
My Curl Code Type 2 for 4 requests:
#!/bin/bash
echo -e "[ + ] Enter Local Server
read Local
curl -X POST -d ' / HTTP/1.1
Host: google.com
Connection: Keep-Alive
Content-Length: 49225
perl -e 'print (A x 49255)
-i '$Local'
curl -X POST -d ' / HTTP/1.1
Host: google.com
Connection: Keep-Alive
Content-Length: 33
-i '$Local'
curl -X POST -d ' / HTTP/1.1
xxxx: POST /loader/cmd.exe?/c+dir HTTP/1.1
-i '$Local'
curl -X POST -d ' / HTTP/1.1
Connection: Keep-Alive
It didn't work, I tried lots of method, I am not familiar with this type of requests. I want to include exactly all this, what you're seeing.
Error was:
./curl.sh: line 2: unexpected EOF while looking for matching
./curl.sh: line 27: syntax error: unexpected end of file
This topic has been up quite some times in the community (forums, blog posts etc) and the conclusion is that this should be done making a REST Post call to share and the url /service/modules/create-site
The reason is that some surf specific stuff like the site dashboard are created from the share side.
However, I have been trying this approach from different angles all day, always ending up with a HTTP 200 in the response and no share site created. Quite frustrating.
I'm running this on Alfresco Enterprise 4.2.3.3 (I suspect my problems is due to a recent change)
To strip this down to something that is easy to reproduce, I'm following Martin Bergljungs blog post on the subject (http://www.ixxus.com/blog201203creating-alfresco-share-sites-javascript/), starting with using curl like this:
create a text file with login credentials (login.txt) with the following content (change to appropriate values):
username=admin&password=admin
create a text file with the json to create a site (site_data.json)
{"visibility" : "PUBLIC","title" : "My Test Site","shortName" : "mytestsite",
"description" : "My Test Site created from command line", "sitePreset" : "site-dashboard"}
Get the JSESSIONID by requesting a ticket:
curl -v -d #login.txt -H "Content-Type:application/x-www-form-urlencoded" http://localhost:8081/share/page/dologin
copy the resulting JSESSIONID value into the following curl call:
curl -v -d #site_data.json -H "Cookie:JSESSIONID=<insert your jsessionid>" -H "Content-Type:application/json" -H "Accept:application/json" http://localhost:8081/share/service/modules/create-site
output from curl:
* Hostname was NOT found in DNS cache
* Trying 127.0.0.1...
* Connected to localhost (127.0.0.1) port 8081 (#0)
> POST /share/service/modules/create-site HTTP/1.1
> User-Agent: curl/7.35.0
> Host: localhost:8081
> Cookie:JSESSIONID=5963B948684F562A278909AF466D2306
> Content-Type:application/json
> Accept:application/json
> Content-Length: 196
>
* upload completely sent off: 196 out of 196 bytes
< HTTP/1.1 200 OK
* Server Apache-Coyote/1.1 is not blacklisted
< Server: Apache-Coyote/1.1
< X-Frame-Options: SAMEORIGIN
< X-XSS-Protection: 1; mode=block
< X-Content-Type-Options: nosniff
< Content-Language: en-US
< Content-Length: 0
< Date: Tue, 02 Dec 2014 13:57:02 GMT
<
* Connection #0 to host localhost left intact
The latter curl call results in a HTTP 200 as seen above, but a login to share reveals there have been no site created what so ever :(
BTW. I have disabled the CSRF Token Filter.
UPDATE:
I have verified that the above approach works to create a site on Alfresco Enterprise 4.1.5
I have verified that it also fails on Alfresco Community 4.2.e
This is reported as a bug: https://issues.alfresco.com/jira/browse/MNT-11706
UPDATE: Since the question was not clear to a reader I have reformulated it now
UPDATE:
Following Dave Websters answer, I been trying again using the following steps, still with CSRF Token disabled:
Login:
curl -v -d #login.txt -H "Content-Type:application/x-www-form-urlencoded" http://localhost:8081/share/page/dologin
Response:
POST /share/page/dologin HTTP/1.1
User-Agent: curl/7.35.0
Host: localhost:8081
Accept: /
Content-Type:application/x-www-form-urlencoded
Content-Length: 29
* upload completely sent off: 29 out of 29 bytes
< HTTP/1.1 302 Found
* Server Apache-Coyote/1.1 is not blacklisted
< Server: Apache-Coyote/1.1
< X-Frame-Options: SAMEORIGIN
< X-XSS-Protection: 1; mode=block
< X-Content-Type-Options: nosniff
< Set-Cookie: JSESSIONID=058A52486E4EB12F94D1F95302732616; Path=/share/; HttpOnly
< Set-Cookie: alfLogin=1417618589; Expires=Wed, 10-Dec-2014 14:56:29 GMT; Path=/share
< Set-Cookie: alfUsername3=admin; Expires=Wed, 10-Dec-2014 14:56:29 GMT; Path=/share
< Location: http://localhost:8081/share
< Content-Length: 0
< Date: Wed, 03 Dec 2014 14:56:29 GMT
Took the cookie values and inserted into Daves code (with the csrf-stuff stripped out):
curl 'http://localhost:8081/share/service/modules/create-site' -H 'Cookie: JSESSIONID=058A52486E4EB12F94D1F95302732616; alfLogin=1417618589; alfUsername3=admin;' -H 'Origin: http://localhost:8081' -H 'Accept-Encoding: gzip,deflate,sdch' -H 'Accept-Language: en-US,en;q=0.8' -H 'User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/32.0.1700.107 Safari/537.36' -H 'Content-Type: application/json' -H 'Accept: */*' -H 'Referer: http://localhost:8081/share/page/site/erik/dashboard' -H 'X-Requested-With: application/json' -H 'Connection: keep-alive' --data-binary $'{"visibility":"PUBLIC","title":"erik'","shortName":"erik'","description":"This site is auto generated","sitePreset":"site-dashboard"}' --compressed
Still no share site generated though, and still a HTTP 200 Response. No errors in the logs either. This is driving me nuts :(
New Update (It works!):
I have now found out that you will need to "touch" a share webscript after making the login call before calling create-site with a post. I do this by making a get request in between. This somehow needs to be done to initialize the share session.
This is the curl command I use to generate sites programatically. I insert the JSESSIONID, LOGINCOOKIECONTENTS and CSRFTOKEN (twice) contents manually, but getting them programatically should work.
curl 'http://localhost:8081/share/service/modules/create-site' -H 'Cookie: JSESSIONID={JSESSIONID}; alfLogin={LOGINCOOKIECONTENTS}; alfUsername3=admin; Alfresco-CSRFToken={CSRFTOKEN};' -H 'Origin: http://localhost:8081' -H 'Accept-Encoding: gzip,deflate,sdch' -H 'Accept-Language: en-US,en;q=0.8' -H 'User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/32.0.1700.107 Safari/537.36' -H 'Content-Type: application/json' -H 'Accept: */*' -H 'Referer: http://localhost:8081/share/page/site/auto-gen-0/dashboard' -H 'X-Requested-With: application/json' -H 'Connection: keep-alive' -H 'Alfresco-CSRFToken: {CSRFTOKEN}' --data-binary $'{"visibility":"PUBLIC","title":"auto-gen'$I'","shortName":"auto-gen-'$I'","description":"This site is auto generated","sitePreset":"site-dashboard"}' --compressed
The expected response is:
{
"success": true
}
I am using both curl and wget to get this url: http://opinionator.blogs.nytimes.com/2012/01/19/118675/
For curl, it returns no output at all, but with wget, it returns the entire HTML source:
Here are the 2 commands. I've used the same user agent, and both are coming from the same IP, and are following redirects. The URL is exactly the same. For curl, it returns immediately after 1 second, so I know it's not a timeout issue.
curl -L -s "http://opinionator.blogs.nytimes.com/2012/01/19/118675/" --max-redirs 10000 --location --connect-timeout 20 -m 20 -A "Mozilla/5.0 (Windows NT 5.2; rv:2.0.1) Gecko/20100101 Firefox/4.0.1" 2>&1
wget http://opinionator.blogs.nytimes.com/2012/01/19/118675/ --user-agent="Mozilla/5.0 (Windows NT 5.2; rv:2.0.1) Gecko/20100101 Firefox/4.0.1"
If NY Times might be cloaking, and not returning the source to curl, what could be different in the headers curl is sending? I assumed since the user agent is the same, the request should look exactly the same from both of these requests. What other "footprints" should I check?
The way to solve is to analyze your curl request by doing curl -v ... and your wget request by doing wget -d ... which shows that curl is redirected to a login page
> GET /2012/01/19/118675/ HTTP/1.1
> User-Agent: Mozilla/5.0 (Windows NT 5.2; rv:2.0.1) Gecko/20100101 Firefox/4.0.1
> Host: opinionator.blogs.nytimes.com
> Accept: */*
>
< HTTP/1.1 303 See Other
< Date: Wed, 08 Jan 2014 03:23:06 GMT
* Server Apache is not blacklisted
< Server: Apache
< Location: http://www.nytimes.com/glogin?URI=http://opinionator.blogs.nytimes.com/2012/01/19/118675/&OQ=_rQ3D0&OP=1b5c69eQ2FCinbCQ5DzLCaaaCvLgqCPhKP
< Content-Length: 0
< Content-Type: text/plain; charset=UTF-8
followed by a loop of redirections (which you must have noticed, because you have already set the --max-redirs flag).
On the other hand, wget follows the same sequence except that it returns the cookie set by nytimes.com with its subsequent request(s)
---request begin---
GET /2012/01/19/118675/?_r=0 HTTP/1.1
User-Agent: Mozilla/5.0 (Windows NT 5.2; rv:2.0.1) Gecko/20100101 Firefox/4.0.1
Accept: */*
Host: opinionator.blogs.nytimes.com
Connection: Keep-Alive
Cookie: NYT-S=0MhLY3awSMyxXDXrmvxADeHDiNOMaMEZFGdeFz9JchiAIUFL2BEX5FWcV.Ynx4rkFI
The request sent by curl never includes the cookie.
The easiest way I see to modify your curl command and obtain the desired resource is by adding -c cookiefile to your curl command. This stores the cookie in the otherwise unused temporary "cookie jar" file called "cookiefile" thereby enabling curl to send the needed cookie(s) with its subsequent requests.
For example, I added the flag -c x directly after "curl " and I obtained the output just like from wget (except that wget writes it to a file and curl prints it on STDOUT).
In my case was because the https_proxy enviroment variable for utility cURL needs set the port in the URL, for example :
Not work with cURL :
https_proxy=http://proxyapp.net.com/
Works with cURL :
https_proxy=http://proxyapp.net.com:80/
With "wget" utility works with and without the port in url, but curl needs it, in case of not set the utility "curl" return error "(56) Proxy CONNECT aborted".
When you get verbosity of the command "curl -v" could see "curl" use port "1080" as default if port in not set at proxy url.
I'd like to pass a raw HTTP request like:
GET /foo/bar HTTP/1.1
Host: example.org
User-Agent: Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10.6; fr; rv:1.9.2.8) Gecko/20100722 Firefox/3.6.8
Accept: */*
Accept-Language: fr,fr-fr;q=0.8,en-us;q=0.5,en;q=0.3
Accept-Encoding: gzip,deflate
Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7
Keep-Alive: 115
Connection: keep-alive
Content-Type: application/x-www-form-urlencoded
X-Requested-With: XMLHttpRequest
Referer: http://example.org/test
Cookie: foo=bar; lorem=ipsum;
to a HTTP client.
I tried cat raw.http | curl but without success.
Any suggestion?
Thx.
Raw data in, raw data out:
nc example.org 80 < raw.http
If you need to pipe the data through some program:
cat raw.http | someprogram | nc example.org 80
Manual page
The question is tagged curl so I thought it was about time there was a curl answer
cat raw.http | curl "telnet://TARGETHOST:80"
For normal use just need to set the TARGETHOST to be the same as "host" header value.
For my purposes(not normal) I was hitting a TARGETHOST that was an ip address with a server that was listening for host headers of specific hosts.
Note that neither of these solutions would work if your need httpS instead of http. In this case you can send it this way:
$ cat raw.http | openssl s_client -connect server:443