I came to a stop to get curl to create a new thread in the Xenoforo forum software.
Httpfox gives me the following data to create a new thread:
Headers:
POST /forum/add-thread HTTP/1.1
Host forum.tld
User-Agent Mozilla/5.0 (Windows NT 6.1; rv:17.0) Gecko/20100101 Firefox/17.0
Accept application/json, text/javascript, */*; q=0.01
Accept-Language en-us,en;q=0.5
Accept-Encoding gzip, deflate
Connection keep-alive
Content-Type application/x-www-form-urlencoded; charset=UTF-8
X-Ajax-Referer http://forum.tld/forum/create-thread
X-Requested-With XMLHttpRequest
Referer http://forum.tld/forum/create-thread
Content-Length 17871
Cookie cookie stuff
Pragma no-cache
Cache-Control no-cache
Raw post Data:
title=title+of+the+thread&message=urlencoded+thread+message&_xfRelativeResolver=http%3A%2F%2Fforum.tld%2Fforum%2Fcreate-thread&watch_thread_state=1&poll%5Bquestion%5D=&poll%5Bresponses%5D%5B%5D=&poll%5Bresponses%5D%5B%5D=&_xfToken=atoken&_xfRequestUri=%2Fforum%2Fcreate-thread&_xfNoRedirect=1&_xfToken=atoken+again&_xfResponseType=json
I extract the token from another page with curl by login in or using cookies.
When I use this curl:
curl -b cookie.txt -L -A "Mozilla/4.0 (compatible; MSIE 5.01; Windows NT 5.0)" -e "http://forum.tld/forum/create-thread" --data-urlencode "title=$title" --data-urlencode "watch_thread_state=1" --data-urlencode "message=$message" --data-urlencode "_xfRelativeResolver=http://forum.tld/forum/create-thread" --data-urlencode "poll[question]=" --data-urlencode "poll[responses][]=" --data-urlencode "_xfToken=$token" --data-urlencode "_xfRequestUri=/forum/create-thread" --data-urlencode "_xfNoRedirect=1" --data-urlencode "_xfResponseType=json" "http://forum.tld/forum/add-thread"
It outputs:
{"error":{"message":"Please enter a valid message."}
The $message gets read from a text file encoded in iso-8859-1.
Any ideas? I'm kind of clueless right now.
Related
A user is able to upload a file. During the upload the file is scanned. If there is an issue with the file Symfony returns a Response(400) and the rest of the file is not uploaded, saving the user and the host time and bandwidth.
This is done via \Symfony\Component\HttpFoundation\Request::getContent(true)
$resource = $request->getContent(true);
The file is scanned a line at a time using:
fgets($resource);
The resource is also closed before the response is sent to the user:
fclose($resource);
However there is unexpected and strange behaviour happening for some user clients.
For example wget:
wget -4 --no-check-certificate --method PUT --timeout=0 --header 'Authorization: Bearer xxx' --body-file='xxx' 'https://example.com/xxx' --content-on-error -d -O -
Response hangs:
---request begin---
PUT /xxx HTTP/1.1
User-Agent: Wget/1.20.3 (linux-gnu)
Accept: */*
Accept-Encoding: identity
Host: xxx
Connection: Keep-Alive
Content-Length: 37767602
Authorization: Bearer xxx
---request end---
[writing BODY file xxx ...
It appears that wget does not understand the upload does not need to be completed, is this a header that php is failing to send or a flag required in the wget command?
A similar command in curl works
curl -k --location --request PUT 'https://example.com/xxx' \
--header 'Authorization: Bearer xxx' \
--data-binary '#/xxx'
Response
< Server: Apache/2.4.38 (Debian)
< Vary: Authorization
< X-Robots-Tag: noindex
< Transfer-Encoding: chunked
* HTTP error before end of send, stop sending
<
* Closing connection 0
* TLSv1.3 (OUT), TLS alert, close notify (256):
This request used to work but now gets a 403. I tried adding a user agent like in this answer but still no good: https://stackoverflow.com/a/38489588/2415706
This second answer further down says to find the referer header but I can't figure out where these response headers are: https://stackoverflow.com/a/56946001/2415706
import requests
headers = {
"user-agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/90.0.4430.93 Safari/537.36",
"referer": "https://www.ziprecruiter.com/Salaries/What-Is-the-Average-Programmer-Salary-by-State"
job_url = "https://ziprecruiter.com/Salaries/What-Is-the-Average-Programmer-Salary-by-State"
job_response = requests.get(job_url, headers=headers, timeout=10)
print(job_response)
This is what I see under Request Headers for the first tab after refreshing the page but there's too much stuff. I assume I only need one of these lines.
:authority: www.ziprecruiter.com
:method: GET
:path: /Salaries/What-Is-the-Average-Programmer-Salary-by-State
:scheme: https
accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9
accept-encoding: gzip, deflate, br
accept-language: en-US,en;q=0.9
cache-control: max-age=0
cookie: __cfduid=dea4372c39465cfa2422e97f84dea45fb1620355067; zva=100000000%3Bvid%3AYJSn-w3tCu9yJwJx; ziprecruiter_browser=99.31.211.77_1620355067_495865399; SAFESAVE_TOKEN=1a7e5e90-60de-494d-9af5-6efdab7ade45; zglobalid=b96f3b99-1bed-4b7c-a36f-37f2d16c99f4.62fd155f2bee.6094a7fb; ziprecruiter_session=66052203cea2bf6afa7e45cae7d1b0fe; experian_campaign_visited=1
sec-ch-ua: " Not A;Brand";v="99", "Chromium";v="90", "Google Chrome";v="90"
sec-ch-ua-mobile: ?0
sec-fetch-dest: document
sec-fetch-mode: navigate
sec-fetch-site: none
sec-fetch-user: ?1
upgrade-insecure-requests: 1
user-agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/90.0.4430.93 Safari/537.36
EDIT: looking at the other tabs, they have referer: "referer": "https://www.ziprecruiter.com/Salaries/What-Is-the-Average-Programmer-Salary-by-State" so I'm trying that now but it is still 403.
Using httpx package it seems to work with:
import httpx
url = 'https://ziprecruiter.com/Salaries/What-Is-the-Average-Programmer-Salary-by-State'
r = httpx.get(url)
print(r.text)
print(r.status_code)
print(r.http_version)
repl.it: https://replit.com/#bertrandmartel/ZipRecruiter
I may be wrong but I think that the server didn't like the TLS negociation for the requests library. It's weird since the above call is using HTTP1.1 in the request and with curl it only works with http2 and TLS1.3
Using a curl binary built with http2 and with openssl supporting TLS1.3, the following works:
docker run --rm curlimages/curl:7.76.1 \
--http2 --tlsv1.3 'https://ziprecruiter.com/Salaries/What-Is-the-Average-Programmer-Salary-by-State' \
-H 'user-agent: Mozilla' \
-s -o /dev/null -w "%{http_code}"
returns:
301
The following commands are failing:
forcing http1.1 and enforcing TLS 1.3
docker run --rm curlimages/curl:7.76.1 \
--http1.1 --tlsv1.3 'https://ziprecruiter.com/Salaries/What-Is-the-Average-Programmer-Salary-by-State' \
-H 'user-agent: Mozilla' \
-s -o /dev/null -w "%{http_code}"
Output: 403
forcing http2 and enforcing TLS 1.2:
docker run --rm curlimages/curl:7.76.1 \
--http2 --tlsv1.2 'https://ziprecruiter.com/Salaries/What-Is-the-Average-Programmer-Salary-by-State' \
-H 'user-agent: Mozilla' \
-s -o /dev/null -w "%{http_code}"
Output: 403
My guess is that it detects something in the TLS negociation but the check is different when there is both TLS1.3 and HTTP/2
Unfortunately, you can't check http/2 with requests/urlib since it's not supported
I am trying to use JSoup to parse content from URLs like https://www.tesco.com/groceries/en-GB/products/300595003
Jsoup.connect(url).get() simply times out, however I can access the website fine in the web browser.
Through trial and error, the simplest working curl command I found was:
curl 'https://www.tesco.com/groceries/en-GB/products/300595003' \
-H 'User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:76.0) Gecko/20100101 Firefox/76.0' \
-H 'Accept-Language: en-GB,en;q=0.5' --compressed
I am able to translate the User-Agent and Accept-Language into JSoup, however I still get timeouts. Is there an equivalent to the --compressed flag for Jsoup, because the curl command will not work without it?
To find out what --compressed option does try using curl with --verbose parameter. It will display full request headers.
Without --compressed:
> GET /groceries/en-GB/products/300595003 HTTP/2
> Host: www.tesco.com
> Accept: */*
> User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:76.0) Gecko/20100101
Firefox/76.0
> Accept-Language: en-GB,en;q=0.5
With --comppressed:
> GET /groceries/en-GB/products/300595003 HTTP/2
> Host: www.tesco.com
> Accept: */*
> Accept-Encoding: deflate, gzip
> User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:76.0) Gecko/20100101
Firefox/76.0
> Accept-Language: en-GB,en;q=0.5
The difference is new Accept-Encoding header so adding .header("Accept-Encoding", "deflate, gzip") should solve your problem.
By the way, for me both jsoup and curl are able to download page source without this header and without --compressed and I'm not getting timeouts, so there's a chance your requests are somehow limited by server for making too many requests.
EDIT:
It works for me using your original command with --http1.1 so there has to be a way to make it work for you as well. I'd start with using Chrome developer tools to take a look at what headers your browser sends and try to pass all of them using .header(...). You can also copy curl command to see all headers and simulate exactly what Chrome is sending:
Is there a way to transform something like this:
curl https://some.api.com/$batch \
-H "curl/7.9.8 (i686-pc-linux-gnu) libcurl 7.9.8 (OpenSSL 0.9.6b) (ipv6 enabled)" \
-d '{ foo: ["bar" "baz"]}'
into something like this (handmade, there could be mistakes):
POST /$batch HTTP/1.1
Host: some.api.com
Content-Type: application/json
Accept-Encoding: gzip
User-Agent: curl/7.9.8 (i686-pc-linux-gnu) libcurl 7.9.8 (OpenSSL 0.9.6b) (ipv6 enabled)
Content-Length: 21
{"foo":["bar","baz"]}
Ideally it'd be some command line option on curl, or I could live with some nc hack. Or is --trace-ascii my best friend?
This topic has been up quite some times in the community (forums, blog posts etc) and the conclusion is that this should be done making a REST Post call to share and the url /service/modules/create-site
The reason is that some surf specific stuff like the site dashboard are created from the share side.
However, I have been trying this approach from different angles all day, always ending up with a HTTP 200 in the response and no share site created. Quite frustrating.
I'm running this on Alfresco Enterprise 4.2.3.3 (I suspect my problems is due to a recent change)
To strip this down to something that is easy to reproduce, I'm following Martin Bergljungs blog post on the subject (http://www.ixxus.com/blog201203creating-alfresco-share-sites-javascript/), starting with using curl like this:
create a text file with login credentials (login.txt) with the following content (change to appropriate values):
username=admin&password=admin
create a text file with the json to create a site (site_data.json)
{"visibility" : "PUBLIC","title" : "My Test Site","shortName" : "mytestsite",
"description" : "My Test Site created from command line", "sitePreset" : "site-dashboard"}
Get the JSESSIONID by requesting a ticket:
curl -v -d #login.txt -H "Content-Type:application/x-www-form-urlencoded" http://localhost:8081/share/page/dologin
copy the resulting JSESSIONID value into the following curl call:
curl -v -d #site_data.json -H "Cookie:JSESSIONID=<insert your jsessionid>" -H "Content-Type:application/json" -H "Accept:application/json" http://localhost:8081/share/service/modules/create-site
output from curl:
* Hostname was NOT found in DNS cache
* Trying 127.0.0.1...
* Connected to localhost (127.0.0.1) port 8081 (#0)
> POST /share/service/modules/create-site HTTP/1.1
> User-Agent: curl/7.35.0
> Host: localhost:8081
> Cookie:JSESSIONID=5963B948684F562A278909AF466D2306
> Content-Type:application/json
> Accept:application/json
> Content-Length: 196
>
* upload completely sent off: 196 out of 196 bytes
< HTTP/1.1 200 OK
* Server Apache-Coyote/1.1 is not blacklisted
< Server: Apache-Coyote/1.1
< X-Frame-Options: SAMEORIGIN
< X-XSS-Protection: 1; mode=block
< X-Content-Type-Options: nosniff
< Content-Language: en-US
< Content-Length: 0
< Date: Tue, 02 Dec 2014 13:57:02 GMT
<
* Connection #0 to host localhost left intact
The latter curl call results in a HTTP 200 as seen above, but a login to share reveals there have been no site created what so ever :(
BTW. I have disabled the CSRF Token Filter.
UPDATE:
I have verified that the above approach works to create a site on Alfresco Enterprise 4.1.5
I have verified that it also fails on Alfresco Community 4.2.e
This is reported as a bug: https://issues.alfresco.com/jira/browse/MNT-11706
UPDATE: Since the question was not clear to a reader I have reformulated it now
UPDATE:
Following Dave Websters answer, I been trying again using the following steps, still with CSRF Token disabled:
Login:
curl -v -d #login.txt -H "Content-Type:application/x-www-form-urlencoded" http://localhost:8081/share/page/dologin
Response:
POST /share/page/dologin HTTP/1.1
User-Agent: curl/7.35.0
Host: localhost:8081
Accept: /
Content-Type:application/x-www-form-urlencoded
Content-Length: 29
* upload completely sent off: 29 out of 29 bytes
< HTTP/1.1 302 Found
* Server Apache-Coyote/1.1 is not blacklisted
< Server: Apache-Coyote/1.1
< X-Frame-Options: SAMEORIGIN
< X-XSS-Protection: 1; mode=block
< X-Content-Type-Options: nosniff
< Set-Cookie: JSESSIONID=058A52486E4EB12F94D1F95302732616; Path=/share/; HttpOnly
< Set-Cookie: alfLogin=1417618589; Expires=Wed, 10-Dec-2014 14:56:29 GMT; Path=/share
< Set-Cookie: alfUsername3=admin; Expires=Wed, 10-Dec-2014 14:56:29 GMT; Path=/share
< Location: http://localhost:8081/share
< Content-Length: 0
< Date: Wed, 03 Dec 2014 14:56:29 GMT
Took the cookie values and inserted into Daves code (with the csrf-stuff stripped out):
curl 'http://localhost:8081/share/service/modules/create-site' -H 'Cookie: JSESSIONID=058A52486E4EB12F94D1F95302732616; alfLogin=1417618589; alfUsername3=admin;' -H 'Origin: http://localhost:8081' -H 'Accept-Encoding: gzip,deflate,sdch' -H 'Accept-Language: en-US,en;q=0.8' -H 'User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/32.0.1700.107 Safari/537.36' -H 'Content-Type: application/json' -H 'Accept: */*' -H 'Referer: http://localhost:8081/share/page/site/erik/dashboard' -H 'X-Requested-With: application/json' -H 'Connection: keep-alive' --data-binary $'{"visibility":"PUBLIC","title":"erik'","shortName":"erik'","description":"This site is auto generated","sitePreset":"site-dashboard"}' --compressed
Still no share site generated though, and still a HTTP 200 Response. No errors in the logs either. This is driving me nuts :(
New Update (It works!):
I have now found out that you will need to "touch" a share webscript after making the login call before calling create-site with a post. I do this by making a get request in between. This somehow needs to be done to initialize the share session.
This is the curl command I use to generate sites programatically. I insert the JSESSIONID, LOGINCOOKIECONTENTS and CSRFTOKEN (twice) contents manually, but getting them programatically should work.
curl 'http://localhost:8081/share/service/modules/create-site' -H 'Cookie: JSESSIONID={JSESSIONID}; alfLogin={LOGINCOOKIECONTENTS}; alfUsername3=admin; Alfresco-CSRFToken={CSRFTOKEN};' -H 'Origin: http://localhost:8081' -H 'Accept-Encoding: gzip,deflate,sdch' -H 'Accept-Language: en-US,en;q=0.8' -H 'User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/32.0.1700.107 Safari/537.36' -H 'Content-Type: application/json' -H 'Accept: */*' -H 'Referer: http://localhost:8081/share/page/site/auto-gen-0/dashboard' -H 'X-Requested-With: application/json' -H 'Connection: keep-alive' -H 'Alfresco-CSRFToken: {CSRFTOKEN}' --data-binary $'{"visibility":"PUBLIC","title":"auto-gen'$I'","shortName":"auto-gen-'$I'","description":"This site is auto generated","sitePreset":"site-dashboard"}' --compressed
The expected response is:
{
"success": true
}