i try use sendgridr from this topic How I can add cc when sending email from R with SendGrid
and I have problem with HTTP error.
in the result I've got:
POST /v3/mail/send HTTP/1.1
Host: api.sendgrid.com
User-Agent: libcurl/7.64.1 r-curl/4.3.2 httr/1.4.2
Accept-Encoding: deflate, gzip
Accept: application/json, text/xml, application/xml, */*
Authorization: Bearer SG.Iu.............xxxx....
Content-Type: application/json
Content-Length: 470
>> {"personalizations":[{
>> "to": [ {"email": "mymail#mail.com"}],
>> "cc": [ {"email": "mymail#mail.com"}]
>> }],
>> "from": {"email": "mymail#mail.com"},
>> "subject": "Testing Sendgrid",
>> "content": [{"type": "text/plain", "value": "
>> Dear friend,
>>
>>
>>
>> I'm testing email.
>>
>>
>>
>> Kind regards,
>>
>> Mati"}]
>> }
HTTP/1.1 400 Bad Request
Server: nginx
Date: Wed, 29 Jun 2022 09:13:15 GMT
Content-Type: application/json
Content-Length: 63
Connection: keep-alive
Access-Control-Allow-Origin: https://sendgrid.api-docs.io
Access-Control-Allow-Methods: POST
Access-Control-Allow-Headers: Authorization, Content-Type, On-behalf-of, x-sg-elas-acl
Access-Control-Max-Age: 600
X-No-CORS-Reason: https://sendgrid.com/docs/Classroom/Basics/API/cors.html
Strict-Transport-Security: max-age=600; includeSubDomains
Kind regards
Mat
Related
Apache Tika should be accessible from Python program via HTTP, but I can't get it to work.
I am using this command to run the server (with and without the two options at the end):
java -jar tika-server-1.17.jar --port 5677 -enableUnsecureFeatures -enableFileUrl
And it works fine with curl:
curl -v -T /tmp/tmpsojwBN http://localhost:5677/tika
* Trying 127.0.0.1...
* Connected to localhost (127.0.0.1) port 5677 (#0)
> PUT /tika HTTP/1.1
> Host: localhost:5677
> User-Agent: curl/7.47.0
> Accept: */*
> Accept-Encoding: gzip, deflate
> Content-Length: 418074
> Expect: 100-continue
>
< HTTP/1.1 100 Continue
* We are completely uploaded and fine
< HTTP/1.1 200 OK
< Content-Type: text/plain
< Date: Sat, 07 Apr 2018 12:28:41 GMT
< Transfer-Encoding: chunked
< Server: Jetty(8.y.z-SNAPSHOT)
But when I try something like (tried different combinations for headers, here I recreated same headers as python-tika client uses):
with tempfile.NamedTemporaryFile() as tmp_file:
download_file(url, tmp_file)
payload = open(tmp_file.name, 'rb')
headers = {
'Accept': 'application/json',
'Content-Disposition': 'attachment; filename={}'.format(
os.path.basename(tmp_file.name))}
response = requests.put(TIKA_ENDPOINT_URL + '/tika', payload,
headers=headers,
verify=False)
I've tried to use payload as well as fileUrl - with the same result of WARN javax.ws.rs.ClientErrorException: HTTP 406 Not Acceptable and java stack trace on the server. Full trace:
WARN javax.ws.rs.ClientErrorException: HTTP 406 Not Acceptable
at org.apache.cxf.jaxrs.utils.SpecExceptions.toHttpException(SpecExceptions.java:117)
at org.apache.cxf.jaxrs.utils.ExceptionUtils.toHttpException(ExceptionUtils.java:173)
at org.apache.cxf.jaxrs.utils.JAXRSUtils.findTargetMethod(JAXRSUtils.java:542)
at org.apache.cxf.jaxrs.interceptor.JAXRSInInterceptor.processRequest(JAXRSInInterceptor.java:177)
at org.apache.cxf.jaxrs.interceptor.JAXRSInInterceptor.handleMessage(JAXRSInInterceptor.java:77)
at org.apache.cxf.phase.PhaseInterceptorChain.doIntercept(PhaseInterceptorChain.java:307)
at org.apache.cxf.transport.ChainInitiationObserver.onMessage(ChainInitiationObserver.java:121)
at org.apache.cxf.transport.http.AbstractHTTPDestination.invoke(AbstractHTTPDestination.java:274)
at org.apache.cxf.transport.http_jetty.JettyHTTPDestination.doService(JettyHTTPDestination.java:261)
at org.apache.cxf.transport.http_jetty.JettyHTTPHandler.handle(JettyHTTPHandler.java:76)
at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1088)
at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1024)
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:135)
at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:255)
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:116)
at org.eclipse.jetty.server.Server.handle(Server.java:370)
at org.eclipse.jetty.server.AbstractHttpConnection.handleRequest(AbstractHttpConnection.java:494)
at org.eclipse.jetty.server.AbstractHttpConnection.headerComplete(AbstractHttpConnection.java:973)
at org.eclipse.jetty.server.AbstractHttpConnection$RequestHandler.headerComplete(AbstractHttpConnection.java:1035)
at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:641)
at org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:231)
at org.eclipse.jetty.server.AsyncHttpConnection.handle(AsyncHttpConnection.java:82)
at org.eclipse.jetty.io.nio.SelectChannelEndPoint.handle(SelectChannelEndPoint.java:696)
at org.eclipse.jetty.io.nio.SelectChannelEndPoint$1.run(SelectChannelEndPoint.java:53)
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
at java.lang.Thread.run(Thread.java:748)
I've also tried to compare ( with nc -l localhost 5677 | less) what is so different with two requests (payload abbreviated):
From curl:
PUT /tika HTTP/1.1
Host: localhost:5677
User-Agent: curl/7.47.0
Accept: */*
Content-Length: 418074
Expect: 100-continue
%PDF-1.4
%<D3><EB><E9><E1>
1 0 obj
<</Creator (Chromium)
From Python requests library:
PUT /tika HTTP/1.1
Host: localhost:5677
Connection: keep-alive
Accept-Encoding: gzip, deflate
Accept: application/json
User-Agent: python-requests/2.13.0
Content-type: application/pdf
Content-Length: 246176
%PDF-1.4
%<D3><EB><E9><E1>
1 0 obj
<</Creator (Chromium)
The question is, what is the correct way to call Tika server from Python?
I've also tried python tika library in client-only mode and using tika-app via jnius. With tika client, as well as using tika-app.jar with pyjnius, I only freezes (call never returns) when I use them in a celery worker. At the same, pyjnius / tika-app and tika-python script both work nicely in a script: I have not figured out what is wrong inside celery worker. I guess, something to do with threading and/or initialization in wrong place. But that is a topic for another question.
And here is what tika-python requests:
PUT /tika HTTP/1.1
Host: localhost:5677
Connection: keep-alive
Accept-Encoding: gzip, deflate
Accept: application/json
User-Agent: python-requests/2.13.0
Content-Disposition: attachment; filename=tmpb3YkTq
Content-Length: 183234
%PDF-1.4
%<D3><EB><E9><E1>
1 0 obj
<</Creator (Chromium)
And now it seems like this is some kind of a problem with tika server:
$ tika-python --verbose --server 'localhost' --port 5677 parse all /tmp/tmpb3YkTq
2018-04-08 09:44:11,555 [MainThread ] [INFO ] Writing ./tmpb3YkTq_meta.json
(<open file '<stderr>', mode 'w' at 0x7f0b688eb1e0>, 'Request headers: ', {'Accept': 'application/json', 'Content-Disposition': 'attachment; filename=tmpb3YkTq'})
(<open file '<stderr>', mode 'w' at 0x7f0b688eb1e0>, 'Response headers: ', {'Date': 'Sun, 08 Apr 2018 06:44:13 GMT', 'Transfer-Encoding': 'chunked', 'Content-Type': 'application/json', 'Server': 'Jetty(8.y.z-SNAPSHOT)'})
['./tmpb3YkTq_meta.json']
Cf:
$ tika-python --verbose --server 'localhost' --port 5677 parse text /tmp/tmpb3YkTq
2018-04-08 09:43:38,326 [MainThread ] [INFO ] Writing ./tmpb3YkTq_meta.json
(<open file '<stderr>', mode 'w' at 0x7fc3eee4a1e0>, 'Request headers: ', {'Accept': 'application/json', 'Content-Disposition': 'attachment; filename=tmpb3YkTq'})
(<open file '<stderr>', mode 'w' at 0x7fc3eee4a1e0>, 'Response headers: ', {'Date': 'Sun, 08 Apr 2018 06:43:38 GMT', 'Content-Length': '0', 'Server': 'Jetty(8.y.z-SNAPSHOT)'})
2018-04-08 09:43:38,409 [MainThread ] [WARNI] Tika server returned status: 406
['./tmpb3YkTq_meta.json']
I'm trying to query data from the Materials Project web API in R.
The documentation provides an example query which is conducted using both curl and python. I've copied the curl command below.
curl -s --header "X-API-KEY: <YOUR-API-KEY>" \
https://materialsproject.org/rest/v2/query \
-F criteria='{"elements": {"$in": ["Li", "Na", "K"], "$all": ["O"]}, "nelements": 2}' \
-F properties='["formula", "formation_energy_per_atom"]'
From reading the httr quickstart guide, it seems to me I should be able to reproduce this query with:
library(httr)
POST(url = "https://www.materialsproject.org/rest/v2/query",
config = add_headers("X-API-KEY" = "<YOUR-API-KEY>",
body = list(criteria = "{'elements': {'$in': ['Li', 'Na', 'K'], '$all': ['O']}, 'nelements': 2}",
properties = "['formula', 'formation_energy_per_atom']"),
encode = "multipart",
verbose())
But while the curl command returns JSON data from the Materials Project database, my R query returns a HTTP/1.1 400 BAD REQUEST. What is curl doing differently than httr in the codes above?
I've tried putting -v on curl and comparing it to the (verbose()) output above, but curl don't show what it's putting in the multipart form.
> Expect: 100-continue
> Content-Type: multipart/form-data; boundary=------------------------d2ef2f3982185118
>
< HTTP/1.1 100 Continue
< HTTP/1.1 200 OK
< Date: Tue, 27 Dec 2016 21:18:58 GMT
< Server: Apache/2.2.15 (CentOS)
< Vary: Accept-Encoding,User-Agent
< Connection: close
< Transfer-Encoding: chunked
< Content-Type: application/json
Meanwhile httr shows:
-> Content-Type: multipart/form-data; boundary=----------------------------5b4873dbc9cd
->
<- HTTP/1.1 100 Continue
>> ------------------------------5b4873dbc9cd
>> Content-Disposition: form-data; name="criteria"
>>
>> {'elements': {'$in': ['Li', 'Na', 'K'], '$all': ['O']}, 'nelements': 2}
>> ------------------------------5b4873dbc9cd
>> Content-Disposition: form-data; name="properties"
>>
>> ['formula', 'formation_energy_per_atom']
>> ------------------------------5b4873dbc9cd--
It's truly a terrible, poorly thought out & lazily implemented API. They seem to like Python so it's unsurprising this would be the case.
The following works:
library(httr)
library(jsonlite)
list(
criteria=toJSON(list(
elements=list(
`$in`=c("Li", "Na", "K"),
`$all`=c("0")
),
nelements=unbox(2)
)),
properties=toJSON(c("formula", "formation_energy_per_atom"))
) -> params
POST(url="https://www.materialsproject.org/rest/v2/query",
add_headers(`X-API-KEY`=Sys.getenv("MATERIALS_PROJECT_API_KEY")),
body=params,
encode="multipart", verbose()) -> res
and here's the verbose() output to prove it:
-> POST /rest/v2/query HTTP/1.1
-> Host: www.materialsproject.org
-> User-Agent: libcurl/7.51.0 r-curl/2.3 httr/1.2.1
-> Accept-Encoding: gzip, deflate
-> Accept: application/json, text/xml, application/xml, */*
-> X-API-KEY: wouldntyouliketoknow
-> Content-Length: 344
-> Expect: 100-continue
-> Content-Type: multipart/form-data; boundary=------------------------34f08173ce0a7818
->
<- HTTP/1.1 100 Continue
>> --------------------------34f08173ce0a7818
>> Content-Disposition: form-data; name="criteria"
>>
>> {"elements":{"$in":["Li","Na","K"],"$all":["0"]},"nelements":2}
>> --------------------------34f08173ce0a7818
>> Content-Disposition: form-data; name="properties"
>>
>> ["formula","formation_energy_per_atom"]
>> --------------------------34f08173ce0a7818--
<- HTTP/1.1 200 OK
<- Date: Wed, 28 Dec 2016 02:08:08 GMT
<- Server: Apache/2.2.15 (CentOS)
<- Vary: Accept-Encoding,User-Agent
<- Content-Encoding: gzip
<- Content-Length: 258
<- Connection: close
<- Content-Type: application/json
<-
It's super picky about the query string structure. They really should have just accepted a JSON body and have been done with it. But half-REDACTED is the way of python folk.
Oh gosh, I just noticed it's a CentOS server supplying the replies. Yep. Those folks really do like pain.
when executing, with a key that has worked in the past but that I haven't used for a few weeks, the following cURL
curl -v -k -s -H "Content-Type: application/json" https://vision.googleapis.com/v1/images:annotate?key=MyKey --data-binary #a.json
where a.json is
{"requests": [{"image": {"content": "SUkqADwmAAD////8gYEoGct1VHdGU..."}, "features": [{"type": "TEXT_DETECTION", "maxResults": 1}]}]}
returns
* Trying 173.194.205.239...
* Connected to vision.googleapis.com (173.194.205.239) port 443 (#0)
* TLS 1.2 connection using TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256
* Server certificate: *.googleapis.com
* Server certificate: Google Internet Authority G2
* Server certificate: GeoTrust Global CA
> POST /v1/images:annotate?key=MyKey HTTP/1.1
> Host: vision.googleapis.com
> User-Agent: curl/7.43.0
> Accept: */*
> Content-Type: application/json
> Content-Length: 13454
> Expect: 100-continue
>
* Done waiting for 100-continue
* We are completely uploaded and fine
< HTTP/1.1 400 Bad Request
< Vary: X-Origin
< Vary: Referer
< Content-Type: application/json; charset=UTF-8
< Date: Wed, 10 Feb 2016 18:02:12 GMT
< Server: ESF
< Cache-Control: private
< X-XSS-Protection: 1; mode=block
< X-Frame-Options: SAMEORIGIN
< X-Content-Type-Options: nosniff
< Alternate-Protocol: 443:quic,p=1
< Alt-Svc: quic=":443"; ma=604800; v="30,29,28,27,26,25"
< Accept-Ranges: none
< Vary: Origin,Accept-Encoding
< Transfer-Encoding: chunked
<
{
"error": {
"code": 400,
"message": "Request Issue Failed.",
"status": "INVALID_ARGUMENT"
}
}
* Connection #0 to host vision.googleapis.com left intact
Version 1 of the Google Cloud Vision API (beta) does not permit TIFF [1]. Here is a list of the currently supported formats:
JPEG
PNG8
PNG24
GIF
Animated GIF (first frame only)
BMP
WEBP
RAW
ICO
[1] https://cloud.google.com/vision/docs/image-best-practices#image_types
Turns out this is because I was sending base64 encoded tif images. Works fine for PNGs. Pretty sure I was told this in the docs.
I have file with request data:
POST /exampleUrl HTTP/1.1
Host: localhost:8080
Connection: keep-alive
Content-Length: 57907
Content-Type: multipart/form-data; boundary=--
Accept: */*
---
Content-Type: application/octet-stream
Content-Disposition: form-data; name='wanted'; filename=NameOfFile
Content-Transfer-Encoding: binary
[DATA]
---
Content-Type: application/octet-stream
Content-Disposition: form-data; filename=ID1
Content-Transfer-Encoding: binary
[DATA]
-----
How can i send it through curl as post request?
in HTTP request and response paradigm :
Is it possible to use chunked transfer encoding with multipart HTTP messages?
If no Why?
Yes, it is possible. Why do you think it's not?
It looks like the entire request body is subject to chunking, and structuring (or providing alternative versions of) multipart data appears orthogonal to the chunking. I can see curl wrapping entire parts into chunks,
$ curl --trace-ascii - https://httpbin.org/post --form "field1=Bonjour." -H "Expect:" -H "User-Agent:" -H "Accept:" --http1.1
[...]
=> Send header, 149 bytes (0x95)
0000: POST /post HTTP/1.1
0015: Host: httpbin.org
0028: Content-Length: 149
003d: Content-Type: multipart/form-data; boundary=--------------------
007d: ----5cb7d87e27c5c1bf
0093:
== Info: TLSv1.2 (OUT), TLS header, Supplemental data (23):
=> Send SSL data, 5 bytes (0x5)
0000: .....
=> Send data, 149 bytes (0x95)
0000: --------------------------5cb7d87e27c5c1bf
002c: Content-Disposition: form-data; name="field1"
005b:
005d: Bonjour.
0067: --------------------------5cb7d87e27c5c1bf--
== Info: We are completely uploaded and fine
[...]
$ curl --trace-ascii - https://httpbin.org/post --form "field1=Bonjour." -H "Expect:" -H "User-Agent:" -H "Accept:" --http1.1 -H "Transfer-Encoding: chunked"
[...]
=> Send header, 156 bytes (0x9c)
0000: POST /post HTTP/1.1
0015: Host: httpbin.org
0028: Transfer-Encoding: chunked
0044: Content-Type: multipart/form-data; boundary=--------------------
0084: ----db8064ac2c1d04aa
009a:
== Info: TLSv1.2 (OUT), TLS header, Supplemental data (23):
=> Send SSL data, 5 bytes (0x5)
0000: .....
=> Send data, 155 bytes (0x9b)
0000: 95
0004: --------------------------db8064ac2c1d04aa
0030: Content-Disposition: form-data; name="field1"
005f:
0061: Bonjour.
006b: --------------------------db8064ac2c1d04aa--
0099:
== Info: Signaling end of chunked upload via terminating chunk.
[...]
=> Send data, 5 bytes (0x5)
0000: 0
0003:
[...]
It's possible to split the request body into arbitrary chunks.
$ cr=$'\r'; lf=$'\n'; delim="xyz:abc:foobar"; h="httpbin.org"; body="--${delim}${cr}${lf}Content-Disposition: form-data; name=\"field1\"${cr}${lf}${cr}${lf}Bonjour.${cr}${lf}--${delim}--${cr}${lf}"; cl=${#body}; { echo -ne "POST /post HTTP/1.1\r\nHost: ${h}\r\nContent-Type: multipart/form-data; boundary=${delim}\r\nContent-Length: ${cl}\r\n\r\n${body}"; sleep 2; } | tee /dev/stderr | openssl s_client -connect "${h}:443" -servername "${h}" -quiet -no_ign_eof
POST /post HTTP/1.1
Host: httpbin.org
Content-Type: multipart/form-data; boundary=xyz:abc:foobar
Content-Length: 97
--xyz:abc:foobar
Content-Disposition: form-data; name="field1"
Bonjour.
--xyz:abc:foobar--
depth=2 C = US, O = Amazon, CN = Amazon Root CA 1
verify return:1
depth=1 C = US, O = Amazon, OU = Server CA 1B, CN = Amazon
verify return:1
depth=0 CN = httpbin.org
verify return:1
HTTP/1.1 200 OK
Date: Thu, 27 Oct 2022 03:57:21 GMT
Content-Type: application/json
Content-Length: 388
Connection: keep-alive
Server: gunicorn/19.9.0
Access-Control-Allow-Origin: *
Access-Control-Allow-Credentials: true
{
"args": {},
"data": "",
"files": {},
"form": {
"field1": "Bonjour."
},
"headers": {
"Content-Length": "97",
"Content-Type": "multipart/form-data; boundary=xyz:abc:foobar",
"Host": "httpbin.org",
"X-Amzn-Trace-Id": "Root=1-635a01a1-569f3ccb69a8ad4d07704cc7"
},
"json": null,
"origin": "WWW.XXX.YYY.ZZZ",
"url": "https://httpbin.org/post"
}
DONE
$ cr=$'\r'; lf=$'\n'; delim="xyz:abc:foobar"; h="httpbin.org"; body="--${delim}${cr}${lf}Conte
nt-Disposition: form-data; name=\"field1\"${cr}${lf}${cr}${lf}Bonjour.${cr}${lf}--${delim}--${cr}${lf}"; chunked=""; whi
le true; do b="${body:0:32}"; body="${body:32}"; chunked+="$(printf "%x" ${#b})${cr}${lf}${b}${cr}${lf}"; (( ${#b} )) ||
break; done; cl=${#chunked}; { echo -ne "POST /post HTTP/1.1\r\nHost: ${h}\r\nContent-Type: multipart/form-data; bounda
ry=${delim}\r\nTransfer-Encoding: chunked\r\nContent-Length: ${cl}\r\n\r\n${chunked}"; sleep 2; } | tee /dev/stderr | op
enssl s_client -connect "${h}:443" -servername "${h}" -quiet -no_ign_eof
POST /post HTTP/1.1
Host: httpbin.org
Content-Type: multipart/form-data; boundary=xyz:abc:foobar
Transfer-Encoding: chunked
Content-Length: 125
20
--xyz:abc:foobar
Content-Dispos
20
ition: form-data; name="field1"
20
Bonjour.
--xyz:abc:foobar--
1
0
depth=2 C = US, O = Amazon, CN = Amazon Root CA 1
verify return:1
depth=1 C = US, O = Amazon, OU = Server CA 1B, CN = Amazon
verify return:1
depth=0 CN = httpbin.org
verify return:1
HTTP/1.1 200 OK
Date: Thu, 27 Oct 2022 04:07:47 GMT
Content-Type: application/json
Content-Length: 388
Connection: keep-alive
Server: gunicorn/19.9.0
Access-Control-Allow-Origin: *
Access-Control-Allow-Credentials: true
{
"args": {},
"data": "",
"files": {},
"form": {
"field1": "Bonjour."
},
"headers": {
"Content-Length": "97",
"Content-Type": "multipart/form-data; boundary=xyz:abc:foobar",
"Host": "httpbin.org",
"X-Amzn-Trace-Id": "Root=1-635a0412-6a407dba17a44c4b09c49a36"
},
"json": null,
"origin": "76.71.106.87",
"url": "https://httpbin.org/post"
}
DONE