Upload file using Requests Library in Robot Framework - python-requests

I'm trying to upload a PDF file through Requests Library in Robot Framework, but the file doesn't get upload.
Whereas, when I tried the same in Insomnia it was work well (echo complete after successful upload).
Test case -
*** Variables ***
&{headers_file} X-Requested-With=XMLHttpRequest
*** Test Case ***
&{file} Create Dictionary file ${CURDIR}/test_certificate_report.pdf flowChunkNumber 1 flowChunkSize 1048576 flowCurrentChunkSize 25020 flowTotalSize 25020 flowIdentifier 25020-test_certificate_reportpdf flowFilename test_certificate_report.pdf flowRelativePath
... test_certificate_report.pdf flowTotalChunks 1
${test_uri} Set Variable /certificates/upload/7d5ebc7a-9ec4-444a-b2c1-f59e03999e58
${response} POST On Session api_reviewer_second ${test_uri} files=&{file} headers=&{headers_file}
Output Log-
POST Request : url=https://api.company.net/certificates/upload/7d5ebc7a-9ec4-444a-b2c1-f59e00488e58
path_url=/certificates/upload/7d5ebc7a-9ec4-444a-b2c1-f59e03999e58
headers={'User-Agent': 'python-requests/2.25.1', 'Accept-Encoding': 'gzip, deflate', 'Accept': '*/*', 'Connection': 'keep-alive', 'X-Requested-With': 'XMLHttpRequest', 'Cookie': 'PHPSESSID=guk1a2tlmpij9l3re9ge01lva5', 'Content-Length': '1303', 'Content-Type': 'multipart/form-data; boundary=f6820b677ad526bf76b0258616c3f9e4'}
body=b'--f6820b677ad526bf76b0258616c3f9e4\r\nContent-Disposition: form-data; name="file"; filename="file"\r\n\r\nC:\\RobotFramework\\bDATA_Test/support_files/test_certificate_report.pdf\r\n--f6820b677ad526bf76b0258616c3f9e4\r\nContent-Disposition: form-data; name="flowChunkNumber"; filename="flowChunkNumber"\r\n\r\n1\r\n--f6820b677ad526bf76b0258616c3f9e4\r\nContent-Disposition: form-data; name="flowChunkSize"; filename="flowChunkSize"\r\n\r\n1048576\r\n--f6820b677ad526bf76b0258616c3f9e4\r\nContent-Disposition: form-data; name="flowCurrentChunkSize"; filename="flowCurrentChunkSize"\r\n\r\n25020\r\n--f6820b677ad526bf76b0258616c3f9e4\r\nContent-Disposition: form-data; name="flowTotalSize"; filename="flowTotalSize"\r\n\r\n25020\r\n--f6820b677ad526bf76b0258616c3f9e4\r\nContent-Disposition: form-data; name="flowIdentifier"; filename="flowIdentifier"\r\n\r\n25020-test_certificate_reportpdf\r\n--f6820b677ad526bf76b0258616c3f9e4\r\nContent-Disposition: form-data; name="flowFilename"; filename="flowFilename"\r\n\r\ntest_certificate_report.pdf\r\n--f6820b677ad526bf76b0258616c3f9e4\r\nContent-Disposition: form-data; name="flowRelativePath"; filename="flowRelativePath"\r\n\r\ntest_certificate_report.pdf\r\n--f6820b677ad526bf76b0258616c3f9e4\r\nContent-Disposition: form-data; name="flowTotalChunks"; filename="flowTotalChunks"\r\n\r\n1\r\n--f6820b677ad526bf76b0258616c3f9e4--\r\n'
POST Response : url=https://api.company.net/certificates/upload/7d5ebc7a-9ec4-444a-b2c1-f59e03999e58
status=200, reason=OK
headers={'Date': 'Thu, 15 Apr 2021 09:19:08 GMT', 'Content-Type': 'application/json; charset=UTF-8', 'Transfer-Encoding': 'chunked', 'Connection': 'keep-alive', 'Server': 'nginx', 'X-Powered-By': 'PHP/7.2.34', 'Expires': 'Thu, 19 Nov 1981 08:52:00 GMT', 'Cache-Control': 'no-store, no-cache, must-revalidate', 'Pragma': 'no-cache', 'Access-Control-Allow-Credentials': 'true', 'Access-Control-Allow-Headers': 'Origin, X-Requested-With, Content-Range, Content-Disposition, Content-Type, Authorization', 'Access-Control-Allow-Methods': 'GET, PATCH, PUT, POST, DELETE, OPTIONS', 'Access-Control-Allow-Origin': 'https://app.company.net', 'X-Content-Type-Options': 'nosniff', 'Content-Security-Policy': "default-src 'self' *.company.net; img-src *", 'Strict-Transport-Security': 'max-age=31536000;'}
body=None
Can anyone please help me to identify the issue?

Replace with these lines
${file}= Get File For Streaming Upload ${CURDIR}/test_certificate_report.pdf
${response}= POST On Session api_reviewer_second ${test_uri} data=${file} headers=&{headers_file}

Related

How to perform a multipart upload with Google Drive REST API v3

I'm trying to make an https request to upload a file by following the guide https://developers.google.com/drive/api/v3/manage-uploads#multipart . This is what I try to send:
POST https://www.googleapis.com/upload/drive/v3/files?uploadType=multipart&key=AIz...Q4 HTTP/1.1\r\n
Authorization: Bearer ya...8A\r\n
Content-Type: multipart/related; boundary=foo_bar_baz\r\n
Content-Length: 150\r\n
\r\n
--foo_bar_baz\n
Content-Type: application/json; charset=UTF-8\n
\n
{\n
"title": "My File.jpg"\n
}\n
\n
--foo_bar_baz\n
Content-Type: text/txt\n
\n
JPEG data\n
--foo_bar_baz--\r\n
Could you tell me what I'm wrong or a working example? Thank you
UPDATE:
Ok, after several attempts I was able to find a correct https request, for example:
POST https://www.googleapis.com/upload/drive/v3/files?uploadType=multipart&key=AI...qeQ4 HTTP/1.1\r\n
Authorization: Bearer yaSN...Q9Q\r\n
Content-Type: multipart/related; boundary=foo_bar_baz\r\n
Content-Length: 167\r\n
\r\n
--foo_bar_baz\n
Content-Type: application/json; charset=UTF-8\n
\n
{\n
"name": "My File.txt"\n
}\n
--foo_bar_baz\n
Content-Type: text/plain; charset=UTF-8\n
\n
aaaaa\n
--foo_bar_baz--
with reply:
HTTP/1.1 200 OK\r\n
X-GUploader-UploadID: ABg...lRCA\r\n
Vary: Origin\r\n
Vary: X-Origin\r\n
Content-Type: application/json; charset=UTF-8\r\n
Cache-Control: no-cache, no-store, max-age=0, must-revalidate\r\n
Pragma: no-cache\r\n
Expires: Mon, 01 Jan 1990 00:00:00 GMT\r\n
Date: Thu, 11 Feb 2021 17:37:39 GMT\r\n
Content-Length: 121\r\n
Server: UploadServer\r\n
Alt-Svc: h3-29=":443"; ma=2592000,h3-T051=":443"; ma=2592000,h3-Q050=":443"; ma=2592000,h3-Q046=":443"; ma=2592000,h3-Q043=":443"; ma=2592000,quic=":443"; ma=2592000; v="46,43"\r\n
\r\n
{\n
"kind": "drive#file",\n
"id": "1_SE...8X",\n
"name": "My File.txt",\n
"mimeType": "text/plain"\n
}\n
I need to set the correct body length, otherwise the server returns an error.
The user UPDATE is the answer
Ok, after several attempts I was able to find a correct https request, for example:
POST https://www.googleapis.com/upload/drive/v3/files?uploadType=multipart&key=AI...qeQ4 HTTP/1.1\r\n
Authorization: Bearer yaSN...Q9Q\r\n
Content-Type: multipart/related; boundary=foo_bar_baz\r\n
Content-Length: 167\r\n
\r\n
--foo_bar_baz\n
Content-Type: application/json; charset=UTF-8\n
\n
{\n
"name": "My File.txt"\n
}\n
--foo_bar_baz\n
Content-Type: text/plain; charset=UTF-8\n
\n
aaaaa\n
--foo_bar_baz--
with reply:
HTTP/1.1 200 OK\r\n
X-GUploader-UploadID: ABg...lRCA\r\n
Vary: Origin\r\n
Vary: X-Origin\r\n
Content-Type: application/json; charset=UTF-8\r\n
Cache-Control: no-cache, no-store, max-age=0, must-revalidate\r\n
Pragma: no-cache\r\n
Expires: Mon, 01 Jan 1990 00:00:00 GMT\r\n
Date: Thu, 11 Feb 2021 17:37:39 GMT\r\n
Content-Length: 121\r\n
Server: UploadServer\r\n
Alt-Svc: h3-29=":443"; ma=2592000,h3-T051=":443"; ma=2592000,h3-Q050=":443"; ma=2592000,h3-Q046=":443"; ma=2592000,h3-Q043=":443"; ma=2592000,quic=":443"; ma=2592000; v="46,43"\r\n
\r\n
{\n
"kind": "drive#file",\n
"id": "1_SE...8X",\n
"name": "My File.txt",\n
"mimeType": "text/plain"\n
}\n
I need to set the correct body length, otherwise the server returns an error.

TikaJAXRS PUT from Python client

Apache Tika should be accessible from Python program via HTTP, but I can't get it to work.
I am using this command to run the server (with and without the two options at the end):
java -jar tika-server-1.17.jar --port 5677 -enableUnsecureFeatures -enableFileUrl
And it works fine with curl:
curl -v -T /tmp/tmpsojwBN http://localhost:5677/tika
* Trying 127.0.0.1...
* Connected to localhost (127.0.0.1) port 5677 (#0)
> PUT /tika HTTP/1.1
> Host: localhost:5677
> User-Agent: curl/7.47.0
> Accept: */*
> Accept-Encoding: gzip, deflate
> Content-Length: 418074
> Expect: 100-continue
>
< HTTP/1.1 100 Continue
* We are completely uploaded and fine
< HTTP/1.1 200 OK
< Content-Type: text/plain
< Date: Sat, 07 Apr 2018 12:28:41 GMT
< Transfer-Encoding: chunked
< Server: Jetty(8.y.z-SNAPSHOT)
But when I try something like (tried different combinations for headers, here I recreated same headers as python-tika client uses):
with tempfile.NamedTemporaryFile() as tmp_file:
download_file(url, tmp_file)
payload = open(tmp_file.name, 'rb')
headers = {
'Accept': 'application/json',
'Content-Disposition': 'attachment; filename={}'.format(
os.path.basename(tmp_file.name))}
response = requests.put(TIKA_ENDPOINT_URL + '/tika', payload,
headers=headers,
verify=False)
I've tried to use payload as well as fileUrl - with the same result of WARN javax.ws.rs.ClientErrorException: HTTP 406 Not Acceptable and java stack trace on the server. Full trace:
WARN javax.ws.rs.ClientErrorException: HTTP 406 Not Acceptable
at org.apache.cxf.jaxrs.utils.SpecExceptions.toHttpException(SpecExceptions.java:117)
at org.apache.cxf.jaxrs.utils.ExceptionUtils.toHttpException(ExceptionUtils.java:173)
at org.apache.cxf.jaxrs.utils.JAXRSUtils.findTargetMethod(JAXRSUtils.java:542)
at org.apache.cxf.jaxrs.interceptor.JAXRSInInterceptor.processRequest(JAXRSInInterceptor.java:177)
at org.apache.cxf.jaxrs.interceptor.JAXRSInInterceptor.handleMessage(JAXRSInInterceptor.java:77)
at org.apache.cxf.phase.PhaseInterceptorChain.doIntercept(PhaseInterceptorChain.java:307)
at org.apache.cxf.transport.ChainInitiationObserver.onMessage(ChainInitiationObserver.java:121)
at org.apache.cxf.transport.http.AbstractHTTPDestination.invoke(AbstractHTTPDestination.java:274)
at org.apache.cxf.transport.http_jetty.JettyHTTPDestination.doService(JettyHTTPDestination.java:261)
at org.apache.cxf.transport.http_jetty.JettyHTTPHandler.handle(JettyHTTPHandler.java:76)
at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1088)
at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1024)
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:135)
at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:255)
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:116)
at org.eclipse.jetty.server.Server.handle(Server.java:370)
at org.eclipse.jetty.server.AbstractHttpConnection.handleRequest(AbstractHttpConnection.java:494)
at org.eclipse.jetty.server.AbstractHttpConnection.headerComplete(AbstractHttpConnection.java:973)
at org.eclipse.jetty.server.AbstractHttpConnection$RequestHandler.headerComplete(AbstractHttpConnection.java:1035)
at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:641)
at org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:231)
at org.eclipse.jetty.server.AsyncHttpConnection.handle(AsyncHttpConnection.java:82)
at org.eclipse.jetty.io.nio.SelectChannelEndPoint.handle(SelectChannelEndPoint.java:696)
at org.eclipse.jetty.io.nio.SelectChannelEndPoint$1.run(SelectChannelEndPoint.java:53)
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
at java.lang.Thread.run(Thread.java:748)
I've also tried to compare ( with nc -l localhost 5677 | less) what is so different with two requests (payload abbreviated):
From curl:
PUT /tika HTTP/1.1
Host: localhost:5677
User-Agent: curl/7.47.0
Accept: */*
Content-Length: 418074
Expect: 100-continue
%PDF-1.4
%<D3><EB><E9><E1>
1 0 obj
<</Creator (Chromium)
From Python requests library:
PUT /tika HTTP/1.1
Host: localhost:5677
Connection: keep-alive
Accept-Encoding: gzip, deflate
Accept: application/json
User-Agent: python-requests/2.13.0
Content-type: application/pdf
Content-Length: 246176
%PDF-1.4
%<D3><EB><E9><E1>
1 0 obj
<</Creator (Chromium)
The question is, what is the correct way to call Tika server from Python?
I've also tried python tika library in client-only mode and using tika-app via jnius. With tika client, as well as using tika-app.jar with pyjnius, I only freezes (call never returns) when I use them in a celery worker. At the same, pyjnius / tika-app and tika-python script both work nicely in a script: I have not figured out what is wrong inside celery worker. I guess, something to do with threading and/or initialization in wrong place. But that is a topic for another question.
And here is what tika-python requests:
PUT /tika HTTP/1.1
Host: localhost:5677
Connection: keep-alive
Accept-Encoding: gzip, deflate
Accept: application/json
User-Agent: python-requests/2.13.0
Content-Disposition: attachment; filename=tmpb3YkTq
Content-Length: 183234
%PDF-1.4
%<D3><EB><E9><E1>
1 0 obj
<</Creator (Chromium)
And now it seems like this is some kind of a problem with tika server:
$ tika-python --verbose --server 'localhost' --port 5677 parse all /tmp/tmpb3YkTq
2018-04-08 09:44:11,555 [MainThread ] [INFO ] Writing ./tmpb3YkTq_meta.json
(<open file '<stderr>', mode 'w' at 0x7f0b688eb1e0>, 'Request headers: ', {'Accept': 'application/json', 'Content-Disposition': 'attachment; filename=tmpb3YkTq'})
(<open file '<stderr>', mode 'w' at 0x7f0b688eb1e0>, 'Response headers: ', {'Date': 'Sun, 08 Apr 2018 06:44:13 GMT', 'Transfer-Encoding': 'chunked', 'Content-Type': 'application/json', 'Server': 'Jetty(8.y.z-SNAPSHOT)'})
['./tmpb3YkTq_meta.json']
Cf:
$ tika-python --verbose --server 'localhost' --port 5677 parse text /tmp/tmpb3YkTq
2018-04-08 09:43:38,326 [MainThread ] [INFO ] Writing ./tmpb3YkTq_meta.json
(<open file '<stderr>', mode 'w' at 0x7fc3eee4a1e0>, 'Request headers: ', {'Accept': 'application/json', 'Content-Disposition': 'attachment; filename=tmpb3YkTq'})
(<open file '<stderr>', mode 'w' at 0x7fc3eee4a1e0>, 'Response headers: ', {'Date': 'Sun, 08 Apr 2018 06:43:38 GMT', 'Content-Length': '0', 'Server': 'Jetty(8.y.z-SNAPSHOT)'})
2018-04-08 09:43:38,409 [MainThread ] [WARNI] Tika server returned status: 406
['./tmpb3YkTq_meta.json']

sending different 'Content-Type' for each single input param one file and other data params

Is there any way to send each different 'Content-Type' in the multipart-form data for each single input param
e.g.
Content-Type: multipart/form-data;boundary=q235Ht2tTWJhuFmC8sJxbQ7YGU7FwQafcZd8B
Accept-Charset: utf-8
"Content-Disposition: form-data; name="creative_id"
"Content-Type: text/plain;charset=ISO-8859-1”
…
"Content-Disposition: form-data; name=“file_role""
"Content-Type: text/plain;charset=ISO-8859-1”
…
"Content-Disposition: form-data; name="Filename""
"Content-Type: text/plain;charset=ISO-8859-1
"Content-Disposition: form-data; name="file"; filename="advertise_A.png"
"Content-Type: image/x-png"
For the whole request, the header will be
Content-Type: multipart/form-data;
but for its params like creative_id and file_role,
I would like to send Content-Type: text/plain;charset=ISO-8859-1
and for the file itself Content-Type: image/x-png
I tried this in two ways, but it doesnt work:
headers = {'Content-Type':'text/plain;charset=ISO-8859-1'}
files = {'file': open(asset_file, 'rb')}
and then in POST (url, files=files, headers=headers, params=values)
OR
files = {'file1': (open(asset_file, 'rb'), 'image/x-png'), 'creative_id': (1727968, 'text/plain;charset=ISO-8859-1'), 'file_role': ('PRIMARY', 'text/plain;charset=ISO-8859-1')}
and then in POST (url, files=files)
You're really close with the second example. If you provide a dict with tuples as its value, the tuples have this form:
(filename, file object or content, [content type], [headers])
where the content type and headers fields are optional.
This means you want to do this:
files = {'file': ('advertise_A.png', open(asset_file, 'rb'), 'image/x-png'), 'creative_id': ('', '1727968', 'text/plain;charset=ISO-8859-1'), 'file_role': ('', 'PRIMARY', 'text/plain;charset=ISO-8859-1'), 'Filename': ('', 'advertise_A.png', 'text/plain;charset=ISO-8859-1')}
r = requests.post(url, files=files)
Doing the above with a file that contains only the string basic_test, I get the following result:
Content-Type: multipart/form-data; boundary=82c444831d6a450ba5c4ced2e1cc7866
--82c444831d6a450ba5c4ced2e1cc7866
Content-Disposition: form-data; name="creative_id"
Content-Type: text/plain;charset=ISO-8859-1
1727968
--82c444831d6a450ba5c4ced2e1cc7866
Content-Disposition: form-data; name="file_role"
Content-Type: text/plain;charset=ISO-8859-1
PRIMARY
--82c444831d6a450ba5c4ced2e1cc7866
Content-Disposition: form-data; name="file"; filename="advertise_A.png"
Content-Type: image/x-png
basic_test
--82c444831d6a450ba5c4ced2e1cc7866
Content-Disposition: form-data; name="Filename"
Content-Type: text/plain;charset=ISO-8859-1
advertise_A.png
--82c444831d6a450ba5c4ced2e1cc7866--

curl send file as post request

I have file with request data:
POST /exampleUrl HTTP/1.1
Host: localhost:8080
Connection: keep-alive
Content-Length: 57907
Content-Type: multipart/form-data; boundary=--
Accept: */*
---
Content-Type: application/octet-stream
Content-Disposition: form-data; name='wanted'; filename=NameOfFile
Content-Transfer-Encoding: binary
[DATA]
---
Content-Type: application/octet-stream
Content-Disposition: form-data; filename=ID1
Content-Transfer-Encoding: binary
[DATA]
-----
How can i send it through curl as post request?

Help, don't know what's wrong with my HTTP multipart POST

POST /upload HTTP/1.1
Host: assets.drop.io
User-Agent: Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.0.10) Gecko/2
009042316 Firefox/3.0.10
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
Content-Length: 5728
Content-Type: multipart/form-data; boundary=--------MalolHCBdarysNYMHZbIvjbD
--------MalolHCBdarysNYMHZbIvjbD
Content-Disposition: form-data; name="api_key"
0b8a12109c3a1bfc4ba94aec926e1f9cfd8bb6f0
--------MalolHCBdarysNYMHZbIvjbD
Content-Disposition: form-data; name="drop_name"
4pgvoxc
--------MalolHCBdarysNYMHZbIvjbD
Content-Disposition: form-data; name="version"
2.0
--------MalolHCBdarysNYMHZbIvjbD
Content-Disposition: form-data; name="rachel"; filename="rachel"
Content-Type: application/octet-stream
BMv¶
--------MalolHCBdarysNYMHZbIvjbD--
The response I get is: Bad Request. And in the HTML of the response there is this:
:MultiPartParseError: bad content body:
'
----------MalolHCBdarysNYMHZbIvjbD' should == '----------MalolHCBdarysNYMHZbIvjbD
'></td></tr>
The answer is in your question. The response says that the divider should be '----------MalolHCBdarysNYMHZbIvjbD ' instead of ' ----------MalolHCBdarysNYMHZbIvjbD'
Look at the leading and trailing spaces.
The schematic syntax of a multipart/form-data message is as follows:
Content-Type: multipart/form-data; boundary=boundary
--boundary
Content-Disposition: form-data; name="field 1"
data 1
--boundary
Content-Disposition: form-data; name="field 2"
data 2
⋮
--boundary
Content-Disposition: form-data; name="field N"
data N
--boundary--
So the inner part boundaries are --boundary (-- followed by the boundary value) and the last is --boundary-- (-- followed by the boundary value followed by --).
In your case you are just missing the leading --.

Resources