I am using my Arduino to do a multipart/form-data request. I am generating the request completely by myself as follows:
client.println(HTTP_METHOD + " " + PATH_NAME + " HTTP/1.1");
client.println("Host: " + String(HOST_NAME));
client.print(F("Content-Type: multipart/form-data; "));
client.print(F("boundary=\"AaB03x\"\r\n"));
client.print(F("Content-Length: "));
client.print(strlen("{json data here}")
+ strlen("{json data here}"));
client.print("\r\nConnection: close\r\n");
// First part
// Boundary
client.print(F("\r\n--AaB03x\r\n"));
// Headers
client.print(F("Content-Disposition: form-data; name=\"header\"\r\n"));
client.print(F("Content-Type: application/ld+json\r\n"));
client.print(F("Content-Length: "));
client.print(strlen("{json data here}"));
client.print(F("\r\n\r\n"));
// Content
client.print(F("{json data here}"));
// Second part
// Boundary
client.print(F("\r\n--AaB03x\r\n"));
// Headers
client.print(F("Content-Disposition: form-data; name=\"payload\"\r\n"));
client.print(F("Content-Type: application/ld+json\r\n"));//
client.print(F("Content-Length: "));
client.print(strlen("{json data here}"));
client.print(F("\r\n\r\n"));
// Content
client.print(F("{json data here}"));
// End of boundary
client.print(F("\r\n--AaB03x--\r\n\r\n"));
However the server returns "Incomplete multipart" which I don't understand since the multipart seems completely fine. I thought it was maybe due to incorrect newlines but I haven't been able to figure out a solution.
The output can be seen below and includes the request sent as well as the response from the server.
GET /router HTTP/1.1
Host: 192.168.178.147
Content-Type: multipart/form-data; boundary="AaB03x"
Content-Length: 4363
Connection: close
--AaB03x
Content-Disposition: form-data; name="header"
Content-Type: application/ld+json
Content-Length: 4143
{
some data here
}
--AaB03x
Content-Disposition: form-data; name="payload"
Content-Type: application/ld+json
Content-Length: 220
{
some data here
}
--AaB03x--
connected to 192.168.178.147
HTTP/1.1 500 Server Error
Connection: close
Cache-Control: must-revalidate,no-cache,no-store
Content-Type: text/html;charset=iso-8859-1
Content-Length: 597
Server: Jetty(9.4.41.v20210516)
<html>
<head>
<meta http-equiv="Content-Type" content="text/html;charset=utf-8"/>
<title>Error 500 Server Error</title>
</head>
<body><h2>HTTP ERROR 500 Server Error</h2>
<table>
<tr><th>URI:</th><td>/router</td></tr>
<tr><th>STATUS:</th><td>500</td></tr>
<tr><th>MESSAGE:</th><td>Server Error</td></tr>
<tr><th>SERVLET:</th><td>org.apache.camel.component.jetty.CamelContinuationServlet-38cb1606</td></tr>
<tr><th>CAUSED BY:</th><td>java.io.IOException: Incomplete Multipart</td></tr>
</table>
<hr>Powered by Jetty:// 9.4.41.v20210516<hr/>
</body>
</html>
Your Content-Length calculations are suspect.
Drop the Content-Length headers for each sub-section in the multipart and try again, those are not needed for multipart, and your calculations are just wrong anyway.
You can see captures of various multipart requests from various libraries and browsers in the Jetty tests.
https://github.com/eclipse/jetty.project/tree/jetty-9.4.41.v20210516/jetty-http/src/test/resources/multipart
(Look at the ones ending in *.raw, which you can generally open in a text editor)
Tip: don't do this yourself, multipart mime is full of edge cases, traps, and ancient tricks. Go grab apache httpcomponents httpmime jar and just use it to generate your raw mime multipart section properly.
Artifacts - https://search.maven.org/artifact/org.apache.httpcomponents/httpmime
Javadoc - https://javadoc.io/doc/org.apache.httpcomponents/httpmime/latest/index.html
Related
I have a strange situation. I want to return the content type application/json; charset=utf-8 from an http handler.
func handleTest() http.HandlerFunc {
return func(w http.ResponseWriter, r *http.Request) {
if r.Header.Get("Accept") != "application/json" {
w.WriteHeader(http.StatusNotAcceptable)
return
}
w.WriteHeader(http.StatusOK)
w.Header().Set("Content-Type", "application/json; charset=utf-8")
json.NewEncoder(w).Encode(map[string]string{"foo": "bar"})
}
}
When I check for this in my unit tests it is correct. This test does not fail.
func TestTestHandler(t *testing.T) {
request, _ := http.NewRequest(http.MethodGet, "/test", nil)
request.Header.Set("Accept", "application/json")
response := httptest.NewRecorder()
handleTest().ServeHTTP(response, request)
contentType := response.Header().Get("Content-Type")
if contentType != "application/json; charset=utf-8" {
t.Errorf("Expected Content-Type to be application/json; charset=utf-8, got %s", contentType)
return
}
}
But when I try with curl (and other clients) it comes out as text/plain; charset=utf-8.
$ curl -H 'Accept: application/json' localhost:8080/test -v
* Trying 127.0.0.1:8080...
* TCP_NODELAY set
* Connected to localhost (127.0.0.1) port 8080 (#0)
> GET /test HTTP/1.1
> Host: localhost:8080
> User-Agent: curl/7.68.0
> Accept: application/json
>
* Mark bundle as not supporting multiuse
< HTTP/1.1 200 OK
< Date: Tue, 28 Dec 2021 13:02:27 GMT
< Content-Length: 14
< Content-Type: text/plain; charset=utf-8
<
{"foo":"bar"}
* Connection #0 to host localhost left intact
I have tried this with curl, insomnia and python. In all 3 cases the content type came out as text/plain; charset=utf-8.
What is causing this problem and how can I fix it?
From the http package docs:
WriteHeader sends an HTTP response header with the provided status code.
and
Changing the header map after a call to WriteHeader (or Write) has no effect unless the modified headers are trailers.
So you are setting the "Content-Type" header after the header has already been sent out to the client. While mocking this likely works because the buffer where the headers are stored can be modified after the WriteHeader call. But when actually using a TCP connection you can't do this.
So simply move your w.WriteHeader(http.StatusOK) so it happens after the w.Header().Set(...)
Can't understand what is wrong. ioutil.ReadAll should use gzip as for other URLs.
Can reproduce with URL: romboutskorea.co.kr
Error:
gzip: invalid header
Code:
resp, err := http.Get("http://" + url)
if err == nil {
defer resp.Body.Close()
if resp.StatusCode == http.StatusOK {
fmt.Printf("HTTP Response Status : %v\n", resp.StatusCode)
bodyBytes, err := ioutil.ReadAll(resp.Body)
if err != nil {
fmt.Printf("HTTP Response Read error. Url: %v\n", url)
log.Fatal(err)
}
bodyString := string(bodyBytes)
fmt.Printf("HTTP Response Content Length : %v\n", len(bodyString))
}
}
The response of this site is wrong. It is claiming gzip encoding but it does not actually compress the content. The response looks something like this:
HTTP/1.1 200 OK
...
Content-Encoding: gzip
...
Transfer-Encoding: chunked
Content-Type: text/html; charset=euc-kr
8000
<html>
<head>
...
The "8000" comes from the chunked transfer encoding but the "..." is the beginning of the unchunked response body. Obviously this is not compressed even though it is claimed so.
It looks like browsers simply work around this broken site by ignoring the wrong encoding specification. Browsers actually work around lot of broken stuff which does not really add motivation for the providers to fix these issues :( But you can see that curl will fail to:
$ curl -v --compressed http://romboutskorea.co.kr/main/index.php?
...
< HTTP/1.1 200 OK
< ...
< Content-Encoding: gzip
< ...
< Transfer-Encoding: chunked
< Content-Type: text/html; charset=euc-kr
<
* Error while processing content unencoding: invalid code lengths set
* Failed writing data
* Curl_http_done: called premature == 1
* Closing connection 0
curl: (23) Error while processing content unencoding: invalid code lengths set
And so does Python:
$ python3 -c 'import requests; requests.get("http://romboutskorea.co.kr/main/index.php?")'
...
requests.exceptions.ContentDecodingError: ('Received response with content-encoding: gzip, but failed to decode it.', error('Error -3 while decompressing data: incorrect header check'))
I see
Content-Type: text/html; charset=euc-kr
Content-Encoding: gzip
Check the Body content: as in here, it could be an HTTP response where the body is first compressed with gzip and then encoded with chunked transfer encoding.
An NewChunkedReader would be needed, as in this example.
I had a similar issue, but I was dealing with a "hand-crafted" PHP script response which did something like this:
header('Content-Encoding: gzip');
echo #gzcompress($return);
I was trying to read the response from GO with:
gzip.NewReader(resp.Body)
But I should be doing:
zlib.NewReader(resp.Body)
From gzcompress PHP docs:
https://www.php.net/manual/en/function.gzcompress.php
'This function compresses the given string using the ZLIB data format.'
'This is not the same as gzip compression, which includes some header data. See gzencode() for gzip compression.'
I need to send simple HTTP POST Request using ESP8266. It's containing data in "form-data".
It should look like this:
POST http://testserver.com
{
"auth_key":"key",
"data":[
{
"key":"temperature",
"value":31.2
},
{
"key":"humidity",
"value":50
}
]
}
For the testing I was using Chrome application - Postman. And the HTTP Request code which I send was looking like this (it was automatically generated):
POST /api/mes HTTP/1.1
Host: testserver.com
Cache-Control: no-cache
Postman-Token: 9b910ed2-afdc-2a11-4963-2f85626cfa4e
Content-Type: multipart/form-data; boundary=----WebKitFormBoundary7MA4YWxkTrZu0gW
------WebKitFormBoundary7MA4YWxkTrZu0gW
Content-Disposition: form-data; name="auth_key"
79bde0ff1efeaee90b1e432c08d324ecfdb532ac42406d7a9a87dd911e95f87e
------WebKitFormBoundary7MA4YWxkTrZu0gW
Content-Disposition: form-data; name="data"
[{"key":"humidity", "value":55}]
------WebKitFormBoundary7MA4YWxkTrZu0gW--
And through Postman everything was okay. So then I sent it through ESP8266:
client.setNoDelay(true);
client.println("POST /api/mes HTTP/1.1");
client.println("Host: testserver.com");
client.println("Cache-Control: no-cache");
client.println("Postman-Token: 9b910ed2-afdc-2a11-4963-2f85626cfa4e");
client.println("Content-Type: multipart/form-data; boundary=----WebKitFormBoundary7MA4YWxkTrZu0gW");
client.println("");
client.println("------WebKitFormBoundary7MA4YWxkTrZu0gW");
client.println("Content-Disposition: form-data; name=\"auth_key\"");
client.println("");
client.println("79bde0ff1efeaee90b1e432c08d324ecfdb532ac42406d7a9a87dd911e95f87e");
client.println("------WebKitFormBoundary7MA4YWxkTrZu0gW");
client.println("Content-Disposition: form-data; name=\"data\"");
client.println("");
client.println("[{\"key\":\"humidity\", \"value\":55}]");
client.println("------WebKitFormBoundary7MA4YWxkTrZu0gW--");
But unfortunately server is returning that the request is not correct. What can cause the problem? I'm struggling with it but I don't have any more ideas.
I am trying to develop a simple CalDAV server for my application and in this regard I am trying to serve the initial PROPFIND request from the calendar client.
The calendar client requests for a set of properties (as shown below) to be set by the server in response to the request:
Method: PROPFIND /calendars/user/ HTTP/1.1
Request Header:
Accept-encoding gzip, deflate
Accept */*
Connection keep-alive
Prefer return=minimal
Host 192.168.0.12:8080
Brief t
User-agent Mac+OS+X/10.10.5 (14F27) CalendarAgent/316.1
Depth 0
Authorization Basic YWRtaW46cGFzc3dvcmQ=
Accept-language en-us
Content-type text/xml
Content-length 743
Request Body:
<?xml version="1.0" encoding="UTF-8"?>
<A:propfind xmlns:A="DAV:">
<A:prop>
<B:calendar-home-set xmlns:B="urn:ietf:params:xml:ns:caldav"/>
<B:calendar-user-address-set xmlns:B="urn:ietf:params:xml:ns:caldav"/>
<A:current-user-principal/>
<A:displayname/>
<C:dropbox-home-URL xmlns:C="http://calendarserver.org/ns/"/>
<C:email-address-set xmlns:C="http://calendarserver.org/ns/"/>
<C:notification-URL xmlns:C="http://calendarserver.org/ns/"/>
<A:principal-collection-set/>
<A:principal-URL/>
<A:resource-id/>
<B:schedule-inbox-URL xmlns:B="urn:ietf:params:xml:ns:caldav"/>
<B:schedule-outbox-URL xmlns:B="urn:ietf:params:xml:ns:caldav"/>
<A:supported-report-set/>
</A:prop>
</A:propfind>
As a response to the above PROPFIND request I set the response as shown below:
Response Header
Accept-encoding gzip, deflate
Accept */*
Connection keep-alive
Prefer return=minimal
Host 192.168.0.12:8080
Brief t
User-agent Mac+OS+X/10.10.5 (14F27) CalendarAgent/316.1
Depth 0
Authorization Basic YWRtaW46cGFzc3dvcmQ=
Accept-language en-us
Content-type text/xml
Content-length 743
Response Body
String response = "<?xml version=\"1.0\" encoding=\"UTF-8\" ?>"+
"<A:multistatus xmlns:A=\"DAV:\" xmlns:cs=\"http://calendarserver.org/ns/\">" +
"<A:response>" +
"<A:href>/calendars/user/</A:href>" +
"<A:propstat>" +
"<A:prop>" +
"<B:calendar-home-set xmlns:B=\"urn:ietf:params:xml:ns:caldav\"><B:href>/admin/calendar/test/</B:href></B:calendar-home-set>"+
"<B:calendar-user-address-set xmlns:B=\"urn:ietf:params:xml:ns:caldav\"/><A:href>mailto:admin#example.de</A:href></B:calendar-user-address-set>"+
"<A:current-user-principal><A:href>/principals/users/admin</A:href></A:current-user-principal>"+
"<A:displayname>Test calendar</A:displayname>"+
"<A:principal-collection-set><A:href>/principals/users/</A:href><A:href>/principals/groups/</A:href></A:principal-collection-set>"+
"<A:principal-URL><A:href>http://192.168.0.12/principals/users/admin/</A:href></A:principal-URL>"+
"<A:supported-report-set xmlns:n2=\"urn:inverse:params:xml:ns:inverse-dav\" xmlns:n3=\"urn:ietf:params:xml:ns:carddav\" xmlns:A=\"DAV:\" xmlns:n1=\"urn:ietf:params:xml:ns:caldav\">"+
"<A:supported-report><A:report><n1:calendar-query/></A:report></A:supported-report>"+
"<A:supported-report><A:report><n1:calendar-multiget/></A:report></A:supported-report>"+
"<A:supported-report><A:report><n2:acl-query/></A:report></A:supported-report>"+
"<A:supported-report><A:report><A:sync-collection/></A:report></A:supported-report>"+
"<A:supported-report><A:report><A:expand-property/></A:report></A:supported-report>"+
"<A:supported-report><A:report><n3:addressbook-query/></A:report></A:supported-report>"+
"<A:supported-report><A:report><n1:free-busy-query/></A:report></A:supported-report>"+
"</A:supported-report-set>"+
"<B:schedule-inbox-URL xmlns='urn:ietf:params:xml:ns:caldav'><href xmlns='DAV:'>/calendars/__uids__/admin/inbox/</href></B:schedule-inbox-URL>"+
"<B:schedule-outbox-URL xmlns='urn:ietf:params:xml:ns:caldav'><href xmlns='DAV:'>/calendars/__uids__/admin/outbox/</href></B:schedule-outbox-URL>"+
"<C:dropbox-home-URL xmlns='http://calendarserver.org/ns/'><href xmlns='DAV:'>/calendars/__uids__/admin/dropbox/</href></C:ropbox-home-URL>"+
"</A:prop>" +
"<A:status>HTTP/1.1 200 OK</A:status>" +
"</A:propstat>" +
"</A:response>" +
"</A:multistatus>";
After sending the response to client, the client again sends back the same properties (not all - as shown below).
<?xml version="1.0" encoding="UTF-8"?>
<A:propfind xmlns:A="DAV:">
<A:prop>
<A:current-user-principal/>
<A:principal-collection-set/>
</A:prop>
</A:propfind>
So can anyone let me know where am I going wrong in setting these properties ?
As Evert mentioned in the comment section, the client is bit chatty. After several observations even I found the same.
For some reason I need to use a blocking call to perform image accessing from google's server.
However, QNetworkAccessManager seems to be async, though there are many work arounds, like calling a eventLoop.exec(); many people online suggested me not to do so.
So I am trying to use TCP socekt.
I want to access the image here:
http://mt1.google.com/vt/lyrs=y&x=0&y=0&z=0
And here is my code:
socket = new QTcpSocket(this);
socket->connectToHost("mt1.google.com", 80, QIODevice::ReadWrite);
if(socket->waitForConnected(5000))
{
qDebug() << "Connected!";
// send
socket->write("/vt/lyrs=y&x=0&y=0&z=0");
socket->waitForBytesWritten(1000);
socket->waitForReadyRead(3000);
qDebug() << "Reading: " << socket->bytesAvailable();
// get the data
qDebug() << socket->readAll();
// close the connection
socket->close();
}
else
{
qDebug() << "Not connected!";
}
But it seems to working at all? What should I write through the tcp socket to get the image?
TCP provides only the transport mechanism. Since you are trying to communicate with a web server, you should compose HTTP messages.
Replace the line
socket->write("/vt/lyrs=y&x=0&y=0&z=0");
with
socket->write("GET /vt/lyrs=y&x=0&y=0&z=0 HTTP/1.1\r\nHost: mt1.google.com\r\nUser-Agent: TestAgent\r\n\r\n");
And you should get the following response :
HTTP/1.1 200 OK
Date: Sun, 14 Jun 2015 14:24:40 GMT
Expires: Sun, 14 Jun 2015 14:24:40 GMT
Cache-Control: private, max-age=3600
Access-Control-Allow-Origin: *
Content-Type: image/jpeg
X-Content-Type-Options: nosniff
Server: paintfe
Content-Length: 10790
X-XSS-Protection: 1; mode=block
X-Frame-Options: SAMEORIGIN
Alternate-Protocol: 80:quic,p=0
IMAGEDATA
Parse the response and extract the IMAGEDATA part.
EDIT : TCP delivers the response divided into chunks. With this approach, you will not be able to receive the whole response since you are trying to receive it in one go.
You should examine the Content-Length header and wait until receiveing the specified amount of bytes.