I am a Golang api that accept multipart/form-data requests. For some clients, however, it fails to parse the form because it doesn't like the boundary being used by the client.
The header from the client is:
Content-Type:[multipart/form-data; boundary================1648430772==]
I've narrowed this down to the ParseMediaType function in the mime package.
If I call:
bad := "multipart/form-data; boundary=1650458473"
d, params, err := mime.ParseMediaType(v)
if err != nil {
fmt.Println("err", err)
}
fmt.Println(d, params)
I get the err: mime: invalid media parameter.
Note that if I do this call with
multipart/form-data; boundary=3fc88aad6d1341a4921fd5ac9efe607c
it succeeds no problem.
According to the https://www.w3.org/Protocols/rfc1341/7_2_Multipart.html spec, it looks to me like these are all valid characters for a boundary.
Is this a bug in the Go mime library? Or is this really an invalid boundary?
The rfc you linked to contains BNF for the boundary and multipart body, it does not contain the BNF for the Content-Type Header Field. So while = in boundary is just fine it's not fine in the parameter value of the Content-Type header. At least not unquoted.
So to fix your first example change the Content-Type to this:
multipart/form-data; boundary="===============1648430772=="
https://play.golang.org/p/3Iuk_ACZaQ
Your second example multipart/form-data; boundary=1650458473 seems to work fine.
https://play.golang.org/p/xJWwBa_QiP
Finally found the answer. In the RFC 2045 doc (https://www.ietf.org/rfc/rfc2045.txt) it states that certain values cannot be used as parameter values in the Content-Type header.
The pertinent section:
tspecials := "(" / ")" / "<" / ">" / "#" /
"," / ";" / ":" / "\" / <">
"/" / "[" / "]" / "?" / "="
; Must be in quoted-string,
; to use within parameter values
So you can use an equal sign, but only if it's quoted, so Go fails on the parsing. The client in this case is sending a technically-incorrect value for the boundary param.
Related
I try to post a form to server and here is the code:
ar request = new http.MultipartRequest("POST", _uri);
request.fields['user_acc'] = _userAcc;
// this issue should be solve
request.fields['user_nick_name'] = '中文名字';
request.fields['user_password'] = _password;
But the server side in the user_nick_name field always got null, note that is always, but I change it into English the server can receive that. I test on postman, the server can got Chinese correctly, so it's MultipartRequest issue on this problem.
My question is: Why the Dart or Flutter team so careless on this so important basic library? They even not consider about this simply issue. I opened a issue on github but no-one response, I think the team is done. So I ask the develop communit here, how to solve this problem anyway?
[UPDATE]
As kindly people suggested, I update my golang server now, if anyone else got this problem, you may wonna answer and suggestions too.
func HandleUserRegister(context *gin.Context) {
userAcc := context.PostForm("user_acc")
userAvatar := context.PostForm("user_avatar")
userNickName := context.PostForm("user_nick_name")
userPassword := context.PostForm("user_password")
userPhone := context.PostForm("user_phone")
userEmail := context.PostForm("user_email")
userGender := context.PostForm("user_gender")
userSign := context.PostForm("user_sign")
userType := context.PostForm("user_type")
userTypeInt, _ := strconv.Atoi(userType)
log.Infof("userAcc: %s, userNickName: %s, userPassword: %s", userAcc, userNickName, userPassword)}
This is based on gin, and this function is the api solver. If anyone wanna help, please help me figure it out.
OK! I update the question now, because it's really weird!. I did those test:
Post multiform via Flutter to Django server, it receives Chinese filed correctly;
Post multiform data via Postman, the golang(gin) server gots Chinese correctly;
Post multiform data via Flutter to golang(gin) server gots Chinese field null;
For more detail, I log the headers from my server for both postman(normal) and flutter (abnormal):
Postman:
request header: map[Content-Type:[multipart/form-data; boundary=--------------------------022341683711652813100488] Postman-Token:[855646d7-5bea-4b8f-b8df-81366226cd49] User-Agent:[PostmanRuntime/7.1.1] Content-Length:[422] Connection:[keep-alive] Cache-Control:[no-cache] Accept:[*/*] Accept-Encoding:[gzip, deflate]]
Flutter:
request header: map[User-Agent:[Dart/2.0 (dart:io)] Content-Type:[multipart/form-data; boundary=dart-http-boundary-.XUeYeqXpg4Yfyh8QhH1T5JB4zi_f3WxX9t7Taxhw91EFqhyki4] Accept-Encoding:[gzip] Content-Length:[574]]
Does anyone can notice the difference and let me know how to change the it make server can receive the Chinese Characters?
#DannyTuppeny is correct. This is a server problem.
When asked to include a non-ASCII field into a multi-part request, the Dart library correctly wraps this with a binary content-transfer-encoding.
String _headerForField(String name, String value) {
var header =
'content-disposition: form-data; name="${_browserEncode(name)}"';
if (!isPlainAscii(value)) {
header = '$header\r\n'
'content-type: text/plain; charset=utf-8\r\n'
'content-transfer-encoding: binary';
}
return '$header\r\n\r\n';
}
(Postman does not and simply sends the utf8 encoded string without any headers.)
Dart/ASCII looks like this:
--dart-http-boundary-HjDS88CmQicdgd8VaHSwPqJK8iR4H6rTG3LovSZy-QXGpU7pAB0
content-disposition: form-data; name="test"
stackover
--dart-http-boundary-HjDS88CmQicdgd8VaHSwPqJK8iR4H6rTG3LovSZy-QXGpU7pAB0
Dart/non-ASCII looks like this:
First boundary: --dart-http-boundary-58NU6u6_Fo22xjH8H7yPCtKuoKgB+A8+RTJ82iIK1gs3nnGMLlp\r\n
Encapsulated multipart part: (text/plain)
content-disposition: form-data; name="test"\r\n
content-type: text/plain; charset=utf-8\r\n
content-transfer-encoding: binary\r\n\r\n
Line-based text data: text/plain
\344\270\255\346\226\207\345\220\215\345\255\227
Boundary: \r\n--dart-http-boundary-58NU6u6_Fo22xjH8H7yPCtKuoKgB+A8+RTJ82iIK1gs3nnGMLlp\r\n
So the problem is that the server is unable to unwrap the value from the encapsulation.
EDIT
Here's the Postman trace I captured yesterday. It's multi-form, but fails to add the content-type-encoding header despite the field being non-ASCII.
MIME Multipart Media Encapsulation, Type: multipart/form-data, Boundary: "--------------------------595246000077585285134204"
[Type: multipart/form-data]
First boundary: ----------------------------595246000077585285134204\r\n
Encapsulated multipart part:
Content-Disposition: form-data; name="name"\r\n\r\n
Data (12 bytes)
0000 e4 b8 ad e6 96 87 e5 90 8d e5 ad 97 ............
Data: e4b8ade69687e5908de5ad97
[Length: 12]
Last boundary: \r\n----------------------------595246000077585285134204--\r\n
I tested by posting to httpbin and the response suggests that the characters were posted correctly:
"user_nick_name":"\u4e2d\u6587\u540d\u5b57"
I tried with both the Stable v1 SDK and a v2 SDK from Flutter. Is it possible the issue is on the server? Have you tried using something like Fiddler to capture what's actually being sent?
Edit: My guess is that your server side code is not correctly reading the data as MultipartForm data (eg. you should be using ParseMultipartForm and reading from MultipartForm).
The problem, it appears, is in formdata.go part of multipart. Go assumes that any multipart part with an Content-Type header is a file (not a field). However, knowing this you can change your server code as follows:
func main() {
r := gin.Default()
r.POST("/sotest", func(c *gin.Context) {
formValue := c.PostForm("form_value")
if formValue == "" {
formFile, _ := c.FormFile("form_value")
file, _ := formFile.Open()
b1 := make([]byte, formFile.Size)
file.Read(b1)
formValue = string(b1)
}
c.JSON(200, gin.H{
"status": "posted",
"formValue": formValue,
})
})
r.Run() // listen and serve on 0.0.0.0:8080
}
When you detect that PostForm returns the empty string, you know that Go has treated the field as a file, in which case you can Open and Read the 'file' and decode it as the utf-8 string that we know it is. Obviously, you could encapsulate the "try as PostForm and if that's empty, try as FormFile" test into a function.
If you don't want to have to test for empty string at the server, you could change your Dart end code to always utf-8 encode even non-ascii strings with
request.files.add(
new http.MultipartFile.fromBytes(
'some_form_value_name',
utf8.encode('the string value'),
contentType: new MediaType('text', 'plain', {'charset': 'utf-8'}),
),
);
and read them at the server with the Open/Read/string method.
I have now solved this. Thanks to Richard and Danny for their help.
1. Reason for this
No matter what happens but this really not only one-side problem, we can not say it's Flutter or Go wrong. But the combination, Flutter + Go server just may be got this issue. The behind reason I still not quit sure, but it must some head not right set (postman can do it right);
2. Solution
We don't only need know why but also how to solve it. Here is what I do:
Do not use the official http package. Using dio, which is a extension Dart package. link: https://pub.dartlang.org/packages/dio
It's more clean and easy to use, so my code becomes to:
FormData _formData = new FormData.from({
"user_acc": _userAcc,
"user_nick_name": _userNickName,
'user_password': _password,
});
Dio dio = new Dio();
Response response = await dio.post(usersUrl, data: _formData);
print(response.data);
I can not post the none-English words now:
INFO[0668] userAcc: ww, userNickName: 小鹿叮叮婴儿湿巾手口专用80抽湿纸巾婴儿湿巾婴儿100抽带盖批发【原价】34.90元【券后】9.9元【省】25元【复制此信息打开手机淘宝即可查看并下单】￥Tnsx0E77pFs￥【必买理由】新品预售80抽*3仙女联盟，更多优惠fd.loliloli.pro , userPassword: ww
INFO[0671] user exist.
I'm using Indy with Lazarus
Here is my code:
IdHTTP1.Request.ContentType := 'text/plain' ;
IdHTTP1.Response.ContentType := 'text/plain' ;
IdHTTP1.Response.Charset := 'ISO-8859-1,utf-8;q=0.7,*;q=0.3' ;
IdHTTP1.Request.CharSet:= 'ISO-8859-1,utf-8;q=0.7,*;q=0.3 ' ;
IdHTTP1.HTTPOptions := IdHTTP1.HTTPOptions + [hoNoProtocolErrorException];
IdHTTP1.Get('http://192.168.25.965:8541/rest/SearchCard('+MYCARD+')',Stream) ;
If I start MYCARD with a letter, the server is picking up the full string. However, if I start with a number, it stops at the first letter.
MYCARD:= '12366854'; //works
MYCARD:= 'A125ASD555'; //Works
MYCARD:= '123YH963'; // The server only sees 123
What am I doing wrong?
First off, the two Request properties you are setting are meaningless in a GET request, and you should not be setting any Response properties at all.
// get rid of these assignments
//IdHTTP1.Request.ContentType := 'text/plain' ;
//IdHTTP1.Response.ContentType := 'text/plain' ;
//IdHTTP1.Response.Charset := 'ISO-8859-1,utf-8;q=0.7,*;q=0.3' ;
//IdHTTP1.Request.CharSet:= 'ISO-8859-1,utf-8;q=0.7,*;q=0.3 ' ;
IdHTTP1.HTTPOptions := IdHTTP1.HTTPOptions + [hoNoProtocolErrorException];
IdHTTP1.Get('http://192.168.25.965:8541/rest/SearchCard('+MYCARD+')', Stream);
Second, using the current version of Indy, I cannot reproduce your issue. TIdHTTP.Get() sends the specified URL as-is, it makes no assumptions about the characters in it (you are responsible for URL encoding). In my testing, 123YH963 works just fine. Here is the actual HTP request being sent:
GET /rest/SearchCard(123YH963) HTTP/1.1
Host: 192.168.25.965:8541
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,/;q=0.8
User-Agent: Mozilla/3.0 (compatible; Indy Library)
As you can see, the full MYCARD text is in the requested resource, as expected. So any truncation must be happening on the server side, not in TIdHTTP itself.
Are you sure you are formatting the URL correctly to begin with? Are you sure it should actually be sent like this:
/rest/SearchCard(123YH963)
And not something more like these instead?
/rest/SearchCard%28123YH963%29
/rest/SearchCard/123YH963
/rest/SearchCard?param=123YH963
I've just set a Warning HTTP header for the first time. In the System.Net .NET namespace, there's a type WarningHeaderValue with this constructor:
WarningHeaderValue(int code, string agent, string text)
But it throws on my agent string saying
The format of value 'Company Name .NET Origin' is invalid.
What format is legal for the agent? I couldn't glean anything useful from the HTTP spec.
warn-agent = ( uri-host [ ":" port ] ) / pseudonym
; the name or pseudonym of the server adding
; the Warning header field, for use in debugging
; a single "-" is recommended when agent unknown
warn-text = quoted-string
warn-date = DQUOTE HTTP-date DQUOTE
What's missing in all these arcane specifications are examples.
In this case, quoted-string means include quotes within the string.
So instead of: "Deprecated API" , it would be "\"Deprecated API\""
Full example:
HttpResponseMessage response = await base.ExecuteAsync(cancellationToken);
string message = $"\"Deprecated API : use {NewUri} instead.\"";
response.Headers.Warning.Add(new WarningHeaderValue(299, "-", message));
return response;
The question is simple. In AutoHotkey, how would you parse a raw HTTP request to access data like the method, http version, cookies and host.
;Example of a raw http request:
rawHttp =
(
POST https://www.example.com/Login HTTP/1.1
Host: www.example.com
Connection: Keep-Alive
Cookie: session=b5j2h46fdthr46t74g5g234g5f3g6753kj73l
username=lalala&password=12345&rememberMe=1
)
;Example usage of the function below:
httpObj := HttpTxtToObj(rawHttp)
MsgBox % "Request method: " httpObj.requestLine.method
MsgBox % "Request URI: " httpObj.requestLine.uri
MsgBox % "Request http version: " httpObj.requestLine.httpVersion
MsgBox % "Request header (Host): " httpObj.headers["Host"]
MsgBox % "Request header (Connection): " httpObj.headers["Connection"]
MsgBox % "Request header (Cookie): " httpObj.headers["Cookie"]
MsgBox % "Request body: " httpObj.body
;This function takes care of everything:
HttpTxtToObj(rawHttp) {
;Split request into "request line", "headers" and "body"(if existent)
RegExMatch(rawHttp, "OU)^(?P<requestLine>.+)\R(?P<headers>.+)\R\R(?P<body>.*)$",request)
If !request.Count()
RegExMatch(rawHttp, "OU)^(?P<requestLine>.+)\R(?P<headers>.+)$",request)
;Split request line into "method" "requestUrl" and "httpVersion"
RegExMatch(request.requestLine, "OU)^(?P<method>[^\s]+)\s(?P<uri>[^\s]+)\s(?P<httpVersion>.+)$", requestLine)
;Make a nice key value array for the headers:
headers := {}
While (p := RegexMatch(request.headers, "OSU)(?P<key>[^:]+):\s(?P<value>[^\R]+(\R|$))", currentHeader, p?p+1:1))
headers.Insert(currentHeader.key, currentHeader.value)
;The body is usually a query string, json string or just text. When you uplaod a file it may even contain binary data.
;All that would make the code quite a bit more complex, so for now we'll just pretend it's normal text, nothing special.
;Now lets return a nice multidimensional array
Return {requestLine: requestLine, headers: headers, body: request.body}
}
Using go, how can I parse the Content-Disposition header retrieved from an http HEAD request to obtain the filename of the file?
Additionally, how do I retrieve the header itself from the http HEAD response? Is something like this correct?
resp, err := http.Head("http://example.com/")
//handle error
contentDisposition := resp.Header.Get("Content-Disposition")
The mime/multipart package specifies a method on the Part type that returns the filename (called FileName), but it's not clear to me how I should construct a Part, or from what.
You can parse the Content-Disposition header using the mime.ParseMediaType function.
disposition, params, err := mime.ParseMediaType(`attachment;filename="foo.png"`)
filename := params["filename"] // set to "foo.png"
This will also work for Unicode file names in the header (e.g. Content-Disposition: attachment;filename*="UTF-8''fo%c3%b6.png").
You can experiment with this here: http://play.golang.org/p/AjWbJB8vUk