An image gets corrupted while being retrieved (through HTTP) and then sent (through HTTP) to a database. Image's raw data is handled in String form.
The service sends a GET for an image file, receives response with the raw image data (response's body) and the Content-Type. Then, a PUT request is sent with the aforementioned request's body and Content-Type header. (The PUT request is constructed by providing the body in String) This PUT request is sent to a RESTful database (CouchDB), creating an attachment (for those unfamiliar with CouchDB an attachment acts like a static file).
Now I have the original image, which my service GETs and PUTs to a database, and this 'copy' of the original image, that I can now GET from the database. If I then `curl --head -v "[copy's url]" it has the Content-Type of the original image, but Content-Length has changed, went from 200kb to about 400kb. If I GET the 'copy' image with a browser, it is not rendered, whereas, the original renders fine. It is corrupted.
What might be the cause? My guess is that while handling the raw data as a string, my framework guesses the encoding wrong and corrupts it. I have not been able to confirm or deny this. How could I handle this raw data/request body in a safe manner, or how could I properly handle the encoding (if that proves to be the problem)?
Details: Play2 Framework's HTTP client, Scala. Below a test to reproduce:
"able to copy an image" in {
def waitFor[T](future:Future[T]):T = { // to bypass futures
Await.result(future, Duration(10000, "millis"))
}
val originalImageUrl = "http://laughingsquid.com/wp-content/uploads/grumpy-cat.jpg"
val couchdbUrl = "http://admin:admin#localhost:5984/testdb"
val getOriginal:ws.Response = waitFor(WS.url(originalImageUrl).get)
getOriginal.status mustEqual 200
val rawImage:String = getOriginal.body
val originalContentType = getOriginal.header("Content-Type").get
// need an empty doc to have something to attach the attachment to
val emptyDocUrl = couchdbUrl + "/empty_doc"
val putEmptyDoc:ws.Response = waitFor(WS.url(emptyDocUrl).put("{}"))
putEmptyDoc.status mustEqual 201
//uploading an attachment will require the doc's revision
val emptyDocRev = (putEmptyDoc.json \ "rev").as[String]
// create actual attachment/static file
val attachmentUrl = emptyDocUrl + "/0"
val putAttachment:ws.Response = waitFor(WS.url(attachmentUrl)
.withHeaders(("If-Match", emptyDocRev), ("Content-Type", originalContentType))
.put(rawImage))
putAttachment.status mustEqual 201
// retrieve attachment
val getAttachment:ws.Response = waitFor(WS.url(attachmentUrl).get)
getAttachment.status mustEqual 200
val attachmentContentType = getAttachment.header("Content-Type").get
originalContentType mustEqual attachmentContentType
val originalAndCopyMatch = getOriginal.body == getAttachment.body
originalAndCopyMatch aka "original matches copy" must beTrue // << false
}
Fails at the last 'must':
[error] x able to copy an image
[error] original matches copy is false (ApplicationSpec.scala:112)
The conversion to String is definitely going to cause problems. You need to work with the bytes as Daniel mentioned.
Looking at the source it looks like ws.Response is just a wrapper. If you get to the underlying class then there are some methods that may help you. On the Java side, someone made a commit on GitHub to expose more ways of getting the response data other than a String.
I'm not familiar with scala but something like this may work:
getOriginal.getAHCResponse.getResponseBodyAsBytes
// instead of getOriginal.body
WS.scala
https://github.com/playframework/playframework/blob/master/framework/src/play/src/main/scala/play/api/libs/ws/WS.scala
WS.java
Here you can see that Response has some new methods, getBodyAsStream() and asByteArray.
https://github.com/playframework/playframework/blob/master/framework/src/play-java/src/main/java/play/libs/WS.java
Related
I am stuck at a problem.
I need to upload a file (a CSV file) to a server by using Delphi. The server have certain requirements that must be followed :
The request must be an HTTP PUT request.
Must contain the Client ID in the URL request, like this : example.com/upload/{Client ID} ({Client ID} would be replaced).
Must contain the future file name that will be received also in the URL after the client ID, like this : example.com/upload/{Client ID}/{Future File Name} ({Future File Name} would be replaced), which can be different from your local file name.
The API Key must be in the request header (header name: x-api-key). Example : lXViWTzFic9sM8qe9Ew7JME8xTdBAOMJHdIjK7XkjQ00OWr.
I could do this in CURL, like this example : (Client ID : d32krpq, and Future file name : 20181023_update.csv)
curl -v -X PUT https://example.com/upload/d32krpq/20181023_update.csv --upload-file 20181023_update.csv --header "x-api-key: lXViWTzFic9sM8qe9Ew7JME8xTdBAOMJHdIjK7XkjQ00OWr"
How to achieve this using Delphi components (using either TNetHttpClient or TRESTClient)?
In Delphi, sending multiple fields (files, custom headers, body fields, ...) in your PUT request is possible in both HTTP client frameworks:
In TNetHTTPClient through the method Put(), you can use the overloaded function with the local file path's string parameter. And in the function's optional parameter of TNetHeaders object, you specify your custom header parameters.
TNetHeaders is a dynamic array of the TNameValuePair record.
So, you have to create a dynamic array of TNameValuePair, then you create an instance of the TNameValuePair record with your API key and add it to the dynamic array.
With the exception that in TNetHTTPClient, you have to specify the URL segments parameter directly in your URL's request by yourself.
And finally, you call the Put() method with your file path's string and TNetHeaders objects, like this example:
procedure TForm1.PutFile;
var
LHTTP: TNetHTTPClient;
LHeaders: array of TNameValuePair;
begin
SetLength(LHeaders, 1);
LHeaders[0] := TNameValuePair.Create('x-api-key', 'lXViWTzFic9sM8qe9Ew7JME8xTdBAOMJHdIjK7XkjQ00OWr');
LHTTP := TNetHTTPClient.Create(nil);
try
LHTTP.Put('https://example.com/upload/d32krpq/20181023_update.csv', 'C:/MyFile.csv', nil, LHeaders);
finally
LHTTP.Free;
end;
end;
For the REST Client library, you can simply use the TRESTRequest.AddFile() function to add a file, and you can use the AddParameter() function to add any custom fields as a body, an URL segment or a header parameter.
Another advantage in TRESTClient is that you can use a base URL or a resource where you define URL segments, like "example.com/upload/{Client ID}/{Future File Name}".
Drop the components TRESTClient, TRESTRequest, and TRESTResponse on the form, set the TRESTClient.BaseURL to your upload server, and write your code. Like in this example:
procedure TForm1.PutFile;
begin
try
try
RESTRequest1.Method := TRESTRequestMethod.rmPUT;
RESTRequest1.AddFile('File1', 'C:\MyFile.csv', TRESTContentType.ctTEXT_CSV);
// Adding the parameters to be included in the URL
RESTRequest1.AddField('Client ID', 'd32krpq', pkURLSEGMENT);
RESTRequest1.AddField('Future File Name', '20181023_update.csv', pkURLSEGMENT);
// Adding a header parameter
RESTRequest1.AddField('x-api-key', 'lXViWTzFic9sM8qe9Ew7JME8xTdBAOMJHdIjK7XkjQ00OWr', pkHTTPHeader);
RESTRequest1.Execute;
except
ShowMessage('Uploading failed');
end;
finally
ShowMessage(RESTRequest1.Response.Content); // here you can see your server's response
end;
end;
Here is an alternative approach using Indy's TIdHTTP component:
FS := TIdReadFileExclusiveStream.Create('20181023_update.csv');
try
IdHTTP1.Request.CustomHeaders.Values['x-api-key'] := 'lXViWTzFic9sM8qe9Ew7JME8xTdBAOMJHdIjK7XkjQ00OWr';
IdHTTP1.Put('https://example.com/upload/d32krpq/20181023_update.csv', FS);
finally
FS.Free;
end;
Stuck on this problem where I received this error everytime making POST request to my actix-web server.
CORS header 'Access-Control-Allow-Origin' missing
my javascript (VueJs running on localhost:3000) :
let data = //some json data
let xhr = new XMLHttpRequest();
xhr.open("POST", "http://localhost:8080/abc");
xhr.setRequestHeader("Content-Type", "application/json");
xhr.onload = () => {
console.log(xhr.responseText);
}
xhr.send(JSON.stringify(data));
My Actix_Web server (running on localhost:8080) :
#[actix_web::main]
async fn main() {
HttpServer::new(move || {
let cors = Cors::default()
.allowed_origin("http://localhost:3000/")
.allowed_methods(vec!["GET", "POST"])
.allowed_header(actix_web::http::header::ACCEPT)
.allowed_header(actix_web::http::header::CONTENT_TYPE)
.max_age(3600);
App::new()
.wrap(cors)
.service(myfunc)
})
.bind(("0.0.0.0", 8080))
.unwrap()
.run()
.await
.unwrap();
}
my cargo.toml dependencies
[dependencies]
actix-web = "4"
actix-cors = "0.6.1"
...
Got any idea?
Okay, so I've done some testing. If you're writing a public API, you probably want to allow all origins. For that you may use the following code:
HttpServer::new(|| {
let cors = Cors::default().allow_any_origin().send_wildcard();
App::new().wrap(cors).service(greet)
})
If you're not writing a public API... well, I'm not sure what they want you to do. I've not figured out how to tell the library to send that header. I guess I will look at the code.
UPDATE:
So funny story, this is how you allow specific origins:
let cors = Cors::default()
.allowed_origin("localhost:3000")
.allowed_origin("localhost:2020");
BUT, and oh boy, is that but juicy. The Access-Control-Allow-Origin response header is only set when there is a Origin request header. That header is normally added by the browser in certain cases 1. So I did that (using the Developer tools in the browser). What did I get? "Origin is not allowed to make this request". I set my origin header to localhost:3000. Turns out, the arctix library simply discards that header if no protocol was provided... (e.g. http://) (I assume it discards it, if it deems its format invalid). That internally results in the header being the string "null". Which is, checks notes, not in the list of allowed origins.
And now the grand finale:
Your origin header needs to be set to (by either you or the browser): "http://localhost:3000".
Your configuration needs to include: .allowed_origin("http://localhost:3000").
After doing that, the server will happily echo back your origin header in the Access-Control-Allow-Origin header. And it will only send that one.
I've no idea if any of that is what the standard specifies (or not). I encourage you to read through it, and if it doesn't comply, please open an issue on GitHub. I would do it myself, but I'm done with programming for today.
Cheers!
I try to post a form to server and here is the code:
ar request = new http.MultipartRequest("POST", _uri);
request.fields['user_acc'] = _userAcc;
// this issue should be solve
request.fields['user_nick_name'] = '中文名字';
request.fields['user_password'] = _password;
But the server side in the user_nick_name field always got null, note that is always, but I change it into English the server can receive that. I test on postman, the server can got Chinese correctly, so it's MultipartRequest issue on this problem.
My question is: Why the Dart or Flutter team so careless on this so important basic library? They even not consider about this simply issue. I opened a issue on github but no-one response, I think the team is done. So I ask the develop communit here, how to solve this problem anyway?
[UPDATE]
As kindly people suggested, I update my golang server now, if anyone else got this problem, you may wonna answer and suggestions too.
func HandleUserRegister(context *gin.Context) {
userAcc := context.PostForm("user_acc")
userAvatar := context.PostForm("user_avatar")
userNickName := context.PostForm("user_nick_name")
userPassword := context.PostForm("user_password")
userPhone := context.PostForm("user_phone")
userEmail := context.PostForm("user_email")
userGender := context.PostForm("user_gender")
userSign := context.PostForm("user_sign")
userType := context.PostForm("user_type")
userTypeInt, _ := strconv.Atoi(userType)
log.Infof("userAcc: %s, userNickName: %s, userPassword: %s", userAcc, userNickName, userPassword)}
This is based on gin, and this function is the api solver. If anyone wanna help, please help me figure it out.
OK! I update the question now, because it's really weird!. I did those test:
Post multiform via Flutter to Django server, it receives Chinese filed correctly;
Post multiform data via Postman, the golang(gin) server gots Chinese correctly;
Post multiform data via Flutter to golang(gin) server gots Chinese field null;
For more detail, I log the headers from my server for both postman(normal) and flutter (abnormal):
Postman:
request header: map[Content-Type:[multipart/form-data; boundary=--------------------------022341683711652813100488] Postman-Token:[855646d7-5bea-4b8f-b8df-81366226cd49] User-Agent:[PostmanRuntime/7.1.1] Content-Length:[422] Connection:[keep-alive] Cache-Control:[no-cache] Accept:[*/*] Accept-Encoding:[gzip, deflate]]
Flutter:
request header: map[User-Agent:[Dart/2.0 (dart:io)] Content-Type:[multipart/form-data; boundary=dart-http-boundary-.XUeYeqXpg4Yfyh8QhH1T5JB4zi_f3WxX9t7Taxhw91EFqhyki4] Accept-Encoding:[gzip] Content-Length:[574]]
Does anyone can notice the difference and let me know how to change the it make server can receive the Chinese Characters?
#DannyTuppeny is correct. This is a server problem.
When asked to include a non-ASCII field into a multi-part request, the Dart library correctly wraps this with a binary content-transfer-encoding.
String _headerForField(String name, String value) {
var header =
'content-disposition: form-data; name="${_browserEncode(name)}"';
if (!isPlainAscii(value)) {
header = '$header\r\n'
'content-type: text/plain; charset=utf-8\r\n'
'content-transfer-encoding: binary';
}
return '$header\r\n\r\n';
}
(Postman does not and simply sends the utf8 encoded string without any headers.)
Dart/ASCII looks like this:
--dart-http-boundary-HjDS88CmQicdgd8VaHSwPqJK8iR4H6rTG3LovSZy-QXGpU7pAB0
content-disposition: form-data; name="test"
stackover
--dart-http-boundary-HjDS88CmQicdgd8VaHSwPqJK8iR4H6rTG3LovSZy-QXGpU7pAB0
Dart/non-ASCII looks like this:
First boundary: --dart-http-boundary-58NU6u6_Fo22xjH8H7yPCtKuoKgB+A8+RTJ82iIK1gs3nnGMLlp\r\n
Encapsulated multipart part: (text/plain)
content-disposition: form-data; name="test"\r\n
content-type: text/plain; charset=utf-8\r\n
content-transfer-encoding: binary\r\n\r\n
Line-based text data: text/plain
\344\270\255\346\226\207\345\220\215\345\255\227
Boundary: \r\n--dart-http-boundary-58NU6u6_Fo22xjH8H7yPCtKuoKgB+A8+RTJ82iIK1gs3nnGMLlp\r\n
So the problem is that the server is unable to unwrap the value from the encapsulation.
EDIT
Here's the Postman trace I captured yesterday. It's multi-form, but fails to add the content-type-encoding header despite the field being non-ASCII.
MIME Multipart Media Encapsulation, Type: multipart/form-data, Boundary: "--------------------------595246000077585285134204"
[Type: multipart/form-data]
First boundary: ----------------------------595246000077585285134204\r\n
Encapsulated multipart part:
Content-Disposition: form-data; name="name"\r\n\r\n
Data (12 bytes)
0000 e4 b8 ad e6 96 87 e5 90 8d e5 ad 97 ............
Data: e4b8ade69687e5908de5ad97
[Length: 12]
Last boundary: \r\n----------------------------595246000077585285134204--\r\n
I tested by posting to httpbin and the response suggests that the characters were posted correctly:
"user_nick_name":"\u4e2d\u6587\u540d\u5b57"
I tried with both the Stable v1 SDK and a v2 SDK from Flutter. Is it possible the issue is on the server? Have you tried using something like Fiddler to capture what's actually being sent?
Edit: My guess is that your server side code is not correctly reading the data as MultipartForm data (eg. you should be using ParseMultipartForm and reading from MultipartForm).
The problem, it appears, is in formdata.go part of multipart. Go assumes that any multipart part with an Content-Type header is a file (not a field). However, knowing this you can change your server code as follows:
func main() {
r := gin.Default()
r.POST("/sotest", func(c *gin.Context) {
formValue := c.PostForm("form_value")
if formValue == "" {
formFile, _ := c.FormFile("form_value")
file, _ := formFile.Open()
b1 := make([]byte, formFile.Size)
file.Read(b1)
formValue = string(b1)
}
c.JSON(200, gin.H{
"status": "posted",
"formValue": formValue,
})
})
r.Run() // listen and serve on 0.0.0.0:8080
}
When you detect that PostForm returns the empty string, you know that Go has treated the field as a file, in which case you can Open and Read the 'file' and decode it as the utf-8 string that we know it is. Obviously, you could encapsulate the "try as PostForm and if that's empty, try as FormFile" test into a function.
If you don't want to have to test for empty string at the server, you could change your Dart end code to always utf-8 encode even non-ascii strings with
request.files.add(
new http.MultipartFile.fromBytes(
'some_form_value_name',
utf8.encode('the string value'),
contentType: new MediaType('text', 'plain', {'charset': 'utf-8'}),
),
);
and read them at the server with the Open/Read/string method.
I have now solved this. Thanks to Richard and Danny for their help.
1. Reason for this
No matter what happens but this really not only one-side problem, we can not say it's Flutter or Go wrong. But the combination, Flutter + Go server just may be got this issue. The behind reason I still not quit sure, but it must some head not right set (postman can do it right);
2. Solution
We don't only need know why but also how to solve it. Here is what I do:
Do not use the official http package. Using dio, which is a extension Dart package. link: https://pub.dartlang.org/packages/dio
It's more clean and easy to use, so my code becomes to:
FormData _formData = new FormData.from({
"user_acc": _userAcc,
"user_nick_name": _userNickName,
'user_password': _password,
});
Dio dio = new Dio();
Response response = await dio.post(usersUrl, data: _formData);
print(response.data);
I can not post the none-English words now:
INFO[0668] userAcc: ww, userNickName: 小鹿叮叮婴儿湿巾手口专用80抽湿纸巾婴儿湿巾婴儿100抽带盖批发【原价】34.90元【券后】9.9元【省】25元【复制此信息打开手机淘宝即可查看并下单】￥Tnsx0E77pFs￥【必买理由】新品预售80抽*3仙女联盟，更多优惠fd.loliloli.pro , userPassword: ww
INFO[0671] user exist.
RequestLogHandler requestLogHandler = new RequestLogHandler();
Slf4jRequestLog requestLog = new CustomSlf4jRequestLog();
requestLogHandler.setRequestLog(requestLog);
Slf4jRequestLog is only logging request method, url and date, and response status code and bytes written.
I definitely want to log body for my PUT/POST requests.
I derived CustomSlf4jRequestLog from Slf4jRequestLog and I tried:
public void log(Request request, Response response) {
StringBuilder sb = new StringBuilder("RequestBody: ");
try {
LOG.info("BODY SIZE: " + request.getContentLength());
BufferedReader in = new BufferedReader(new InputStreamReader(request.getInputStream()));
String line;
while ((line = in.readLine()) != null) {
sb.append(line);
}
LOG.info(sb.toString());
Unfortunately no body is printed out as it is already processed by handler?
Is it possible to get body of request here?
(I really care about body because I have JsonProvider and I want to see a whole body of request when Json fails to parse data) Or when my app fails I want to see what caused that without adding logging for each input request.
Servlet spec getting in your way here.
The actual servlet that is processing the request has already read the request body, rending further reads from request.getInputStream() invalid (you are at EOF)
If you want to capture the request body, you'll need to create a custom Servlet Filter, and then wrap the HttpServletRequest, overriding the getInputStream() AND getReader() methods with your own implementations that make a copy of the data that was read.
Then its up to you to determine what you want to do with that copy of the request body content.
or
You can just use a network capture utility like Wireshark to see what the request body was. Even if you use HTTPS, you can configure Wireshark with your server certificate to inspect encrypted conversation.
The logger is calling getInputStream() on the same request again. You are not allowed to read the same data twice. Yo should create a ServletRequestWrapper to make a copy of the body of the request.
I tried sending a HTTP Get to a device's Restful API via Postman and it worked fine returning all text I was expecting. Postman suggested Ruby code for the request was the following:
url = URI('http://192.168.1.5/rest/op/BD1FD3D893613E79')
http = Net::HTTP.new(url.host, url.port)
request = Net::HTTP::Get.new(url)
request.basic_auth 'admin', 'admin'
request["accept"] = 'Application/json'
response = http.request(request)
puts response.read_body
but when I tried that in my code, it is returning a truncated response (missing lines) and I have to resend the same Get multiple times to get the entire text response response.
Is there anything missing in the Ruby code above that's causing this truncated response?
Update 1
I tried this
url = URI('http://192.168.1.5/rest/op/BD1FD3D893613E79')
http = Net::HTTP.new(url.host, url.port)
request = Net::HTTP::Get.new(url)
request.basic_auth 'admin', 'admin'
request["accept"] = 'Application/json'
response = http.request(request)
puts response.read_body
response.read_body do |segment|
puts segment.to_s
end
and that generated this error
IOError (Net::HTTPOK#read_body called twice):
Update 2
I tried this
1073 url = URI('http://192.168.1.5/rest/op/BD1FD3D893613E79')
1074 http = Net::HTTP.new(url.host, url.port)
1075 request = Net::HTTP::Get.new(url.to_s)
1076 request.basic_auth 'admin', 'admin'
1077 request["accept"] = 'Application/json'
1078 response = http.request(request)
1079 response.read_body do |segment|
1080 puts segment.to_s
1081 end
and got this error
IOError (Net::HTTPOK#read_body called twice):
app/controllers/Apps_controller.rb:1079:in `block in get_config'
app/controllers/Apps_controller.rb:1045:in `each'
app/controllers/Apps_controller.rb:1045:in `get_config'
Based on: http://ruby-doc.org/stdlib-1.9.3/libdoc/net/http/rdoc/Net/HTTPResponse.html#method-i-read_body
read_body returns the body as stream, so you should iterate it, something like:
response.read_body do |segment|
puts segment
end
If you want to get the full body just use: response.body
Looks very similar....
We are getting truncated responses when pulling a set of artifacts (zip files) from a Bamboo server and writing to local storage. It happens inconsistently, on only one of 8 servers, and not always the same file. Seems always to be the last portion of the file that is missing.
Using Ruby 2.0.0p598 and net/http.
The zip files range in size from 2 mb to 34 mb.
While we haven't figured out why the response body has been truncated, our workaround is to compare the expected content length (in the response header), to the response body size. If they don't match, try it again. Example code:
package = "packages/applicationname.zip"
def retrieve(package)
# generates the Bamboo url for the given package
sourceURL = package_url package
expectedLength = 0
bodyLength = 1
# Try to catch the bad download and re-issue the GET request,
while expectedLength != bodyLength do
# add a counter if worried about getting stuck
# issue the request to get the current package zip file
response = my_request_wrapper sourceURL
expectedLength = response['content-length'].to_i
theBody = response.body
bodyLength = theBody.size
if expectedLength != bodyLength then
puts "!! SIZE MISMATCH !!"
else
# the response body is good, process as needed
open package, 'wb' do |io|
io.write theBody
end
end
end
end
def my_request_wrapper(url)
uri = URI.parse(url)
http = Net::HTTP.new(uri.host, uri.port)
http.use_ssl = true
http.verify_mode = OpenSSL::SSL::VERIFY_NONE
req = Net::HTTP::Get.new(uri.request_uri)
req.basic_auth("user", "pwd")
return http.request req
end
I was using OSX Sierra, rvm, and ruby 2.3.0
Updating my rvm, ruby, and gems seems to have fixed the problem for me
I think you must call http.start before you call http.request.
Not sure why, but I see the same.