I'm using Ocaml CURL to make http request to a server. I try to send it using the method post and I found this function (https://gist.github.com/zbroyar/1432555) :
let post ?(content_type = "text/html") url data =
let r,c = init_conn url in
Curl.set_post c true;
Curl.set_httpheader c [ "Content-Type: " ^ content_type ];
Curl.set_postfields c data;
Curl.set_postfieldsize c (String.length data);
Curl.perform c;
let rc = Curl.get_responsecode c in
Curl.cleanup c;
rc, (Buffer.contents r)
I understand that the value data contains the post values but in which form ? It should be "login=foo pass=bar" or "login=foo, pass=bar" ? Anyone has an idea ?
This is a POST request, it can send any kind of data using any form, that's why there is a Content-Type header to tell the server how to decode the data.
By default, this function will send HTML, if you want to use the form encoding, replace "text/html" with "application/x-www-form-urlencoded".
Related
I had set Content-Type in RequestSpecBuilder as "ContentType.JSON". But on making a GET request, I get Content-Type as "application/xml" in response. How do i get back a json response?
I have tried below approaches:
1. Set content type in RequestSpecBuilder object using setContentType method of RequestSpecBuilder class to "ContentType.JSON" and pass RequestSpecBuilder object in spec method of RequestSpecification --- got "application/xml" in response
Set content type in RequestSpecification object using contentType method of RequestSpecification and pass ContentType.JSON as parameter --- still got "application/xml" in response
Note: The webservice URL requires ".json" to be explicitly specified to get a json response else by default it returns a "xml" response. However, I wanted to set content type by using RequestSpecBuilder.
Eg:
for Json response: URL -- http://ergast.com/api/f1/2017/circuits.json
for Xml response: URL -- http://ergast.com/api/f1/2017/circuits
Code:
#Test
public void test_AddHeader() {
//Use of RequestSpecification
String pathUrl = "http://ergast.com/api/f1/2017/circuits";
RequestSpecBuilder requestSpecBuilder = new RequestSpecBuilder();
requestSpecBuilder = requestSpecBuilder.
setBaseUri(pathUrl).
setContentType(ContentType.JSON).
addQueryParam("limit", "10"); //added query param
RequestSpecification addRequestSpec = requestSpecBuilder.build();
RequestSpecification httpRequest = RestAssured.given().spec(addRequestSpec).contentType(ContentType.JSON);
Response httpResponse = httpRequest.get();
System.out.println(httpResponse.getContentType()); //returns application/xml
System.out.println(httpResponse.getStatusLine()); //returns HTTP/1.1 200 OK
System.out.println(httpResponse.getBody().asString());//returns XML response
}
You are expecting JSON from Response but you are passing setContentType to your RequestSpecBuilder. This will just create your POST payload in json format. It does not do anything to your response.
What you can do instead is Create a ResponseBuilder and do a setContentType to JSON there. Hope this will help you.
I try to post a form to server and here is the code:
ar request = new http.MultipartRequest("POST", _uri);
request.fields['user_acc'] = _userAcc;
// this issue should be solve
request.fields['user_nick_name'] = '中文名字';
request.fields['user_password'] = _password;
But the server side in the user_nick_name field always got null, note that is always, but I change it into English the server can receive that. I test on postman, the server can got Chinese correctly, so it's MultipartRequest issue on this problem.
My question is: Why the Dart or Flutter team so careless on this so important basic library? They even not consider about this simply issue. I opened a issue on github but no-one response, I think the team is done. So I ask the develop communit here, how to solve this problem anyway?
[UPDATE]
As kindly people suggested, I update my golang server now, if anyone else got this problem, you may wonna answer and suggestions too.
func HandleUserRegister(context *gin.Context) {
userAcc := context.PostForm("user_acc")
userAvatar := context.PostForm("user_avatar")
userNickName := context.PostForm("user_nick_name")
userPassword := context.PostForm("user_password")
userPhone := context.PostForm("user_phone")
userEmail := context.PostForm("user_email")
userGender := context.PostForm("user_gender")
userSign := context.PostForm("user_sign")
userType := context.PostForm("user_type")
userTypeInt, _ := strconv.Atoi(userType)
log.Infof("userAcc: %s, userNickName: %s, userPassword: %s", userAcc, userNickName, userPassword)}
This is based on gin, and this function is the api solver. If anyone wanna help, please help me figure it out.
OK! I update the question now, because it's really weird!. I did those test:
Post multiform via Flutter to Django server, it receives Chinese filed correctly;
Post multiform data via Postman, the golang(gin) server gots Chinese correctly;
Post multiform data via Flutter to golang(gin) server gots Chinese field null;
For more detail, I log the headers from my server for both postman(normal) and flutter (abnormal):
Postman:
request header: map[Content-Type:[multipart/form-data; boundary=--------------------------022341683711652813100488] Postman-Token:[855646d7-5bea-4b8f-b8df-81366226cd49] User-Agent:[PostmanRuntime/7.1.1] Content-Length:[422] Connection:[keep-alive] Cache-Control:[no-cache] Accept:[*/*] Accept-Encoding:[gzip, deflate]]
Flutter:
request header: map[User-Agent:[Dart/2.0 (dart:io)] Content-Type:[multipart/form-data; boundary=dart-http-boundary-.XUeYeqXpg4Yfyh8QhH1T5JB4zi_f3WxX9t7Taxhw91EFqhyki4] Accept-Encoding:[gzip] Content-Length:[574]]
Does anyone can notice the difference and let me know how to change the it make server can receive the Chinese Characters?
#DannyTuppeny is correct. This is a server problem.
When asked to include a non-ASCII field into a multi-part request, the Dart library correctly wraps this with a binary content-transfer-encoding.
String _headerForField(String name, String value) {
var header =
'content-disposition: form-data; name="${_browserEncode(name)}"';
if (!isPlainAscii(value)) {
header = '$header\r\n'
'content-type: text/plain; charset=utf-8\r\n'
'content-transfer-encoding: binary';
}
return '$header\r\n\r\n';
}
(Postman does not and simply sends the utf8 encoded string without any headers.)
Dart/ASCII looks like this:
--dart-http-boundary-HjDS88CmQicdgd8VaHSwPqJK8iR4H6rTG3LovSZy-QXGpU7pAB0
content-disposition: form-data; name="test"
stackover
--dart-http-boundary-HjDS88CmQicdgd8VaHSwPqJK8iR4H6rTG3LovSZy-QXGpU7pAB0
Dart/non-ASCII looks like this:
First boundary: --dart-http-boundary-58NU6u6_Fo22xjH8H7yPCtKuoKgB+A8+RTJ82iIK1gs3nnGMLlp\r\n
Encapsulated multipart part: (text/plain)
content-disposition: form-data; name="test"\r\n
content-type: text/plain; charset=utf-8\r\n
content-transfer-encoding: binary\r\n\r\n
Line-based text data: text/plain
\344\270\255\346\226\207\345\220\215\345\255\227
Boundary: \r\n--dart-http-boundary-58NU6u6_Fo22xjH8H7yPCtKuoKgB+A8+RTJ82iIK1gs3nnGMLlp\r\n
So the problem is that the server is unable to unwrap the value from the encapsulation.
EDIT
Here's the Postman trace I captured yesterday. It's multi-form, but fails to add the content-type-encoding header despite the field being non-ASCII.
MIME Multipart Media Encapsulation, Type: multipart/form-data, Boundary: "--------------------------595246000077585285134204"
[Type: multipart/form-data]
First boundary: ----------------------------595246000077585285134204\r\n
Encapsulated multipart part:
Content-Disposition: form-data; name="name"\r\n\r\n
Data (12 bytes)
0000 e4 b8 ad e6 96 87 e5 90 8d e5 ad 97 ............
Data: e4b8ade69687e5908de5ad97
[Length: 12]
Last boundary: \r\n----------------------------595246000077585285134204--\r\n
I tested by posting to httpbin and the response suggests that the characters were posted correctly:
"user_nick_name":"\u4e2d\u6587\u540d\u5b57"
I tried with both the Stable v1 SDK and a v2 SDK from Flutter. Is it possible the issue is on the server? Have you tried using something like Fiddler to capture what's actually being sent?
Edit: My guess is that your server side code is not correctly reading the data as MultipartForm data (eg. you should be using ParseMultipartForm and reading from MultipartForm).
The problem, it appears, is in formdata.go part of multipart. Go assumes that any multipart part with an Content-Type header is a file (not a field). However, knowing this you can change your server code as follows:
func main() {
r := gin.Default()
r.POST("/sotest", func(c *gin.Context) {
formValue := c.PostForm("form_value")
if formValue == "" {
formFile, _ := c.FormFile("form_value")
file, _ := formFile.Open()
b1 := make([]byte, formFile.Size)
file.Read(b1)
formValue = string(b1)
}
c.JSON(200, gin.H{
"status": "posted",
"formValue": formValue,
})
})
r.Run() // listen and serve on 0.0.0.0:8080
}
When you detect that PostForm returns the empty string, you know that Go has treated the field as a file, in which case you can Open and Read the 'file' and decode it as the utf-8 string that we know it is. Obviously, you could encapsulate the "try as PostForm and if that's empty, try as FormFile" test into a function.
If you don't want to have to test for empty string at the server, you could change your Dart end code to always utf-8 encode even non-ascii strings with
request.files.add(
new http.MultipartFile.fromBytes(
'some_form_value_name',
utf8.encode('the string value'),
contentType: new MediaType('text', 'plain', {'charset': 'utf-8'}),
),
);
and read them at the server with the Open/Read/string method.
I have now solved this. Thanks to Richard and Danny for their help.
1. Reason for this
No matter what happens but this really not only one-side problem, we can not say it's Flutter or Go wrong. But the combination, Flutter + Go server just may be got this issue. The behind reason I still not quit sure, but it must some head not right set (postman can do it right);
2. Solution
We don't only need know why but also how to solve it. Here is what I do:
Do not use the official http package. Using dio, which is a extension Dart package. link: https://pub.dartlang.org/packages/dio
It's more clean and easy to use, so my code becomes to:
FormData _formData = new FormData.from({
"user_acc": _userAcc,
"user_nick_name": _userNickName,
'user_password': _password,
});
Dio dio = new Dio();
Response response = await dio.post(usersUrl, data: _formData);
print(response.data);
I can not post the none-English words now:
INFO[0668] userAcc: ww, userNickName: 小鹿叮叮婴儿湿巾手口专用80抽湿纸巾婴儿湿巾婴儿100抽带盖批发【原价】34.90元【券后】9.9元【省】25元【复制此信息打开手机淘宝即可查看并下单】￥Tnsx0E77pFs￥【必买理由】新品预售80抽*3仙女联盟，更多优惠fd.loliloli.pro , userPassword: ww
INFO[0671] user exist.
I'm trying to use the Groovy HTTPBuilder library to delete some data from Firebase via a HTTP DELETE request. If I use curl, the following works
curl -X DELETE https://my.firebase.io/users/bob.json?auth=my-secret
Using the RESTClient class from HTTPBuilder works if I use it like this:
def client = new RESTClient('https://my.firebase.io/users/bob.json?auth=my-secret')
def response = client.delete(requestContentType: ContentType.ANY)
However, when I tried breaking down the URL into it's constituent parts, it doesn't work
def client = new RESTClient('https://my.firebase.io')
def response = client.delete(
requestContentType: ContentType.ANY,
path: '/users/bob.json',
query: [auth: 'my-secret']
)
I also tried using the HTTPBuilder class instead of RESTClient
def http = new HTTPBuilder('https://my.firebase.io')
// perform a POST request, expecting TEXT response
http.request(Method.DELETE, ContentType.ANY) {
uri.path = '/users/bob.json'
uri.query = [auth: 'my-secret']
// response handler for a success response code
response.success = { resp, reader ->
println "response status: ${resp.statusLine}"
}
}
But this also didn't work. Surely there's a more elegant approach than stuffing everything into a single string?
There's an example of using HttpURLClient in the tests to do a delete, which in its simplest form looks like:
def http = new HttpURLClient(url:'https://some/path/')
resp = http.request(method:DELETE, contentType:JSON, path: "destroy/somewhere.json")
def json = resp.data
assert json.id != null
assert resp.statusLine.statusCode == 200
Your example is very close to the test for the delete in a HTTPBuilder.
A few differences I see are:
Your path is absolute and not relative
Your http url path doesn't end with trailing slash
You're using content type ANY where test uses JSON. Does the target need the content type to be correct? (Probably not as you're not setting it in curl example unless it's doing some voodoo on your behalf)
Alternatively you could use apache's HttpDelete but requires more boiler plate. For a HTTP connection this is some code I've got that works. You'll have to fix it for HTTPS though.
def createClient() {
HttpParams params = new BasicHttpParams()
HttpProtocolParams.setVersion(params, HttpVersion.HTTP_1_1)
HttpProtocolParams.setContentCharset(params, "UTF-8")
params.setBooleanParameter(ClientPNames.HANDLE_REDIRECTS, true)
SchemeRegistry registry = new SchemeRegistry()
registry.register(new Scheme("http", PlainSocketFactory.getSocketFactory(), 80))
ClientConnectionManager ccm = new PoolingClientConnectionManager(registry)
HttpConnectionParams.setConnectionTimeout(params, 8000)
HttpConnectionParams.setSoTimeout(params, 5400000)
HttpClient client = new DefaultHttpClient(ccm, params)
return client
}
HttpClient client = createClient()
def url = new URL("http", host, Integer.parseInt(port), "/dyn/admin/nucleus$component/")
HttpDelete delete = new HttpDelete(url.toURI())
// if you have any basic auth, you can plug it in here
def auth="USER:PASS"
delete.setHeader("Authorization", "Basic ${auth.getBytes().encodeBase64().toString()}")
// convert a data map to NVPs
def data = [:]
List<NameValuePair> nvps = new ArrayList<NameValuePair>(data.size())
data.each { name, value ->
nvps.add(new BasicNameValuePair(name, value))
}
delete.setEntity(new UrlEncodedFormEntity(nvps))
HttpResponse response = client.execute(delete)
def status = response.statusLine.statusCode
def content = response.entity.content
I adopted the code above from a POST version, but the principle is the same.
Is it possible to submit a Freebase mqlread request via POST in Python? I have tried to search for documentation but everything refers to GET. Thanks.
It is possible.
You will need issue a POST and add a specific header: X-HTTP-Method-Override: GET (basically tells the server to emulate a GET with the POST's content). Specifically for me I used the Content-Encoding: application/x-www-form-urlencode.
Here's the relevant part of my code (coffeescript) if it helps:
mqlread = (query, queryEnvelope, cb) ->
## build URL
url = urlparser.format
protocol: 'https'
host: 'www.googleapis.com'
pathname: 'freebase/v1/mqlread'
## build POST body
queryEnvelope ?= {}
queryEnvelope.key = config.GOOGLE_API_SERVER_KEY
queryEnvelope.query = JSON.stringify query
options =
url: url
method: 'POST'
headers:
'X-HTTP-Method-Override': 'GET'
'User-Agent': config.wikipediaScraperUserAgent
timeout: 3000
form: queryEnvelope
## invoke API
request options, (err, response, body) ->
if err then return cb err
if response.statusCode != 200
try
json = JSON.parse(body)
errmsg = json?.error?.message or "(unknown JSON)"
catch e
errmsg = body?[..50]
return cb "#{response.statusCode} #{errmsg}"
r = JSON.parse response.body
decodeStringsInResponse r
cb null, r
I don't think POST is supported for MQLread, but you could use the HTTP Batch facility.
Here's an example in Python:
https://github.com/tfmorris/freebase-python-samples/blob/master/client-library/mqlread-batch.py
I'm trying to figure out how the best way to easily send HTTP/HTTPS requests and to handle gzip/deflate compressed responses along with cookies.
The best I found was https://github.com/mikeal/request which handles everything except compression. Is there a module or method that will do everything I ask?
If not, can I combine request and zlib in some manner? I tried to combine zlib and http.ServerRequest, and it failed miserably.
For anyone coming across this in recent times, the request library supports gzip decompression out of the box now. Use as follows:
request(
{ method: 'GET'
, uri: 'http://www.google.com'
, gzip: true
}
, function (error, response, body) {
// body is the decompressed response body
console.log('server encoded the data as: ' + (response.headers['content-encoding'] || 'identity'))
console.log('the decoded data is: ' + body)
}
)
From the github readme https://github.com/request/request
gzip - If true, add an Accept-Encoding header to request compressed
content encodings from the server (if not already present) and decode
supported content encodings in the response. Note: Automatic decoding
of the response content is performed on the body data returned through
request (both through the request stream and passed to the callback
function) but is not performed on the response stream (available from
the response event) which is the unmodified http.IncomingMessage
object which may contain compressed data. See example below.
Note: as of 2019, request has gzip decompression built in. You can still decompress requests manually using the below method.
You can simply combine request and zlib with streams.
Here is an example assuming you have a server listening on port 8000 :
var request = require('request'), zlib = require('zlib');
var headers = {
'Accept-Encoding': 'gzip'
};
request({url:'http://localhost:8000/', 'headers': headers})
.pipe(zlib.createGunzip()) // unzip
.pipe(process.stdout); // do whatever you want with the stream
Here's a working example that gunzips the response
function gunzipJSON(response){
var gunzip = zlib.createGunzip();
var json = "";
gunzip.on('data', function(data){
json += data.toString();
});
gunzip.on('end', function(){
parseJSON(json);
});
response.pipe(gunzip);
}
Full code: https://gist.github.com/0xPr0xy/5002984
Check out the examples at http://nodejs.org/docs/v0.6.0/api/zlib.html#examples
zlib is now built into node.
Looking inside the source code - you must set the gzip param on the request lib itself for gzip to work. Not sure if this was intentional or not, but this is the current implementation. No extra headers are needed.
var request = require('request');
request.gzip = true;
request({url: 'https://...'}, // use encoding:null for buffer instead of UTF8
function(error, response, body) { ... }
);
All the answers here did not work and I was getting raw bytes back instead and the gzip flag was not working either. As it turns out you need to set the encoding to null to prevent requests from transforming the response to utf-8 encoding and instead keeps the binary response.
const request = require("request-promise-native");
const zlib = require("zlib");
const url = getURL("index.txt");
const dataByteBuffer = await request(url, { encoding: null });
const dataString = zlib.gunzipSync(response);