I have a controller action that returns a CSV document produced asynchronously by a data source. I subscribed to an observable data stream and I write to the http response with Response.WriteAsync(). It seems to work apart of the fact that the terminating (empty) chunk is missing at the end of the response (The missing data in hex is 30 0d 0a 0d 0a for the terminating 0\r\n\r\n).
I have few questions:
- is it okay to use WriteAsync in this scenario - call it multiple times as the data comes in - i certainly do not want to buffer the whole dataset and return it at once as it can grow huge
- whose responsibility it is to write the terminating chunk (should it write it explicitly in OnComplete handler with WriteAsync as well?)
example response from an action that returns a json object (note the terminator at the end of the response):
HTTP/1.1 200 OK
Date: Tue, 05 Feb 2019 09:25:23 GMT
Server: Kestrel
Transfer-Encoding: chunked
21
{"results":[{"statement_id":0}]}
0
Response from my action:
HTTP/1.1 200 OK
Date: Tue, 05 Feb 2019 09:27:42 GMT
Content-Type: text/csv
Server: Kestrel
Transfer-Encoding: chunked
47
ID,Timestamp,BadQualityCount,MAX_value,MEAN_value,MIN_value,TotalPoints
65
...
5d
SwitchStatus_On_a797c2c2-de78-4fe8-9e4b-5d64f51d1e00,2019-0204T11:52:17.3680000Z,0,0,0,0,1
Controller code:
using (var disposable = observableContent.Subscribe(
async current =>
await Response.WriteAsync(current),
() =>
{
//Response.WriteAsync(Environment.NewLine);
//Response.WriteAsync("0");
Response.Body.Flush();
completed = true;
}
))
SpinWait.SpinUntil(() => completed);
return Ok();
It turned out the terminator is written correctly if the action signature is changed from Task to Task.
Related
I'm attempting to test my REST API by making a POST request to a MongoDB cloud Atlas DB server. I know Postman is available, but I wanted to use something different like Httpie. I already checked this question but I'm still stuck.
How to send a POST request using HTTPie?
I'm trying to get text='john smith'
when I use
`http -f POST :5000/api/posts text='john smith'`
I get this response.
`HTTP/1.1 201 Created
Access-Control-Allow-Origin: *
Connection: keep-alive
Content-Length: 0
Date: Tue, 19 Feb 2019 20:33:36 GMT
X-Powered-By: Express`
But when I use...
http -f GET :5000/api/posts
I get back ...
`[
{
"_id": "5c6c6820c2f6eb15ea9e8e08",
"createdAt": "2019-02-19T20:33:36.468Z",
"text": null
}
]`
This is my Nodejs API for the post
router.post('/', async(req, res) => {
const posts = await loadPostCollection();
await posts.insertOne({
text: req.body.text,
createdAt: new Date()
});
res.status(201).send();
});
I recently discovered the dataforseo api and tryed to call it via R
library(httr)
username <- 'mygmailadress#gmail.com'
password <- 'mypassword'
dataforseo_api <- POST('https://api.dataforseo.com/v2/op_tasks_post/$data',
authenticate(username,password),
body = list(grant_type = 'client_credentials'),
type = "basic",
verbose()
)
This is the message I have received:
<- HTTP/1.1 401 Unauthorized
<- Server: nginx/1.14.0 (Ubuntu)
<- Date: Sun, 08 Jul 2018 13:31:34 GMT
<- Content-Type: application/json
<- Transfer-Encoding: chunked
<- Connection: keep-alive
<- WWW-Authenticate: Basic realm="Rest Server"
<- Cache-Control: no-cache, must-revalidate
<- Expires: 0
<- Access-Control-Allow-Origin: *
<- Access-Control-Allow-Methods: POST, GET, OPTIONS
<- Access-Control-Allow-Headers: Content-Type, Access-Control-Allow-Headers, Authorization, X-Requested-With
Do you know where my issue should come? Can you please help?
It looks like you're improperly configuring config. I don't see a config= in your code. The body is also not encoded correctly.
Also, in the API documentation I don't see anything about grant_type. It looks like an array of tasks should go there, e.g. something like:
{882394209: {'site': 'ranksonic.com', 'crawl_max_pages': 10}}
Response:
{'results_count': 1, 'results_time': '0.0629 sec.', 'results': {'2308949': {'post_id': 2308949, 'post_site': 'ranksonic.com',
'task_id': 882394209, 'status': 'ok'}}, 'status': 'ok'}
OK, so first off we need set_config or config=:
username <- 'Hack-R#stackoverflow.com' # fake email
password <- 'vxnyM9s7FAKESeIO' # fake password
set_config(authenticate(username,password), override = TRUE)
GET("https://api.dataforseo.com/v2/cmn_se")
Response [https://api.dataforseo.com/v2/cmn_se]
Date: 2018-07-08 16:20
Status: 200
Content-Type: application/json
Size: 551 kB
{
"status": "ok",
"results_time": "0.0564 sec.",
"results_count": 2187,
"results": [
{
"se_id": 37,
"se_name": "google.com.af",
"se_country_iso_code": "AF",
"se_country_name": "Afghanistan",
...
GET("https://api.dataforseo.com/v2/cmn_se/$country_iso_code")
Response [https://api.dataforseo.com/v2/cmn_se/$country_iso_code]
Date: 2018-07-08 15:48
Status: 200
Content-Type: application/json
Size: 100 B
{
"status": "ok",
"results_time": "0.0375 sec.",
"results_count": 0,
"results": []
GET("https://api.dataforseo.com/v2/cmn_se/$op_tasks_post")
Response [https://api.dataforseo.com/v2/cmn_se/$op_tasks_post]
Date: 2018-07-08 16:10
Status: 200
Content-Type: application/json
Size: 100 B
{
"status": "ok",
"results_time": "0.0475 sec.",
"results_count": 0,
"results": []
That was one thing. Also to POST data they need you to specify it as json, e.g. encode = "json". From their docs:
All POST data should be sent in the JSON format (UTF-8 encoding). The
keywords are sent by POST method passing tasks array. The data should
be specified in the data field of this POST array. We recommend to
send up to 100 tasks at a time.
Further:
The task setting is done using POST method when array of tasks is sent to
the data field. Each of the array elements has the following
structure:
then it goes on to list 2 required fields and many optional ones.
Note also that you can use reset_config() after as a better practice. If you're going to be running this a lot, sharing it, or using more than 1 computer I would also suggest to put your credentials in environment variables instead of your script for security and ease.
Another final word of advice is that you may want to just leverage their published Python client library and large compilation of examples. Since every new API request is something you'll be pioneering in R without their support, it may pay off to just do the data collection in Python.
This is an interesting API. If you get over to the Open Data Stack Exchange you should consider sharing it with that community.
I can't unserstand what I am doing wrong, but when I am sending next request with curl, I am getting error:
echo {"id":1,"question":"aaa"},{"id":2,"question":"bbb?"} | curl -X POST --data-binary #- --dump - http://localhost:8529/_db/otest/_api/document/?collection=sitetestanswers
HTTP/1.1 100 (Continue)
HTTP/1.1 400 Bad Request
Server: ArangoDB
Connection: Keep-Alive
Content-Type: application/json; charset=utf-8
Content-Length: 100
{"error":true,"errorMessage":"failed to parse json object: expecting EOF","code":400,"errorNum":600}
Any ideas? I tied wrap it's to [...]. Nothing do not help.
With [...] validator mark this as valid
Same with D. Here is my code:
void sendQuestionsToArangoDB(Json questions)
{
string collectionUrl = "http://localhost:8529/_db/otest/_api/document/?collection=sitetestanswers";
auto rq = Request();
rq.verbosity = 2;
string s = `{"id":"1","question":"foo?"},{"id":2}`;
auto rs = rq.post(collectionUrl, s, "application/json");
writeln("SENDED");
}
--
POST /_db/otest/_api/document/?collection=sitetestanswers HTTP/1.1
Content-Length: 37
Connection: Close
Host: localhost:8529
Content-Type: application/json
HTTP/1.1 400 Bad Request
Server: ArangoDB
Connection: Close
Content-Type: application/json; charset=utf-8
Content-Length: 100
100 bytes of body received
For D I use this lib: https://github.com/ikod/dlang-requests
Same issue with vibed.
ArangoDB do not understand JSON if it's come ass array like [...]. It should be passed as key-value. So if you need pass array it should have key mykey : [].
Here is working code:
import std.stdio;
import requests.http;
void main(string[] args)
{
string collectionUrl = "http://localhost:8529/_db/otest/_api/document?collection=sitetestanswers";
auto rq = Request();
rq.verbosity = 2;
string s = `{"some_data":[{"id":1, "question":"aaa"},{"id":2, "question":"bbb"}]}`;
auto rs = rq.post(collectionUrl, s, "application/json");
writeln("SENDED");
}
otest - DB name
sitetestanswers - collection name (should be created in DB)
echo '[{"id":1,"question":"aaa"},{"id":2,"question":"bbb?"}]'
should do the trick. You need to put ticks around the JSON. The array brackets are necessary otherwise this is not valid JSON.
You are trying to send multiple documents. The data in the original question separates the documents by comma ({"id":1,"question":"aaa"},{"id":2,"question":"bbb?"}) which is invalid JSON. Thus the failed to parse json object answer from ArangoDB.
Putting the documents into angular brackets ([ ... ]) as some of the commentors suggested will make the request payload valid JSON again.
However, you're sending the data to a server endpoint that handles a single document. The API for POST /_api/document/?collection=... currently accepts a single document at a time. It does not work with multiple documents in a single request. It expects a JSON object, and whenever it is sent something different it will respond with an error code.
If you're looking for batch inserts, please try the API POST /_api/import, described in the manual here: https://docs.arangodb.com/HttpBulkImports/ImportingSelfContained.html
This will work with multiple documents in a single request. ArangoDB 3.0 will also allow sending multiple documents to the POST /_api/document?collection=... API, but this version is not yet released. A technical preview will be available soon however.
This is my first day to have fun with Symfony and drupal 8, so please excuse me if my question is very obvious.
With drupal 7:
drupal_json_output(array('products' => array_values($products)));
exit;
the json output is clean:
{"products":["item_1","item_2",....]}
With drupal 8:
use Symfony\Component\HttpFoundation\JsonResponse;
// some process
print new JsonResponse(array('products' => array_values($products)));
exit;
It outputs with the headers:
HTTP/1.0 200 OK
Cache-Control: no-cache
Content-Type: application/json
Date: Wed, 18 Jul 2012 07:53:26 GMT
{"products":["item_1","item_2",....]}
How do you get rid of those headers?
I am stuck to read the reference here.
Any hint is very much appreciated.
You can get only the "content" of a response by calling $response->getContent().
In your case you could do
use Symfony\Component\HttpFoundation\JsonResponse;
// some process
$response = new JsonResponse(array('products' => array_values($products)));
print $response->getContent();
exit;
However, be aware that this would be a bad practice because you would lose the response headers in the process, and wouldn't tell for example, what the content-type of you response is (in this case: "application/json") etc ...
I do not know how to do this properly with drupal, any tips is appreciated.
I have created my own HTTP server. I need to return a PDF file (generated by Jasper Reports) to the web browser. However, when I read the PDF file and write its contents to the socket, the web browser receives a blank PDF file. When I save this file and compare it to the original, I see that many of the characters have been converted from their original value to 0x3F (which is '?').
When I read the file, my debug output shows that the correct values are read and that the correct values are written to the socket. Can anyone help me?
Here is the code (minus all the debug code) that reads the PDF file:
File f = new File(strFilename);
long len = f.length();
byteBuffPdfData = ByteBuffer.allocate( (int)len );
FileInputStream in = new FileInputStream(strFilename);
boolean isEOF = false;
while (!isEOF)
{
int iValue = in.read();
if (iValue == -1)
{
isEOF = true;
}
else
{
byteBuffPdfData.put( (byte)iValue );
}
}
Next is the code that writes from the byte buffer to the socket...
printWriter = new PrintWriter( socket.getOutputStream(), true );
printWriter.write(strHttpHeaders);
// Headers:
// HTTP/1.0 200 OK
// Date: Wed, 18 Nov 2009 21:04:36
// Expires: Wed, 18 Nov 2009 21:09:36
// Cache-Control: public
// Content-Type: application/pdf
// Content-Length: 1811
// Connection: keep-alive
//
byteBuffPdfData.rewind();
while(byteBuffPdfData.hasRemaining())
{
printWriter.print( (char)byteBuffPdfData.get() );
}
printWriter.flush();
socket.close();
Any help that can be offered is greatly appreciated. I am sure that I need to do something with the character sets but at this point I have tried a million things and nothing seems to work.
John
I'm not sure what language you're writing in here (looks like Java, though), but is it possible something is trying to do a charset conversion (perhaps to or from Unicode characters)? The ? seems reasonable as a 'substitution' for characters that can't be represented in ASCII. (I recognize that you aren't deliberately trying to do any such conversion, but maybe something of the sort is happening in the libraries you're using.)