I'm playing with arduino's wifi shield and trying to strip http header out by searching for CRLF (\r\n) in my loop().
while (client.available()) {
char c = client.read();
// I need to check to see if it's crlf
// and parse the response.
}
What would be the easiest way to do this? I'm only interested in the response not the header. I thought about putting this in the buffer and look for current and previous character to match crlf (\r\n).
Suggestions? Thanks.
Try using the TextFinder library http://playground.arduino.cc/Code/TextFinder
You should probably consider using the existing HttpClient library or, perhaps, just take a look at how the parsing of HTTP response is implemented there.
Related
I'm trying to get live trading data from the Internet via HTTP, but it is updated continuously, so if I GET the data, it will keep downloading as long as there is data available. Until I stop the downloading stream, then I can access the data.
How to access the stream of data while the downloading is in progress?
I tried using Indy's TIdHTTP, so I can use SSL, but I tried the IdIOHandlerStream, but it was already used for IdSSLIOHandlerSocketOpenSSL. So I'm absolutely clueless here.
This is in response to a "multipart/form-data" request.
Please guide me...
Lrequest.Values['__RequestVerificationToken'] := RequestVerificationToken;
Lrequest.Values['acct'] := 'demo';
Lrequest.Values['pwd'] := 'demo';
try
Response.Text := Onhttp.Post('https://trading/data', Lrequest);
Form1.Memo1.Lines.Add(TimeToStr(Time) + ': ' + Response.Text);
except
on E: Exception do
Form1.Memo1.Lines.Add(TimeToStr(Time) + ': ' + E.ClassName +
' error raised, with message : ' + E.Message);
end;
UPDATE:
The data is an endless JSON string, like this:
{"id":"data","val":[{"rc":2,"tpc":"\\RealTime\\Global\\SGDIDR.FX","item":[{"val":{"F009":"10454.90","F011":"-33.1"}}]}]}
{"id":"data","val":[{"rc":2,"tpc":"\\RealTime\\Global\\SGDIDR.FX","item":[{"val":{"F009":"10458.80","F011":"-29.2"}}]}]}
and so on, and so on...
You can't use TIdIOHandlerStream to interface with a TCP connection, that is not what it is designed for. It is meant for performing I/O operations using user-provided TStream objects, ie for debugging previously captured sessions.
TIdHTTP is not really designed to handle endless HTTP responses in most cases, as you have described. What is the exact format that the server is delivering its live data as? What do the HTTP response headers look like? It is really difficult to answer your question without know the exact format being used.
However, that being said, there are some cases to consider, depending on what the server is actually sending:
if the server is using a MIME-based server-push format, like multipart/x-mixed-replace, you can enable the hoNoReadMultipartMIME flag in the TIdHTTP.HTTPOptions property, and then read the MIME data yourself from the TIdHTTP.IOHandler after TIdHTTP.Get() exits. For instance, you can use TIdMessageDecoderMIME to help you parse the MIME parts, see New TIdHTTP hoNoReadMultipartMIME flag in Indy's blog, or Delphi Indy TIdHttp and multipart/x-mixed-replace with Text and jpeg image.
Otherwise, if the server is using Transfer-Encoding: chunked, where each data update is sent as a new HTTP chunk, you can use the TIdHTTP.OnChunkReceived event. Or, you can enable the hoNoReadChunked flag in the TIdHTTP.HTTPOptions property, and then read the chunks yourself from the TIdHTTP.IOHandler after TIdHTTP.Get() exits. See New TIdHTTP flags and OnChunkReceived event in Indy's blog.
Otherwise, you could give TIdHTTP.Get() a TIdEventStream to write into, and then use that stream's OnWrite event to access the raw bytes. Or, you could write your own TStream-derived class that overrides the virtual Write() method. Either way, you would be responsible for manually parsing and buffering the raw body data as they are being written to the stream.
Otherwise, you may have to resort to using TIdTCPClient instead, implementing the HTTP protocol manually, then you would be solely responsible for reading in the HTTP response body however you want.
Here is my code:
CIwHTTP http;
std::string output="";
char buffer[1024];
int32 httpCallback(void* sys_data, void* user_data) {
http.ReadData(buffer,http.ContentLength());
output += buffer;
return 0;
}
http.Get(url.c_str(), httpCallback, 0);
The content-length header is properly set in the API. For some reason only part of the API output is received. Sometimes it gets the entire API string and sometimes it returns different portions of the string. It seems random. Help!
You are passing ContentLength() to ReadData, but your buffer has only 1024 bytes. Most likely you have stack overflow - pun intended.
You can either call ReadData in a loop with 1024 until it returns zero, or dynamically allocate buffer on the heap.
IwHTTP::Get() only performs the callback once the headers for the response have been received.
You then need to use IwHTTP::ReadContent() to actually read the remainder of the response in a series of callbacks, as hinted at in one of the other comments.
Please see the IwHTTP Example in our API reference documentation for more details.
Hope this helps!
I am working in a language that has extremely low-level TCP support (if you must know, it's UnrealScript). The response received after making a POST request includes the entire HTTP header, status code, body, etc. as a string.
So, I need to parse the response to extract the body text manually. The HTTP 1.1 specification says:
Response = Status-Line
*(( general-header
| response-header
| entity-header ) CRLF)
CRLF
[message-body]
Am I correct in assuming that the best way to do this is to split the string along a double CRLF (carriage return/line feed) and return the second part of this split?
Or are there weird HTTP edge cases I should be aware of?
Am I correct in assuming that the best way to do this is to split the string along a double CRLF
Yes - but what appears in the body may be compressed using three different compressions methods even if you told the server you don't accept compressed responses.
Further the body may be split into chunks, in between each chunk is an indicator of the size of the next chunk.
Do you really have no scope for using an off the shelf component for parsing? (I would recommend lib curl).
I'm reading source from a website by building a legit connection, like this in Java:
final Socket sock = new Socket(hostname, 80);
PrintWriter writer = new PrintWriter(sock.getOutputStream(), true);
writer.println("GET /path HTTP/1.1");
writer.println("Host: " + hostname);
writer.println();
//...
while (!sock.isClosed() && (line = reader.readLine()) != null) {
System.out.println(line);
}
and it works good, except that there are some weird lines in the output which are not there when I browse the website with - say- Firefox.
The problem is some lines of source get interrupted for some random different information and I don't know why I get information like that to ruin my source.
<div clas
16d0
s="span5">
or
<td style="text-align:c
2000
enter; vertical-align:middle">information</td>
What is this and how do I fix it?
Looks like the server is sending you Chunked data. Can you send HTTP/1.0 instead of 1.1? That should ensure no chunking is performed on the response.
You are reading the HTTP stream raw off a socket, instead of using an existing HTTP reader.
If you really want to do this, you should read the HTTP specification. In your case especially sect. 3.6 concerning chunked transfer.
I am using connector on Erlang. It gets requests from Javascript like "controler?q=value". If I send a value in Unicode, the browser sends an encoded string.
http://127.0.0.1:8001/controler?q=%D1%82%D0%B5%D1%81%D1%82
How to convert this string to UTF-8?
I wrote a module for similar purpose some time ago: https://gist.github.com/816291
You can use it like this:
io:format("~ts~n",[uri:decode_uri_component("%D1%82%D0%B5%D1%81%D1%82")]).
ัะตัั
ok
You might want to have a look at the unicode module.