libcurl: write call back function not invoked - http

I am using libcurl to send a POST request, and am trying to get response using the callback function. Below is the relevant code.
main ()
{
...
curl_global_init(CURL_GLOBAL_ALL);
CURL *curl = curl_easy_init ();
curl_easy_setopt(curl, CURLOPT_URL, url_string);
curl_easy_setopt(curl, CURLOPT_POST, 1);
if (strlen(query_string) > 0)
curl_easy_setopt(curl, CURLOPT_POSTFIELDS, query_string);
curl_easy_setopt(curl, CURLOPT_WRITEDATA, &write_buffer);
curl_easy_setopt(curl, CURLOPT_WRITEFUNCTION, WriteCB);
CURLcode res = curl_easy_perform(curl);
if (CURLE_OK == res)
printf("response: %s\n", write_buffer.data);
else
printf("curl failed\n");
curl_easy_cleanup(curl);
curl_global_cleanup();
...
}
struct BufferType
{
Str data;
BufferType() {};
size_t Append(char *src, size_t size, size_t nmemb)
{
data.Append(Str(src, size * nmemb));
return size * nmemb;
}
};
size_t WriteCB(char *data, size_t size, size_t nmemb, BufferType *buffer)
{
printf("WriteCB: %s\n", data);
fflush(stdout);
return buffer->Append(data, size, nmemb);
}
When I launched the program, I can see it is executed (the server responds with "200 OK"). But the program just hangs there, here is the output:
WriteCB: HTTP/1.1 100 Continue
WriteCB:
More info: if I use GET method for other URL, and change the two lines related to POST to
curl_easy_setopt(curl, CURLOPT_HTTPGET, 1);
Then the code works fine.
What could be wrong?

SOLVED
In the command line, I let the user to specify query string, and I have a statement such that if the "query_string" is empty then do not call
"curl_easy_setopt(curl, CURLOPT_POSTFIELDS, query_string);".
Setting curl to verbose shows that the request header has "Expect: 100-continue". So I guess it is because the query string is not set yet. Even it is empty, it should be set.

Glad you found the issue. I originally though you might not be using POST correctly with libcurl which is easy to do. So, I tried your code but didn't find any issue with it.
So, it had to be either an issue with the how you were setting it up or the server itself. The callbacks were behaving as expected. I wanted to try it out because I generally use POST with libcurl like this:
struct curl_httppost* post = NULL;
struct curl_httppost* last = NULL;
curl_formadd(&post, &last, ..., CURLFORM_END);
Here is an example

Related

Using R, how to get response headers with `download.file`?

I have R 4.20+, so I believe utils::download.file is using capability libcurl.
I can get the headers of a url with base::curlGetHeaders(url).
Is there a parameter I can pass in download.file to return the headers, so I can't get them in the same call. Under the hood, download.file is processing the header somehow, as it is receiving it.
How to return response headers I get with curlGetHeaders(url) from the function download.file?
I am aware of external packages (e.g., Rcurl) but for the download to occur, the headers have to be received within R:::base.
Update
Here is the source code from R
"libcurl" = {
headers <- if(length(headers)) paste0(nh, ": ", headers)
status <- .Internal(curlDownload(url, destfile, quiet, mode, cacheOK,
headers))
},
The function curlDownload has traditional curl options here (libcurl.c):
curl_easy_setopt(hnd[i], CURLOPT_HTTPHEADER, headers);
That sets the header, not return it. Why are the raw curl functions not publicly exposed. C exposes them as does PHP... see
Can PHP cURL retrieve response headers AND body in a single request?
$ch = curl_init();
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
curl_setopt($ch, CURLOPT_HEADER, 1);
// ...
$response = curl_exec($ch);
So I guess curlDownload needs:
curl_easy_setopt(hnd[i], CURLOPT_HEADER, 1);
library (curl)
In this library, under the hood, the same syntax is being used. How to expose the syntax directly to me? From download.c:
curl_easy_setopt(handle, CURLOPT_URL, NULL);
curl_easy_setopt(handle, CURLOPT_NOPROGRESS, 1);
curl_easy_setopt(handle, CURLOPT_WRITEFUNCTION, NULL);
curl_easy_setopt(handle, CURLOPT_WRITEDATA, NULL);
This doc mentions the handle_data function, within the curl package.
https://cran.r-project.org/web/packages/curl/curl.pdf
For example, suppose you need to peek at the status code, content type or other headers before reading all the lines. Once you have a working connection, many other functions in R will accept that ... you can use the various read.csv(x), read.delim(x), scan(x) and so on.
library('curl');
h <- new_handle();
x <- curl('https://cran.r-project.org', handle=h);
open(x);
handle_data(h)$status_code;
handle_data(h)$type;
parse_headers(handle_data(h)$headers);
y <- readLines(x);
close(x);

libcURL get request

pretty useless at code but I can but try.
I'm trying to send a HTTP request [GET] via localhost to a radio sever program. [Windows C++]
I can send this from my browser or cmd and the result is volume is set to zero
http://127.0.0.1:8282/parameter/hd1/spk/volume?set=0
I can connect with my c++ app, tried both winsock & libcURL. But the volume does not change
this is pretty much the example from cURL, I'm trying the smallest incremental steps I can.
#include <stdio.h>
#include <curl/curl.h>
int main(void)
{
CURL* curl;
CURLcode res;
/* In windows, this will init the winsock stuff */
curl_global_init(CURL_GLOBAL_ALL);
/* get a curl handle */
curl = curl_easy_init();
if (curl) {
curl_easy_setopt(curl, CURLOPT_URL, "127.0.0.1:8282");
curl_easy_setopt(curl, CURLOPT_HTTPGET, "/parameter/hd1/spk/volume?=0");
/* Perform the request, res will get the return code */
res = curl_easy_perform(curl);
/* Check for errors */
if (res != CURLE_OK)
fprintf(stderr, "curl_easy_perform() failed: %s\n",
curl_easy_strerror(res));
/* always cleanup */
curl_easy_cleanup(curl);
}
curl_global_cleanup();
return 0;
}
EDIT
This is ridiculous, but this did it, removed the CURLOPT_HTTPGET bit, and put the full path in CURLOPT_URL,
it's not supposed to work like that is it?
curl_easy_setopt(curl, CURLOPT_URL, "127.0.0.1:8282/parameter/hd1/spk/volume?=0");

How in Qt5 to check if url is available?

Using Qt5, how to simply check if given url is available?
Not using special functions for signal slots, but simply using something like bool isUrlAvailable(QString url), are there any function like this?
Update QUrl.isValid() is incorrect answer, it is just checks if url is correctly formed.
Update 2 QUrl.host() is incorrect answer too, it is just returns host part of given url, it does not check for its availability.
Update 3 pinging host is also incorrect, because url may be available, but does not accept icmp echo (=ping)
Yes, you can do a HEAD request to a given URL.
bool urlExists (QString url_string) {
QUrl url(url_string);
QTcpSocket socket;
socket.connectToHost(url.host(), 80);
if (socket.waitForConnected()) {
socket.write("HEAD " + url.path().toUtf8() + " HTTP/1.1\r\n"
"Host: " + url.host().toUtf8() + "\r\n\r\n");
if (socket.waitForReadyRead()) {
QByteArray bytes = socket.readAll();
if (bytes.contains("200 OK")) {
return true;
}
}
}
return false;
}
This is just an example for 200 OK and you might also want to check if the status code is some other in 2XX or in 3XX (redirection) class.
So taking from pajaja + a few other SO answers + a tutorial I found
(http://www.blikoon.com/networking/http-potocol-writting-a-simple-client-using-qt-qtcpsocket-and-troubleshooting-using-telnet)
I came up with this tweaked version because the one above didn't work
bool urlExists(QUrl theurl){
QTextStream out(stdout);
QTcpSocket socket;
QByteArray buffer;
socket.connectToHost(theurl.host(), 80);
if (socket.waitForConnected()) {
//Standard http request
socket.write("GET / HTTP/1.1\r\n"
"host: " + theurl.host().toUtf8() + "\r\n\r\n");
if (socket.waitForReadyRead()) {
while(socket.bytesAvailable()){
buffer.append(socket.readAll());
int packetSize=buffer.size();
while(packetSize>0)
{
//Output server response for debugging
out << "[" << buffer.data() << "]" <<endl;
//set Url if 200, 301, or 302 response given assuming that server will redirect
if (buffer.contains("200 OK") ||
buffer.contains("302 Found") ||
buffer.contains("301 Moved")) {
return true;
}
buffer.remove(0,packetSize);
//packetSize=getPacketSize(buffer);
packetSize=buffer.size();
} //while packet size >0
} //while socket.bytesavail
} //socket wait for ready read
}//socket write
return false;
}
The QTextStream prints what is being read from the socket so you can know what conditions to add and why your http request didn't work (I used it to figure out that I needed 301 and 302). The while loops are a modified version of ratchetfreak's answer here
How to read complete data in QTcpSocket?
to make sure you get everything out of the socket. I'm testing if I need to change the "/index.html" part of the socket write but so far it seems fine.
Edit: Should just be "GET /" not "GET /index.html"

RInside: parseEvalQ 'Parse Error' causes each subsequent call to parseEvalQ to give a 'Parse Error' even if exception handled

My code, which tries to emulate an R shell via C++, allows a user to send R commands over a tcp connection which are then passed to the R instance through the RInside::parseEvalQ function, during runtime. I have to be able to handle badly formatted commands. Whenever a bad command is given as an argument to parseEvalQ I catch the runtime error thrown (looking at RInside.cpp my specific error is flagged with 'PARSE_ERROR' 'status' within the parseEval(const string&, SEXP) function), what() gives a "St9exception" exception.
I have two problems, the first more pressing than the second:
1a . After an initial Parse Error any subsequent call to parseEvalQ results in another Parse Error even if the argument is valid. Is the embedded R instance being corrupted in some way by the parse error?
1b . The RInside documentation recommends using Rcpp::Evaluator::run to handle R exceptions in C++ (which I suspect are being thrown somewhere within the R instance during the call to parseEval(const string&, SEXP), before it returns the error status 'PARSE_ERROR'). I have experimented trying to use this but can find no examples on the web of how to practically use Rcpp::Evaluator::run.
2 . In my program I re-route stdout and stderr (at C++ level) to the file descriptor of my tcp connection, any error messages from the RInside instance get sent to the console, however regular output does not. I send RInside the command 'sink(stderr(), type="output")' in an effort to re-route stdout to stderr (as stderr appears to be showing up in my console) but regular output is still not shown. 'print(command)' works but i'd like a cleaner way of passing stdout straight to the console as in a normal R shell.
Any help and/or thoughts would be much appreciated. A distilled version of my code is shown below:
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
using namespace std;
string request_cpp;
ostringstream oss;
int read(FILE* tcp_fd)
{
/* function to read input from FILE* into the 'request_cpp' string */
}
int write(FILE* tcp_fd, const string& response)
{
/* function to write a string to FILE* */
}
int main(int argc, char* argv[])
{
// create RInside object
RInside R(argc,argv);
//socket
int sd = socket(PF_INET, SOCK_STREAM, 0);
addr.sin_family = AF_INET;
addr.sin_port = htons(40650);
// set and accept connection on socket
inet_pton(AF_INET, "127.0.0.1", &addr.sin_addr);
bind(sd,(struct sockaddr*)&addr, sizeof(addr));
listen(sd,1);
int sd_i = accept(sd, 0, 0);
//re-route stdout and stderr to socket
close(1);
dup(sd_i);
close(2);
dup(sd_i);
// open read/write file descriptor to socket
FILE* fp = fdopen(sd_i,"r+");
// emulate R prompt
write(fp,"> ");
// (attempt to) redirect R's stdout to stderr
R.parseEvalQ("sink(stderr(),type=\"output\");");
// read from socket and pass commands to RInside
while( read(fp) )
{
try
{
// skip empty input
if(request_cpp == "")
{
write(fp, "> ");
continue;
}
else if(request_cpp == "q()")
{
break;
}
else
{
// clear string stream
oss.str("");
// wrap command in try
oss << "try(" << request_cpp << ");" << endl;
// send command
R.parseEvalQ(oss.str());
}
}
catch(exception e)
{
// print exception to console
write(fp, e.what());
}
write(fp, "> ");
}
fclose(fp);
close(sd_i);
exit(0);
}
I missed this weeks ago as you didn't use the 'r' tag.
Seems like you are re-implementing Simon's trusted rserver. Why not use that directly?
Otherwise, for Rcpp question, consider asking on our rcpp-devel list.

Sending feedback data via HTTPS POST from wxWidgets application

I have a wxWidgets application and would like to add a way for users to submit feedback in a simple (implementation and usage), reliable, cross-platform and secure way. Using HTTP POST over SSL seems to fit those requirements best (although I'll consider answers that suggest other approaches). However, support for HTTPS in wxWidgets seems limited.
Here are some of the options I've considered and the problems with them:
wxSMTP: no SSL/TLS support that I've found. Relies on user having a correct mail configuration (sendmail, MAPI).
wxHTTP: everything but SSL/HTTPS support.
wxSSL: everything if it wasn't incomplete and long dead.
wxCURL: everything but complicated to build/incorporate (in fact currently release fails to build).
libcurl: just link with and call into libcurl directly. This is the solution I've settled on (and I have a working prototype) but it feels very non-wx and while libcurl is cross-platform, Windows is definitely not its native platform so it adds significant dependency and build complexity to the project.
I've decided to go with libCURL linked to openssl. Both of the packages are able availabe on most Linux system and can be fairly easily built on Windows and OS X.
Here is example C code that sends feedback:
#include <stdio.h>
#include <string.h>
#include <curl/curl.h>
#include <curl/types.h>
#include <curl/easy.h>
int main(int argc, char *argv[])
{
CURL *curl;
CURLcode res;
struct curl_httppost *formpost=NULL;
struct curl_httppost *lastptr=NULL;
struct curl_slist *headerlist=NULL;
static const char buf[] = "Expect:";
curl_global_init(CURL_GLOBAL_ALL);
/* Fill in the name field */
curl_formadd(&formpost,
&lastptr,
CURLFORM_COPYNAME, "name",
CURLFORM_COPYCONTENTS, "John Doe",
CURLFORM_END);
/* Fill in the comments field */
curl_formadd(&formpost,
&lastptr,
CURLFORM_COPYNAME, "comments",
CURLFORM_COPYCONTENTS, "using HTTPS POST\nline 2\nline 3",
CURLFORM_END);
curl = curl_easy_init();
/* initalize custom header list (stating that Expect: 100-continue is not
wanted */
headerlist = curl_slist_append(headerlist, buf);
if(curl) {
/* what URL that receives this POST */
curl_easy_setopt(curl, CURLOPT_URL,
"https://some_host.com/path/feedback.php");
// Uncomment to disable certificate checks
//curl_easy_setopt(curl, CURLOPT_SSL_VERIFYPEER, 0);
//curl_easy_setopt(curl, CURLOPT_SSL_VERIFYHOST, 0);
curl_easy_setopt(curl, CURLOPT_HTTPHEADER, headerlist);
curl_easy_setopt(curl, CURLOPT_HTTPPOST, formpost);
res = curl_easy_perform(curl);
printf("Curl result: %d\n", res);
/* always cleanup */
curl_easy_cleanup(curl);
/* then cleanup the formpost chain */
curl_formfree(formpost);
/* free slist */
curl_slist_free_all (headerlist);
}
return 0;
}
That can be compiled on Linux like so:
gcc post.c -lcurl -o post
Here is example PHP code that accepts that post:
<?php
$recipient = "john#some_host.com";
if (empty($_POST)) {
// We only accpet POSTs
header('HTTP/1.0 403 Forbidden');
exit;
} else {
// Handle a POST
$message .= "Submitted at ".date("F j, Y, g:i a")."\n\n";
$message .= "Name:\n";
$message .= $_POST['name']."\n\n";
$message .= "-------------------------------------------\n\n";
$message .= "Comments:\n\n";
$message .= $_POST['comments']."\n\n";
// Send message to email address
$sent = mail($recipient, "Feedback",
$message, "From: Feedback <noreply#some_host.com>\r\n");
if ($sent) {
?>
<html>
<body>
Got POST and sent email:
<pre><? echo $message; ?></pre>
</body>
</html>
<?php
} else {
// Return an error
header('HTTP/1.0 500 Internal Server Error', true, 500);
exit;
}
}
?>
Boost::Asio is an alternative I've used before, even in a wxWidgets app. I've used it to download files before (and only have sample code for that), but sadly don't have (and can't quickly find) sample code for HTTP(S) POSTs.
Boost::asio has a "problem" in that it's Boost (which is either a big problem because "ugh, Boost", or not a problem "Awesome, Boost!"

Resources