I have R 4.20+, so I believe utils::download.file is using capability libcurl.
I can get the headers of a url with base::curlGetHeaders(url).
Is there a parameter I can pass in download.file to return the headers, so I can't get them in the same call. Under the hood, download.file is processing the header somehow, as it is receiving it.
How to return response headers I get with curlGetHeaders(url) from the function download.file?
I am aware of external packages (e.g., Rcurl) but for the download to occur, the headers have to be received within R:::base.
Update
Here is the source code from R
"libcurl" = {
headers <- if(length(headers)) paste0(nh, ": ", headers)
status <- .Internal(curlDownload(url, destfile, quiet, mode, cacheOK,
headers))
},
The function curlDownload has traditional curl options here (libcurl.c):
curl_easy_setopt(hnd[i], CURLOPT_HTTPHEADER, headers);
That sets the header, not return it. Why are the raw curl functions not publicly exposed. C exposes them as does PHP... see
Can PHP cURL retrieve response headers AND body in a single request?
$ch = curl_init();
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
curl_setopt($ch, CURLOPT_HEADER, 1);
// ...
$response = curl_exec($ch);
So I guess curlDownload needs:
curl_easy_setopt(hnd[i], CURLOPT_HEADER, 1);
library (curl)
In this library, under the hood, the same syntax is being used. How to expose the syntax directly to me? From download.c:
curl_easy_setopt(handle, CURLOPT_URL, NULL);
curl_easy_setopt(handle, CURLOPT_NOPROGRESS, 1);
curl_easy_setopt(handle, CURLOPT_WRITEFUNCTION, NULL);
curl_easy_setopt(handle, CURLOPT_WRITEDATA, NULL);
This doc mentions the handle_data function, within the curl package.
https://cran.r-project.org/web/packages/curl/curl.pdf
For example, suppose you need to peek at the status code, content type or other headers before reading all the lines. Once you have a working connection, many other functions in R will accept that ... you can use the various read.csv(x), read.delim(x), scan(x) and so on.
library('curl');
h <- new_handle();
x <- curl('https://cran.r-project.org', handle=h);
open(x);
handle_data(h)$status_code;
handle_data(h)$type;
parse_headers(handle_data(h)$headers);
y <- readLines(x);
close(x);
Related
Using the following code to try and create the signature and get the bearer token.
<?php
$tm=time();
$param_str = "grant_type=client_credentials&oauth_consumer_key=xxxx&oauth_nonce=xxx&oauth_signature_method=HMAC-SHA256&oauth_timestamp=".$tm."&oauth_version=1.0";
//die($param_str);
$base_str = "POST&" .urlencode("https://account.api.here.com/oauth2/token") . "&" . urlencode($param_str);
//die($base_str);
$sign_key = urlencode("xxxxxxxxxxxxxxxxxxxxxxxxx")."&";
$signature= hash_hmac("sha256",$base_str,$sign_key);
$url = "https://account.api.here.com/oauth2/token";
$ch = curl_init( $url );
$headers = [
'Content-Type: application/x-www-form-urlencoded',
'Authorization: OAuth oauth_consumer_key="xxx",oauth_nonce="xxx",oauth_signature="'.$signature.'",oauth_signature_method="HMAC-SHA256",oauth_timestamp="'.$tm.'",oauth_version="1.0"',
'Cache-Control: no-cache'
];
$payload="grant_type=client_credentials";
curl_setopt( $ch, CURLOPT_POSTFIELDS, $payload );
curl_setopt($ch, CURLOPT_HTTPHEADER, $headers);
# Return response instead of printing.
curl_setopt( $ch, CURLOPT_RETURNTRANSFER, true );
# Send request.
$result = curl_exec($ch);
curl_close($ch);
?>
Tried various combinations. Getting the same error (Signature mismatch. Authorization signature or client credential is wrong). Even tried copying the exact url encoded string from the document, replacing relevant information and still not working. Is there something I am not understanding at all from the documentation or something I am missing here in my code.
The reason for signature mismatch is that the one you created is different than the one server created. Check the following –
Did you convert the signing key and base string into bytes before
passing it to HMAC-SHA256 hashing algorithm.
Did you convert the output of HMAC-SHA256 hashing algorithm into
base64 string.
also check this link if this can help you.
I am using libcurl to send a POST request, and am trying to get response using the callback function. Below is the relevant code.
main ()
{
...
curl_global_init(CURL_GLOBAL_ALL);
CURL *curl = curl_easy_init ();
curl_easy_setopt(curl, CURLOPT_URL, url_string);
curl_easy_setopt(curl, CURLOPT_POST, 1);
if (strlen(query_string) > 0)
curl_easy_setopt(curl, CURLOPT_POSTFIELDS, query_string);
curl_easy_setopt(curl, CURLOPT_WRITEDATA, &write_buffer);
curl_easy_setopt(curl, CURLOPT_WRITEFUNCTION, WriteCB);
CURLcode res = curl_easy_perform(curl);
if (CURLE_OK == res)
printf("response: %s\n", write_buffer.data);
else
printf("curl failed\n");
curl_easy_cleanup(curl);
curl_global_cleanup();
...
}
struct BufferType
{
Str data;
BufferType() {};
size_t Append(char *src, size_t size, size_t nmemb)
{
data.Append(Str(src, size * nmemb));
return size * nmemb;
}
};
size_t WriteCB(char *data, size_t size, size_t nmemb, BufferType *buffer)
{
printf("WriteCB: %s\n", data);
fflush(stdout);
return buffer->Append(data, size, nmemb);
}
When I launched the program, I can see it is executed (the server responds with "200 OK"). But the program just hangs there, here is the output:
WriteCB: HTTP/1.1 100 Continue
WriteCB:
More info: if I use GET method for other URL, and change the two lines related to POST to
curl_easy_setopt(curl, CURLOPT_HTTPGET, 1);
Then the code works fine.
What could be wrong?
SOLVED
In the command line, I let the user to specify query string, and I have a statement such that if the "query_string" is empty then do not call
"curl_easy_setopt(curl, CURLOPT_POSTFIELDS, query_string);".
Setting curl to verbose shows that the request header has "Expect: 100-continue". So I guess it is because the query string is not set yet. Even it is empty, it should be set.
Glad you found the issue. I originally though you might not be using POST correctly with libcurl which is easy to do. So, I tried your code but didn't find any issue with it.
So, it had to be either an issue with the how you were setting it up or the server itself. The callbacks were behaving as expected. I wanted to try it out because I generally use POST with libcurl like this:
struct curl_httppost* post = NULL;
struct curl_httppost* last = NULL;
curl_formadd(&post, &last, ..., CURLFORM_END);
Here is an example
I'm facing one issue while sending messages to linked connections. I have used two APIs, https://api.linkedin.com/v1/people/ to update profile and https://api.linkedin.com/v1/people/~/mailbox to sending messages to connections.
Updating the profile is working fine for me, and I am even getting one 1-level connection. But the problem is arising when sending messages to connections.
Below is the XML code I'm using to send messages.
$xmlPostData = '<?xml version="1.0" encoding="UTF-8"?>
<mailbox-item>
<recipients>
<recipient>
<person path="/people/'.$iMemberId.'" />
</recipient>
</recipients>
<subject>Invitation to Connect</subject>
<body>Please Join On Eduroadmap</body>
<item-content>
<invitation-request>
<connect-type>friend</connect-type>
<authorization>
<name>'.$sAuthName.'</name>
<value>'.$sAuthValue.'</value>
</authorization>
</invitation-request>
</item-content>
</mailbox-item>';
In above code, $iMemberId, $sAuthName , $sAuthValue are the connection details which I have retrieved from https://api.linkedin.com/v1/people/~/connections api.
Below is the curl I am using to post the above XML.
$ch = curl_init();
curl_setopt($ch,CURLOPT_URL,$sUrl);
curl_setopt($ch, CURLOPT_HTTPHEADER, array('Content-Type: text/xml'));
curl_setopt($ch, CURLOPT_HEADER, false);
curl_setopt($ch, CURLOPT_POST, true);
curl_setopt($ch, CURLOPT_POSTFIELDS,$xmlPostData);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
In the above code, $sUrl is => https://api.linkedin.com/v1/people/~/mailbox?oauth2_access_token=, and I'm appending an access token that is correct I am sure.
What am I missing? I have tried a lot, but I found nothing.
Please visit http://developer.linkedin.com/thread/3255. Anyway, from where did you get $sAuthName and $sAuthValue? It is obtained from the https://api.linkedin.com/v1/people/~/connections** API and using (api-standard-profile-request). Request?
These are the steps:
Example: the API call must me api-standard-profile request.
http://api.linkedin.com/v1/people/id=abcdj32:(first-name,last-name,api-standard-profile-request)
Get the $sAuthName and $sAuthValue for your need from the generated response of the above API call. You can get it from the value tag inside the <api-standard-profile-request>, and it will be like this: <value>name:6hhdh6</value>
For your need, $sAuthName = name & $sAuthValue =6hhdh6
$iMemberId should not be id data from the XML response
Scope should be "w_messages" enabled
XML data should be proper with these
Try these.
I'm POSTing data to an external API (using PHP, if it's relevant).
Should I URL-encode the POST variables that I pass?
Or do I only need to URL-encode GET data?
UPDATE: This is my PHP, in case it is relevant:
$fields = array(
'mediaupload'=>$file_field,
'username'=>urlencode($_POST["username"]),
'password'=>urlencode($_POST["password"]),
'latitude'=>urlencode($_POST["latitude"]),
'longitude'=>urlencode($_POST["longitude"]),
'datetime'=>urlencode($_POST["datetime"]),
'category'=>urlencode($_POST["category"]),
'metacategory'=>urlencode($_POST["metacategory"]),
'caption'=>($_POST["description"])
);
$fields_string = http_build_query($fields);
$ch = curl_init();
curl_setopt($ch, CURLOPT_URL,$url);
curl_setopt($ch,CURLOPT_POST,count($fields));
curl_setopt($ch,CURLOPT_POSTFIELDS,$fields);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
$response = curl_exec($ch);
General Answer
The general answer to your question is that it depends. And you get to decide by specifying what your "Content-Type" is in the HTTP headers.
A value of "application/x-www-form-urlencoded" means that your POST body will need to be URL encoded just like a GET parameter string. A value of "multipart/form-data" means that you'll be using content delimiters and NOT url encoding the content.
This answer has a much more thorough explanation if you'd like more information.
Specific Answer
For an answer specific to the PHP libraries you're using (CURL), you should read the documentation here.
Here's the relevant information:
CURLOPT_POST
TRUE to do a regular HTTP POST.
This POST is the normal application/x-www-form-urlencoded kind, most commonly used by HTML forms.
CURLOPT_POSTFIELDS
The full data to post in a HTTP "POST" operation. To post a file, prepend a filename with # and use the full path. The filetype can be explicitly specified by following the filename with the type in the format ';type=mimetype'. This parameter can either be passed as a urlencoded string like 'para1=val1¶2=val2&...' or as an array with the field name as key and field data as value. If value is an array, the Content-Type header will be set to multipart/form-data. As of PHP 5.2.0, value must be an array if files are passed to this option with the # prefix.
#DougW has clearly answered this question, but I still like to add some codes here to explain Doug's points. (And correct errors in the code above)
Solution 1: URL-encode the POST data with a content-type header :application/x-www-form-urlencoded .
Note: you do not need to urlencode $_POST[] fields one by one, http_build_query() function can do the urlencoding job nicely.
$fields = array(
'mediaupload'=>$file_field,
'username'=>$_POST["username"],
'password'=>$_POST["password"],
'latitude'=>$_POST["latitude"],
'longitude'=>$_POST["longitude"],
'datetime'=>$_POST["datetime"],
'category'=>$_POST["category"],
'metacategory'=>$_POST["metacategory"],
'caption'=>$_POST["description"]
);
$fields_string = http_build_query($fields);
$ch = curl_init();
curl_setopt($ch, CURLOPT_URL,$url);
curl_setopt($ch, CURLOPT_POST,1);
curl_setopt($ch, CURLOPT_POSTFIELDS,$fields_string);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
$response = curl_exec($ch);
Solution 2: Pass the array directly as the post data without URL-encoding, while the Content-Type header will be set to multipart/form-data.
$fields = array(
'mediaupload'=>$file_field,
'username'=>$_POST["username"],
'password'=>$_POST["password"],
'latitude'=>$_POST["latitude"],
'longitude'=>$_POST["longitude"],
'datetime'=>$_POST["datetime"],
'category'=>$_POST["category"],
'metacategory'=>$_POST["metacategory"],
'caption'=>$_POST["description"]
);
$ch = curl_init();
curl_setopt($ch, CURLOPT_URL,$url);
curl_setopt($ch, CURLOPT_POST,1);
curl_setopt($ch, CURLOPT_POSTFIELDS,$fields);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
$response = curl_exec($ch);
Both code snippets work, but using different HTTP headers and bodies.
curl will encode the data for you, just drop your raw field data into the fields array and tell it to "go".
Above posts answers questions related to URL Encoding and How it works, but the original questions was "Should I URL-encode POST data?" which isn't answered.
From my recent experience with URL Encoding, I would like to extend the question further.
"Should I URL-encode POST data, same as GET HTTP method. Generally, HTML Forms over the Browser if are filled, submitted and/or GET some information, Browsers will do URL Encoding but If an application exposes a web-service and expects Consumers to do URL-Encoding on data, is it Architecturally and Technically correct to do URL Encode with POST HTTP method ?"
$agent = "Mozilla/4.0 (compatible; MSIE 5.01; Windows NT 5.0)";
$ch=curl_init();
curl_setopt ($ch, CURLOPT_URL,$url );
curl_setopt($ch, CURLOPT_USERAGENT, $agent);
curl_setopt ($ch, CURLOPT_RETURNTRANSFER, 1);
curl_setopt ($ch,CURLOPT_VERBOSE,false);
curl_setopt($ch, CURLOPT_TIMEOUT, 5);
curl_setopt($ch,CURLOPT_SSL_VERIFYPEER, FALSE);
curl_setopt($ch,CURLOPT_SSLVERSION,3);
curl_setopt($ch,CURLOPT_SSL_VERIFYHOST, FALSE);
$page=curl_exec($ch);
//echo curl_error($ch);
$httpcode = curl_getinfo($ch, CURLINFO_HTTP_CODE);
curl_close($ch);
I'm using cUrl at my project. I'm getting HTTP CODE of web sites. But when i try "asdasd" ($url="asdasd") , returned HTTP CODE is 200 . But "asdasd" isn't a web site. Why HTTP CODE is 200 ?
You might want to check the return value from curl_exec first. It says in the manual:
Returns TRUE on success or FALSE on failure. However, if the CURLOPT_RETURNTRANSFER option is set, it will return the result on success, FALSE on failure.
For curl_getinfo there's a side note:
Information gathered by this function is kept if the handle is re-used. This means that unless a statistic is overridden internally by this function, the previous info is returned.
So that HTTP 200 might well be the result of a previous cURL invocation.
Another possible error can happen if you develop in local and you use custom managed DNS services, then you probably can get some managed error pages like OpenDNS is doing.