I am using the Twitter REST API to retrieve data in JSON format. Twitter's developer page makes it easy by providing a command that can be pasted directly into the terminal and executed. The following command works in the terminal.
curl --get 'https://api.twitter.com/1.1/statuses/user_timeline.json' --data 'count=3200&screen_name=BernieSanders' --header 'Authorization: OAuth oauth_consumer_key="####", oauth_nonce="####", oauth_signature="####", oauth_signature_method="HMAC-SHA1", oauth_timestamp="1467719924", oauth_token="####", oauth_version="1.0"' --verbose
I am trying to get the JSON data into R, and would like to execute this same command in the R console. I have tried curlconverter using the exact same code, get an error that there is an unexpected symbol. However, the code is exactly the same. Is there a more suitable package for executing this code?
curlExample <- "curl --get 'https://api.twitter.com/1.1/statuses/user_timeline.json' --data 'count=3200&screen_name=BernieSanders' --header 'Authorization: OAuth oauth_consumer_key="####", oauth_nonce="####", oauth_signature="####", oauth_signature_method="HMAC-SHA1", oauth_timestamp="1467719924", oauth_token="####", oauth_version="1.0"' --verbose"
I think you are going to need to escape those quotation marks. Try this instead:
curlExample <- "curl --get 'https://api.twitter.com/1.1/statuses/user_timeline.json' --data 'count=3200&screen_name=BernieSanders' --header 'Authorization: OAuth oauth_consumer_key=\"####\", oauth_nonce=\"####\", oauth_signature=\"####\", oauth_signature_method=\"HMAC-SHA1\", oauth_timestamp=\"1467719924\", oauth_token=\"####\", oauth_version=\"1.0\"' --verbose"
That the quotes were not being escaped properly was actually evident in your question, because the Stack Overflow markup rendered the curlExample string in multiple colors.
I believe https://cran.r-project.org/web/packages/twitteR/twitteR.pdf is the way to do this in R.
But specifically in your case, you have unescaped " in your string.
Related
I have been working with an API call to structure it in JSON format so I might later push it into a database. Then code looks like this:
getPage() {
curl --fail -X GET 'https://api.app.com/v1/test?page=1&pageSize=1000&sort=desc' \
-H 'Authorization: Bearer 123abc456pickupsticks789' \
-H 'cache-control: no-cache'
}
getPage \
| jq -c '.items | .[] | {landing_id: .landing_id, submitted_at: .submitted_at, answers: .answers, email: .hidden.email}' \
> testpush.json
When I run it though, it produces this error: jq: error (at :0): Cannot iterate over null (null)
I've looked at solutions such as this one, or this one from this site, and this response.
The common solution seemed to be using a ? in front of [] and I tried it in the jq line towards the bottom, but it still does not work. It just produces an empty json file.
Am I misreading the takeaway from those other answers and not putting my ? in the right place?>
That means your field .items doesn't exist or is null if interpreting the output of your curl as a JSON.
As a side note, the shorter version of your query:
.items[] | {landing_id, submitted_at, answers, email: .hidden.email}
I have this curl command that I can call in bash
curl -X POST -H 'Content-Type: text/csv' --data-binary #data/data.csv https://some.url.com/invocations > data/churn_scored.jsonl
which posts a CSV file to the API endpoint and I redirect the results into a .jsonl file.
I can't find where I can specify a data file to POST to the endpoint like using curl's #.
What's the way to achieve the CURL post using R's curl package (or any other package)? The redirection of output I can figure out in another way.
This is a very helpful site in converting curl commands to other languages: https://curl.trillworks.com/#r
When plugging in your curl command there I got this.
require(httr)
headers = c(
`Content-Type` = 'text/csv'
)
data = upload_file('data/data.csv')
res <- httr::POST(url = 'https://some.url.com/invocations', httr::add_headers(.headers=headers), body = data)
To specific symbol #. From man curl:
--data-binary <data>
(HTTP) This posts data exactly as specified with no extra processing whatsoever.
If you start the data with the letter #, the rest should be a filename. Data is
posted in a similar manner as --data-ascii does, except that newlines are preserved
and conversions are never done.
If this option is used several times, the ones following the first will append data
as described in -d, --data.
It seems no need to worry about #
As it mentioned by #chinsoon12, httr is a good nice method to handle request:
-X or --request translates to VERB function POST(), which includes --data-binary
-H or --header translates to add_headers() but there are special functions for setting content type (see below)
So it looks like:
library(httr)
response <- POST(
url = "https://some.url.com/invocations",
body = upload_file(
path = path.expand("data/data.csv"),
type = 'text/csv'),
verbose()
)
# get response and write to you disk
I used Plumber (https://github.com/trestletech/plumber) to create a web API for a R source code, which purpose is to process a json_string as an input, and converting it into a R dataframe.
R code:
#* #param data
#* #post /predict
jsonToDF <- function(data) {
library(jsonlite)
dat <- fromJSON(data)
}
As you can see from the code above, I'm using the library jsonlite and the fromJSON() method.
Then I exposed the service locally via:
library(plumber)
r <- plumb("code.R")
r$run(port=54321)
I used the curl request:
curl -v -i -X POST -H "Content-Type: application/json" --data "testData='{\"a\":\"b\"}'" http://localhost:54321/predict
to send a json_string ({"a":"b"}) but this is the error i got:
<simpleError: lexical error: invalid char in json text.
'{"a":"b"}'
(right here) ------^
I used R console to double check the validity of the input, and i did not get the same error message.
> z <- fromJSON('{"a":"b"}')
> z
$a
[1] "b"
Can anyone explain to me why it works in R console, but not as an input in the data field of the curl requests? Thank you very much!
I managed to get the desired input by changing "data='{\"a\":\"b\"}'" to "data={\"a\":\"b\"}". I simply removed the single quotation marks from the --data field of the curl command.
I think R automatically treats {\"a\":\"b\"} as a string after receiving it from curl, so there was no need for me to explicitly declare it as '{\"a\":\"b\"}' when sending it over curl.
I am trying to execute a CURL call by passing parameters.
However, it does not return the intended output.
Here is the code:
curl -X GET 'http://admin-app.prod.order-services.cp.glb.prod.walmart.com/order-services-admin/plutus-admin/published-transactions/count?fromDate=$from_date&toDate=$to_date'
Please let me know how I can pass $from_date and $to_date in the above URL.
I think you want to substitute the values of variables $from_date and $to_date in the link. Variables are not substituted in single quoted strings. Try replacing single quotes with double quotes:
curl -X GET "http://admin-app.prod.order-services.cp.glb.prod.walmart.com/order-services-admin/plutus-admin/published-transactions/count?fromDate=$from_date&toDate=$to_date"
I'm trying to use Postman for upload a WKT layer by the cURL code below.
curl -X POST
'https://gfe.cit.api.here.com/2/layers/upload.json?layer_id=123&app_id={APP_ID}&app_code={APP_CODE}'
-H 'content-type: multipart/form-data; boundary=----WebKitFormBoundary7MA4YWxkTrZu0gW'
-F 'zipfile=#C:\xampp\htdocs\here\southern.wkt.zip'`
but the response message is
{"issues":[{"message":"Multipart should contain exactly one part"}],"error_id":"2e1e3994-69d6-43bb-8224-2a869b5255ae","response_code":"400 Bad Request"}
Am I doing something wrong?
Make sure your WKT file doesn't contain additional indentation(spaces) avoid giving spaces and instead use equal tabs and give proper column names
refer this