R translating curl commands to post multipart form - problem - r

I have the following curl command that, when run from command line, works perfectly:
curl -X POST -u "myusername|myemail#domain.com:myPassword"
-H "Content-Type: multipart/form-data"
--form file=#MyFileForUploading.csv
https://mysite-data.herokuapp.com/api/mymarket/setups/uploads
[Apologies: this is not a working example as I cannot provide the real url and credentials. I am hoping you can help me with the translation from curl to httr without running the example yourselves.]
Here's my attempt to translate the above to the language of R's httr, which did NOT work:
library(httr)
POST("https://mysite-data.herokuapp.com/api/mymarket/setups/uploads",
config = authenticate("myusername|myemail#domain.com", "myPassword"),
body = upload_file("MyFileForUploading.csv", type = "text/csv"),
encode = "multipart")
The curl command serves to upload a csv file being used as setup for a web-based trading interface. Setup includes things like trader initial holdings of objects, trader permissions (to buy and sell), etc. All this is simply stored as a csv file (columns = setup parameters; rows = traders).
Can anyone see an obvious mis-translation? I am very ignorant about both curl and httr. My translation is based on learning from examples and I wouldn't be surprised if there's an obvious failure, for example, with the content-type part of the command.
Thanks!

You're really close. This works with environment values setup in "~/Renviron":
library("httr")
post_url <- Sys.getenv("POST_URL")
username <- Sys.getenv("USERNAME")
password <- Sys.getenv("PASSWORD")
csv_file <- Sys.getenv("CSV_FILE")
POST(
url = post_url,
config = authenticate(username, password),
body = list(file = upload_file(csv_file)),
encode = "multipart",
verbose()
) -> response
The key is the file = as you used in your CURL command.

Related

Download & decompress JSON file in R using curl or RCurl

I have the following bash script to download & decompress a JSON file:
#!/bin/sh -ex
# Ensure data directory (or a link) exists.
test -e results || mkdir results
# Download and decompress data.
curl -u $GISAID_USERNAME:$GISAID_PASSWORD --retry 4 \
https://www.epicov.org/epi3/3p/$GISAID_FEED/export/provision.json.xz \
| xz -d -T8 > results/gisaid.json
Ideally I would like to have an R function to download & decompress this file in a given directory, with the environment variables above $GISAID_USERNAME, $GISAID_PASSWORD & $GISAID_FEED passed as arguments. Would anyone know how to accomplish this, e.g. using package curl or RCurl? (It would also be OK not to decompress it and leave it as .json.xz, as I would be reading the file later using
library(jsonlite)
GISAID_json <- jsonlite::stream_in(gzfile(".//data//GISAID_json//provision.json.xz"))
Something like this should work:
library(curl)
library(glue)
custom_curl <- function(user, pass, feed, dest) {
custom_handle <- curl::new_handle()
curl::handle_setopt(
custom_handle,
username = user,
password = pass
)
url <- glue::glue(
"https://www.epicov.org/epi3/3p/{feed}/export/provision.json.xz"
)
curl::curl_download(url, dest, handle = custom_handle)
}
custom_curl('my_user', 'xxxxxx', 'feed1', 'dest/filename.json.xz')
As I can't test in the real files and url, I'm not sure if little tinkering in the function is needed, but at least is a starter point for you.
Does executing your terminal commands in R using the system function already help you?
Put your terminal call into system() and it should execute and create your file. Afterwards read in the file. Of course you would have to replace the $GISAID_USERNAME, $GISAID_PASSWORD with your actual information. If the login information or the url should be flexible, you can put together a string beforehand, since system() expects a string with the command to execute.
system("curl -u $GISAID_USERNAME:$GISAID_PASSWORD --retry 4 \
https://www.epicov.org/epi3/3p/$GISAID_FEED/export/provision.json.xz \
| xz -d -T8 > results/gisaid.json")
Afterwards just read in the (hopefully) created file.
Couldn't test with your setup, but for me e.g. this small example successfully creates a file:
system("curl https://raw.githubusercontent.com/SteffenMoritz/imputeTS/master/pkgdown/favicon/favicon.ico > /Users/Steve/Downloads/x.ico")

How to analyze a document from blob using Form-Recognizer Microsoft-Cognitive?

I am using Microsoft-Cognitive service (Form-Recognizer) to analyze the document and read its content for further operations. I wanted to analyze the pdf file which is uploaded at the blob.
I am able to get the content using local file path but once I am providing the blob URL then it is not able to open the file.
file-path looks like: https:\blob\SupplierformUpdate1.pdf
curl: (26) couldn't open file: the File path
I have trained it properly and got the model id.
I tried an analyze service.
curl -X POST "https://<Endpoint>/formrecognizer/v1.0-preview/custom/models/<modelID>/analyze" -H "Content-Type: multipart/form-data" -F "form=#\"<path to your form>\";type=<file type>" -H "Ocp-Apim-Subscription-Key: <subscription key>"
https://learn.microsoft.com/en-us/azure/cognitive-services/form-recognizer/quickstarts/curl-train-extract#train-a-form-recognizer-model
I want to get the content by providing the blob URL of the pdf file.
To analyze a form and extract data with Form Recognizer you can send the file as a multipart/form.

Unable to upload to Watson Analytics data with R

I have the following code:
curl -v -X PUT -H "X-IBM-Client-Id:YOUR_CLIENT_ID" -H "X-IBM-Client-Secret:YOUR_CLIENT_SECRET" -H "Authorization: Bearer YOUR_ACCESS_TOKEN" -H "Content-Type:text/csv" https://api.ibm.com/watsonanalytics/run/data/v1/datasets/ID_OF_DATA_SET/content -d "YOUR_DATA"
That code is the example they gave me in order to upload data, when I try to translate the to R it looks like this:
I am using the following libraries:
library(RCurl)
library(RJSONIO)
library(XML)
library(httr)
library(readr)
and the code looks like :
Upload_data<-PUT( url = "https://api.ibm.com/watsonanalytics/run/data/v1/datasets/DataID/content",
add_headers('Content-Type' = 'text/csv'),
add_headers('Accept'= 'application/json'),
add_headers('X-IBM-Client-Secret' = Secret),
add_headers('X-IBM-Client-Id' = Client),
add_headers('Authorization: Bearer'=Auth, body= "Province,Population density,Area,Population,
Ontario,97,83858,8169929
Quebec,337,30510,11007020
Alberta,111,547030,63601002
Manitoba,233,357021,81799600
British Columbia,393,41526,16824400
Costa Rica, 1,10000, 1000000"))
Seems that it is not working, do you know some way to fix it?
If I'm not mistaken, I believe that R is not yet supported through the Watson Analytics API Explorer, which is what your PUT request is using "https://api.ibm.com/watsonanalytics/"
The supported languages include Python, JavaScript, Java, C#, and PHP.

CURL Command To Create A File On Server

I have a mini program/server built on one of my computers (Machine1) and I am trying to create or overwrite a file through cURL on another computer (Machine2). So Machine2 is connected to Machine1. Ive been looking through cURL's documentation for command that will do this but have had no luck and as well on stack overflow.
https://curl.haxx.se/docs/manpage.html
I have also tried the examples on this SO post:
HTTP POST and GET using cURL in Linux
Any idea as to what the command might be through command prompt? (equivalent of a POST command). I have tried so far using -O, -K, -C and a multitude of others which have not worked.
In command line, all you need to do is using curl --form to simulate a multipart/form-data POST request:
curl --form "testfile=#thefilename.jpg" http://<Machine2>/<Path>
testfile is the field name used for form, if you don't care, just use any english word.
# is used here to make file thefilename.jpg get attached in the post as a file upload. Refer to curl man doc.
In server side, URL http://<Machine2>/<Path> should be listened. When curl send the previous POST request, server side program should get it, extract the attached file (thefilename.jpg), and save to disk.

R - curl Github API to access private repos

All I'm trying to do is read all the repos and issues in my organizations private repos. I can from my Windows 7 cmd.exe execute
curl -u "user:pass" https://api.github.com/orgs/:org/repos
and I get back all of my repositories. I can pipe this to a file:
curl -u "user:pass" https://api.github.com/orgs/:org/repos > "C:\Users\Location\file.txt"
and this saves the JSON output. I can replicate this in R but in what seems like a terrible way.
fullRepos = system('curl -s -u "user:pass" -G https://api.github.com/orgs/:org/repos',
intern=T,show.output.on.console = F)
This captures the output (intern = T) and the -s gets rid of the progress bars so I can collapse the lines and turn it into a data frame. This gets back all the repositories, public and private.
I tried using RCurl to do the same thing but the code below only provides the public repositories. The httpheader is because otherwise it the API rejects my call.
RCurl::getURL(url="https://api.github.com/orgs/:org/repos",userpwd ="user:pass",
httpheader = c('User-Agent' = "A user agent"))
I also tried httr and it also only provides the public repositories.
httr::GET(url="https://api.github.com/orgs/:org/repos",userpwd="user:pass")
What am I doing wrong with RCurl and httr? I'd rather have a workflow that doesn't make a system command and then paste the lines together.
We can use the authenticate() helper function in httr to build the authentication header for us w/o having to manually create it. Also, verbose() can be used to debug HTTP issues:
httr::GET(url="https://api.github.com/orgs/:‌​org/repos",
httr::authenticate("user", "pass"),
httr::verbose())

Resources