R how do I switch off JSON outgoing/incoming header messages? - r

Question

I have an issue using an R script as a data source in Microsoft PowerBi. I think this is fundermentally an issue with PowerBi, but in the short term I'll need to find a solution in R.
Essentially, PowerBi doesn't appear to be able to handle the messages that would be sent to the console if I was using R Studio.
Within the R script I'm using a REST API to request data from a URL. The JSON message that is received is converted into an R data frame. When using the script as a datasource in PowerBi, this only works if I set the verbose settings to FALSE i.e. if I was using R Studio no messages (in particular data in) are sent to the console.
response <- GET(<url>,
body = list(),
add_headers(.headers = c('<identity token>' = ID_to_use)),
verbose(data_out = FALSE,
data_in = FALSE,
info = FALSE,
ssl = FALSE),
encode = "json")
However, I do not have the option to switch off the incoming/outgoing JSON header messages (which is going to come back to bite!).
<< {"identity":" <token>"}
* Connection #54 to <host> left intact
No encoding supplied: defaulting to UTF-8.
-> GET <URL request> HTTP/1.1
-> Host: <host>
-> User-Agent: libcurl/7.64.1 r-curl/4.3 httr/1.4.1
-> Accept-Encoding: deflate, gzip
-> Accept: application/json, text/xml, application/xml, */*
-> <Identity>: <Identity>
->
<- HTTP/1.1 200 OK
<- X-Session-Expiry: 3599
<- Content-Type: application/json
<- Transfer-Encoding: chunked
<- Date: Thu, 06 Aug 2020 16:14:26 GMT
<- Server: <Server>
<-
No encoding supplied: defaulting to UTF-8.
No encoding supplied: defaulting to UTF-8.
No encoding supplied: defaulting to UTF-8.
From R help
.
.
verbose() uses the following prefixes to distinguish between different components of the http messages:
* informative curl messages
-> headers sent (out)
>> data sent (out)
*> ssl data sent (out)
<- headers received (in)
<< data received (in)
<* ssl data received (in)
.
.
Switching the verbose settings to FALSE works for a single request, however, I need to put the request into a loop and keep requesting more data until the API gateway indicates there is no more data to be received. PowerBi appears to fail when in the script five or more request/replies are sent/received.
Just from observation, I assume this is to do with the JSON Header messages piling up.
I've tried a number of approaches but nothing seems to work: sink('NUL'), invisible(), capture.output().
Any help would be appreciated.

Answer 1

I found a hacky solution, which at least solved the problem I had in R, but not in PowerBi.
By writing a "wrapper" R script (see below) which calls my main script THE_SCRIPT.R using a shell command. THE_SCRIPT dumps out a CSV file, which I then read in the wrapper script:
#Required by PowerBi
library(mice)
#set the directory, between R and the shell it's a pain to deal with spaces in the directories and quotes
setwd("C:/Program Files/R/R-3.6.2/bin/")
system("Rscript.exe C:\\Users\\<USER>\\Documents\\THE_SCRIPT.R > Nul 2>&1")
A_DATA_TABLE <- read.csv("C:\\Users\\<USER>\\Documents\\THE_FILE.csv")
However, this still didn't resolve the issue when running it in PowerBi.
Note, I tried sink('Nul 2>&1') in R, didn't work.