Related
Good evening everybody,
I am in a fight with PowerQuery. I am trying to recieve an answer from a website in PowerBI (Power Query). For obvious reasons I renamed passwords and such. When I run the code below, PowerQuery answers that:
Expression.Error: The content-length heading must be changed using the
appropriate property or method. Parameter name: name Details: 103
It does not matter if I use another number. I tried also to remove the header entirly but that results in:
DataSource.Error: Web.Contents could not retrieve the content from
'https://xxxx.nl/api/token/' (500): Internal Server Error Details:
DataSourceKind=Web DataSourcePath=https://xxxx.nl/api/token
Url=https://xxxx.nl/api/token/
I do miss something but I cannot find out what that is. Could you find out what it is? Thanks in advance!
let
url = "https://xxxxx.nl/api/token/",
body = Text.ToBinary("{""""username"""":""""xxxx"""",""""password"""":""""xxxx"""",""""group"""":""""xxxx"""",""""deleteOtherSessions"""":false"),
Data = Web.Contents(
url,
[
Headers = [
#"authority" = "xxx.nl",
#"method" = "POST",
#"path" = "/api/Token",
#"scheme" = "https",
#"accept" = "application/json",
#"accept-encoding" = "gzip, deflate",
#"transfer-encoding" = "deflate",
#"accept-language" = "nl-NL,nl;q=0.7",
#"cache-control" = "no-cache",
#"content-length" = "103",
#"content-type" = "application/json",
#"expires" = "Sat, 01 Jan 2000 00:00:00 GMT",
#"origin" = "https://xxxx.nl",
#"pragma" = "no-cache",
#"referer" = "https://xxxxx.nl/",
#"sec-fetch-dest" = "empty",
#"sec-fetch-mode" = "cors",
#"sec-fetch-site" = "same-origin",
#"sec-gpc" = "1",
#"user-agent" = "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/109.0.0.0 Safari/537.36"
],
Content = body
]
)
in
Data
I am getting time-out with GET function from httr package in R with this settings:
GET("https://isir.justice.cz/isir/common/index.do", add_headers(.headers = c('"authority"="isir.justice.cz",
"scheme"="https",
"path"="/isir/common/index.do",
"cache-control"="max-age=0",
"sec-ch-ua-mobile"="?0",
"sec-ch-ua-platform"= "Windows",
"upgrade-insecure-requests"="1",
"accept"="text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9",
"sec-fetch-site"="none",
"sec-fetch-mode"="navigate",
"sec-fetch-user"="?1",
"sec-fetch-dest"="document",
"accept-encoding"="gzip, deflate, br",
"accept-language"="cs-CZ,cs;q=0.9"'
)))
But the seemingly same query via powershell returns a webpage.
Invoke-WebRequest -UseBasicParsing -Uri "https://isir.justice.cz/isir/common/index.do" `
-WebSession $session `
-Headers #{
"method"="GET"
"authority"="isir.justice.cz"
"scheme"="https"
"path"="/isir/common/index.do"
"cache-control"="max-age=0"
"sec-ch-ua-mobile"="?0"
"sec-ch-ua-platform"="`"Windows`""
"upgrade-insecure-requests"="1"
"accept"="text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9"
"sec-fetch-site"="none"
"sec-fetch-mode"="navigate"
"sec-fetch-user"="?1"
"sec-fetch-dest"="document"
"accept-encoding"="gzip, deflate, br"
"accept-language"="cs-CZ,cs;q=0.9"
}
Do I have a problem with my R code or is it simple a matter of difference between using R vs powershell?
Your code didn't run for me as it had an extra ' somewhere. Correcting this, it ran fine. If you keep getting timeout messages, you can increase the maximum request time using timeout():
library(httr)
x <- GET("https://isir.justice.cz/isir/common/index.do", timeout(10), add_headers(
.headers = c("authority" = "isir.justice.cz",
"scheme" = "https",
"path" = "/isir/common/index.do",
"cache-control" = "max-age=0",
"sec-ch-ua-mobile" = "?0",
"sec-ch-ua-platform" = "Windows",
"upgrade-insecure-requests" = "1",
"accept" = "text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9",
"sec-fetch-site" = "none",
"sec-fetch-mode" = "navigate",
"sec-fetch-user" = "?1",
"sec-fetch-dest" = "document",
"accept-encoding" = "gzip, deflate, br",
"accept-language" = "cs-CZ,cs;q=0.9")
))
As a sidenote: there is a successor package by the same people called httr2. I'm also still using httr but it's probably a good idea to learn the new package. Here is how that would look like:
library(httr2)
req <- request("https://isir.justice.cz/isir/common/index.do") %>%
req_headers("authority" = "isir.justice.cz",
"scheme" = "https",
"path" = "/isir/common/index.do",
"cache-control" = "max-age=0",
"sec-ch-ua-mobile" = "?0",
"sec-ch-ua-platform" = "Windows",
"upgrade-insecure-requests" = "1",
"accept" = "text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9",
"sec-fetch-site" = "none",
"sec-fetch-mode" = "navigate",
"sec-fetch-user" = "?1",
"sec-fetch-dest" = "document",
"accept-encoding" = "gzip, deflate, br",
"accept-language" = "cs-CZ,cs;q=0.9") %>%
req_timeout(seconds = 10)
# check your request in a dry run
req %>%
req_dry_run()
#> GET /isir/common/index.do HTTP/1.1
#> Host: isir.justice.cz
#> User-Agent: httr2/0.1.1 r-curl/4.3.2 libcurl/7.80.0
#> authority: isir.justice.cz
#> scheme: https
#> path: /isir/common/index.do
#> cache-control: max-age=0
#> sec-ch-ua-mobile: ?0
#> sec-ch-ua-platform: Windows
#> upgrade-insecure-requests: 1
#> accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9
#> sec-fetch-site: none
#> sec-fetch-mode: navigate
#> sec-fetch-user: ?1
#> sec-fetch-dest: document
#> accept-encoding: gzip, deflate, br
#> accept-language: cs-CZ,cs;q=0.9
resp <- req_perform(req)
resp
#> <httr2_response>
#> GET https://isir.justice.cz/isir/common/index.do
#> Status: 200 OK
#> Content-Type: text/html
#> Body: In memory (116916 bytes)
Created on 2022-01-03 by the reprex package (v2.0.1)
I have a similar problem as: How to translate a POST request from Postman to httr:POST in R. Below is a view of what I do in Postman:
Postman_image_post
In Postman, I receive a response with the requested data. However when I try to use the same URL and body in R using httr:post, I do not receive any data.
The body I want to parse to httr::post (which is identically to the one used in Postman) is
body <- '{ "table": "Fravær løbende år",
"time": [
{
"y1": "2021",
"m1": "07"
}
],
"control": [
"kom_regx",
"hko_krl",
"s_ym"
],
"data": [
"frav_fuldtid",
"frav_dagsv",
"frav_dage_pr_fuldtid"
],
"selection": [
{
"name": "Udvalgte population",
"filters": {
"lif": [
"10101",
"10103",
"10191",
"10102"
]
}
}
],
"options": {
"totals": false,
"outputFormat": "csv",
"actions": [],
"tableName": "Fravær - løbende år",
"subLimit": 5,
"modelName": "Fravær",
"noStkPlus": true,
"timeIncreasing": false,
"hideTimeline": 1
}
}'
I have tried different inputs to the httr::POST, none of it working. For example:
test <- httr::POST('https://www.krl.dk/sirka/sirkaApi/tableApi', body = body, encode = 'json')
When I use verbose() in the function I receive the following:
httr::POST('https://www.krl.dk/sirka/sirkaApi/tableApi', body = body, encode = 'json', verbose())
POST /sirka/sirkaApi/tableApi HTTP/1.1
-> Host: www.krl.dk
-> User-Agent: libcurl/7.64.1 r-curl/4.3 httr/1.4.2
-> Accept-Encoding: deflate, gzip
-> Cookie: sails.sid=s%3A4B90dPjNtz45qB_YM9O_QpAy4eXuFhvg.ML2mJi%2FSh8J%2FVy9hdukQmmyHjUT9OqUt1bvxLKJUO3o
-> Accept: application/json, text/xml, application/xml, */*
-> Content-Length: 689
->
>> { "table": "Fravær løbende år",
>> "time": [
>> {
>> "y1": "2021",
>> "m1": "07"
>> }
>> ],
>> "control": [
>> "kom_regx",
>> "hko_krl",
>> "s_ym"
>> ],
>> "data": [
>> "frav_fuldtid",
>> "frav_dagsv",
>> "frav_dage_pr_fuldtid"
>> ],
>> "selection": [
>> {
>> "name": "Udvalgte population",
>> "filters": {
>> "lif": [
>> "10101",
>> "10103",
>> "10191",
>> "10102"
>> ]
>> }
>> }
>> ],
>> "options": {
>> "totals": false,
>> "outputFormat": "csv",
>> "actions": [],
>> "tableName": "Fravær - løbende år",
>> "subLimit": 5,
>> "modelName": "Fravær",
>> "noStkPlus": true,
>> "timeIncreasing": false,
>> "hideTimeline": 1
>> }
>> }
<- HTTP/1.1 200 OK
<- Cache-Control: no-cache, no-store, must-revalidate
<- Pragma: no-cache
<- Expires: -1
<- Access-Control-Allow-Origin: https://www.krl.dk
<- Connection: close
<- Content-Type: text/html; charset=utf-8
<- Content-Length: 34
<- ETag: W/"22-ccYOpHnkItqR5AChAXo9C/1+jkk"
<- Vary: Accept-Encoding
<- Date: Thu, 04 Nov 2021 11:44:09 GMT
<-
Response [https://www.krl.dk/sirka/sirkaApi/tableApi]
Date: 2021-11-04 11:44
Status: 200
Content-Type: text/html; charset=utf-8
Size: 34 B
Can anyone tell me, what I'm doing wrong in httr:POST, when the URL and body is working fine in Postman-app?
/////////
EDIT
Headers from postman:
I have data being pulled from an API, but for the purposes of this, I will use dummy data:
test <- structure(list(Id = 201:203, firstname = c("Jackie", "Liz", "Jack"),
lastname = c("Jormpjomp", "Lemon", "Donaghy"),
address = c("4 Main St.", "5 Main St.", "6 Main St."),
zip = c(89044L, 60301L, 85281L),
dob = c(NA, "7/1/88", "2/13/90"),
phone = c("333-333-3333","4444", "555-555-5555"),
statecode = c("NV", "WI", "AZ")),
class = "data.frame",
row.names = c(1, 2, 3))
First I isolate all the needed variables as their own values:
Ids <- test$Id
firstnames <- test$firstname
lastnames <- test$lastname
addresses <- test$address
zips <- test$zip
dobs <- test$dob
phones <- test$phone
Then I create a character vector to add on to the final API call:
data_upsert = paste0(
'{ "Id": ', Ids, ',
"firstName": "', firstnames, '",
"lastname": "', lastnames, '",
"dateOfBirth": "', dobs, '",
"phones": [
{ "phoneNumber": "', phones, '" } ],
"addresses": [
{ "addressLine1": "', addresses, '",
"zipOrPostalCode": "', zips, '",
} ] }
')
Then I make a variable for my header - this will stay the same throughout
headers_upsert = c(
`Accept` = 'application/json',
`Authorization` = 'Basic JFOJDIFvdhSOFOHAD83820348voqpTOESV==',
`Content-Type` = 'application/json'
)
Finally, I complete the API call, as such:
upsert <- httr::POST(url = 'https://api.secure.com/v1/people/Create', httr::add_headers(.headers=headers_upsert), body = data_upsert)
Running that creates a response that looks like this:
Response [https://api.secure.com/v1/people/Create]
Date: 2021-08-31 20:28
Status: 201
Content-Type: application/json; charset=utf-8
Size: 58 B
{
"Id": 222323178,
"status": "UnmatchedStored"
I then want to store this response in table form:
resContent <- content(res, as="text")
resJSON <- jsonlite::fromJSON(resContent)
resTable <- as.data.frame(resJSON)
If you run everything above, it obviously only works with the first row in test, but I'm looking for a sleek way to write a function and loop which:
A) Runs the API call for all three rows
B) Creates a table with all three responses
EDIT: Based on Bing's response:
After running Bing's response, it accomplishes the first part, but the issue comes in making the table at the end.
The results for response looks like this:
[[1]]
Response [https://api.secure.com/v1/people/111322450]
Date: 2021-09-01 15:02
Status: 200
Content-Type: application/json; charset=utf-8
Size: 1.56 kB
{
"Id": 111322450,
"firstName": "Jackie",
"lastName": "Jormpjomp",
"middleName": null,
"suffix": null,
"title": "Mr.",
"contactMode": "Person",
"organizationContactCommonName": null,
"organizationContactOfficialName": null,
...
[[2]]
Response [https://api.secure.com/v1/people/findOrCreate/]
Date: 2021-09-01 15:02
Status: 201
Content-Type: application/json; charset=utf-8
Size: 58 B
{
"Id": 111323215,
"status": "UnmatchedStored"
[[3]]
Response [https://api.secure.com/v1/people/findOrCreate/]
Date: 2021-09-01 15:02
Status: 201
Content-Type: application/json; charset=utf-8
Size: 58 B
{
"Id": 111323216,
"status": "UnmatchedStored"
When I run:
resContent=map(response , httr::content, as="text")
resJSON <- map(resContent, jsonlite::fromJSON)
resTable <- map(resJSON, as.data.frame)
resTable is still stored as a List that looks like this EDIT:
$data
$data[[1]]
Response [https://api.secure.com/v1/people/111322450]
Date: 2021-09-01 18:24
Status: 200
Content-Type: application/json; charset=utf-8
Size: 1.58 kB
{
"Id": 111322450,
"firstName": "Jackie",
"lastName": "Jormpjomp",
"middleName": null,
"suffix": null,
"title": null,
"contactMode": "Person",
"organizationContactCommonName": null,
"organizationContactOfficialName": null,
...
$data[[2]]
Response [https://api.secure.com/v1/people/findOrCreate/]
Date: 2021-09-01 18:24
Status: 201
Content-Type: application/json; charset=utf-8
Size: 58 B
{
"Id": 111323215,
"status": "UnmatchedStored"
$data[[3]]
Response [https://api.secure.com/v1/people/findOrCreate/]
Date: 2021-09-01 18:24
Status: 201
Content-Type: application/json; charset=utf-8
Size: 58 B
{
"Id": 111323216,
"status": "UnmatchedStored"
$args
$args[[1]]
[1] "map(jsonlite::fromJSON)"
attr(,"type")
[1] "map"
$args[[2]]
[1] "map(as.data.frame)"
attr(,"type")
[1] "map"
attr(,"class")
[1] "jqr"
The only data I need from each response is Id
EDIT #2
Running the following:
resContent=map(response , httr::content)
resTable <- map(resContent, ~.x$Id) %>% as.data.frame()
returns the following error:
Error in as.data.frame.default(.) :
cannot coerce class ‘"jqr"’ to a data.frame
httr::POST is not vectorized. You will need to loop through each one. You can use lapply or a tidy version like:
library(purrr)
response = map(data_upsert,
~httr::POST(url = 'https://www.google.com/',
httr::add_headers(.headers=headers_upsert),
body = .x))
See if these works. Edits:
resContent=map(response , httr::content)
resTable <- map(resContent, ~.x$Id) #%>% as.data.frame()
I'm trying to query QPX Exprss (Google) from R (httr) but for whatever reason I'm getting 0 results. This is my query:
x <- list(
request = list(
slice = list(c(origin = "BOS", destination = "LAX", date = "2014-07-29")),
passengers = c(adultCount = 1, infantInLapCount = 0, infantInSeatCount = 0,
childCount = 0, seniorCount = 0),
solutions = 10,
refundable = "false")
)
And this is the command:
POST("https://www.googleapis.com/qpxExpress/v1/trips/search?key=MY_KEY",
body = toJSON(x), add_headers("Content-Type" = "application/json"), verbose(),
add_headers(Expect = ""))
Finally, the response from Google:
* About to connect() to www.googleapis.com port 443 (#0)
* Trying 173.194.66.95... * connected
* Connected to www.googleapis.com (173.194.66.95) port 443 (#0)
* successfully set certificate verify locations:
* CAfile: C:/Users/XXXX/Documents/R/win-library/3.1/httr/cacert.pem
CApath: none
* SSL re-using session ID
* SSL connection using ECDHE-RSA-RC4-SHA
* Server certificate:
* subject: C=US; ST=California; L=Mountain View; O=Google Inc; CN=*.googleapis.com
* start date: 2014-07-02 13:35:47 GMT
* expire date: 2014-09-30 00:00:00 GMT
* subjectAltName: www.googleapis.com matched
* issuer: C=US; O=Google Inc; CN=Google Internet Authority G2
* SSL certificate verify ok.
> POST /qpxExpress/v1/trips/search?key=MY_KEY HTTP/1.1
Host: www.googleapis.com
Accept: */*
Accept-Encoding: gzip
user-agent: curl/7.19.6 Rcurl/1.95.4.1 httr/0.3
Content-Type: application/json
Content-Length: 220
< HTTP/1.1 200 OK
< Cache-Control: no-cache, no-store, max-age=0, must-revalidate
< Pragma: no-cache
< Expires: Fri, 01 Jan 1990 00:00:00 GMT
< Date: Mon, 21 Jul 2014 10:39:46 GMT
< ETag: "FHaaT3rgbj6tTc1zJmPkVQ6bD-8/wa9h__cUdEwRE2bp0yW5NTA6fec"
< Content-Type: application/json; charset=UTF-8
< Content-Encoding: gzip
< X-Content-Type-Options: nosniff
< X-Frame-Options: SAMEORIGIN
< X-XSS-Protection: 1; mode=block
< Server: GSE
< Alternate-Protocol: 443:quic
< Transfer-Encoding: chunked
<
* Connection #0 to host www.googleapis.com left intact
Response [https://www.googleapis.com/qpxExpress/v1/trips/search?key=MY_KEY]
Status: 200
Content-type: application/json; charset=UTF-8
{
"kind": "qpxExpress#tripsSearch",
"trips": {
"kind": "qpxexpress#tripOptions",
"requestId": "UTlu4NDcLz3Ypcicp0KKI3",
"data": {
"kind": "qpxexpress#data",
"airport": [
{
"kind": "qpxexpress#airportData", ...
Has anybody had any luck with this?
Thanks a bunch!
Carlos
In case anybody is interested. These are the answers from Duncan and Hadley:
The postForm command from Windows would be as follows:
library(RCurl)
library(RJSONIO)
x <- list(
request = list(
passengers = list(
adultCount = 1
),
slice = list(
list(
origin = "BOS",
destination = "LAX",
date = "2014-07-29"
)
),
refundable = "false",
solutions = 10
)
)
postForm("https://www.googleapis.com/qpxExpress/v1/trips/search?key=my_KEY",
.opts = list(postfields = toJSON(x),
cainfo = system.file("CurlSSL", "cacert.pem", package = "RCurl"),
httpheader = c('Content-Type' = 'application/json',
Accept = 'application/json'),
verbose = TRUE
))
You can omit
cainfo = system.file("CurlSSL", "cacert.pem", package = "RCurl")
if you are running on a Linux machine.
If you rather use httr, here it is the command:
library(httr)
url <- "https://www.googleapis.com/qpxExpress/v1/trips/search"
json <- jsonlite::toJSON(x, auto_unbox = TRUE)
POST(url, query = list(key = my_Key), body = json, content_type_json())
Cheers.
Carlos