Creating a function/loop - r

I have data being pulled from an API, but for the purposes of this, I will use dummy data:
test <- structure(list(Id = 201:203, firstname = c("Jackie", "Liz", "Jack"),
lastname = c("Jormpjomp", "Lemon", "Donaghy"),
address = c("4 Main St.", "5 Main St.", "6 Main St."),
zip = c(89044L, 60301L, 85281L),
dob = c(NA, "7/1/88", "2/13/90"),
phone = c("333-333-3333","4444", "555-555-5555"),
statecode = c("NV", "WI", "AZ")),
class = "data.frame",
row.names = c(1, 2, 3))
First I isolate all the needed variables as their own values:
Ids <- test$Id
firstnames <- test$firstname
lastnames <- test$lastname
addresses <- test$address
zips <- test$zip
dobs <- test$dob
phones <- test$phone
Then I create a character vector to add on to the final API call:
data_upsert = paste0(
'{ "Id": ', Ids, ',
"firstName": "', firstnames, '",
"lastname": "', lastnames, '",
"dateOfBirth": "', dobs, '",
"phones": [
{ "phoneNumber": "', phones, '" } ],
"addresses": [
{ "addressLine1": "', addresses, '",
"zipOrPostalCode": "', zips, '",
} ] }
')
Then I make a variable for my header - this will stay the same throughout
headers_upsert = c(
`Accept` = 'application/json',
`Authorization` = 'Basic JFOJDIFvdhSOFOHAD83820348voqpTOESV==',
`Content-Type` = 'application/json'
)
Finally, I complete the API call, as such:
upsert <- httr::POST(url = 'https://api.secure.com/v1/people/Create', httr::add_headers(.headers=headers_upsert), body = data_upsert)
Running that creates a response that looks like this:
Response [https://api.secure.com/v1/people/Create]
Date: 2021-08-31 20:28
Status: 201
Content-Type: application/json; charset=utf-8
Size: 58 B
{
"Id": 222323178,
"status": "UnmatchedStored"
I then want to store this response in table form:
resContent <- content(res, as="text")
resJSON <- jsonlite::fromJSON(resContent)
resTable <- as.data.frame(resJSON)
If you run everything above, it obviously only works with the first row in test, but I'm looking for a sleek way to write a function and loop which:
A) Runs the API call for all three rows
B) Creates a table with all three responses
EDIT: Based on Bing's response:
After running Bing's response, it accomplishes the first part, but the issue comes in making the table at the end.
The results for response looks like this:
[[1]]
Response [https://api.secure.com/v1/people/111322450]
Date: 2021-09-01 15:02
Status: 200
Content-Type: application/json; charset=utf-8
Size: 1.56 kB
{
"Id": 111322450,
"firstName": "Jackie",
"lastName": "Jormpjomp",
"middleName": null,
"suffix": null,
"title": "Mr.",
"contactMode": "Person",
"organizationContactCommonName": null,
"organizationContactOfficialName": null,
...
[[2]]
Response [https://api.secure.com/v1/people/findOrCreate/]
Date: 2021-09-01 15:02
Status: 201
Content-Type: application/json; charset=utf-8
Size: 58 B
{
"Id": 111323215,
"status": "UnmatchedStored"
[[3]]
Response [https://api.secure.com/v1/people/findOrCreate/]
Date: 2021-09-01 15:02
Status: 201
Content-Type: application/json; charset=utf-8
Size: 58 B
{
"Id": 111323216,
"status": "UnmatchedStored"
When I run:
resContent=map(response , httr::content, as="text")
resJSON <- map(resContent, jsonlite::fromJSON)
resTable <- map(resJSON, as.data.frame)
resTable is still stored as a List that looks like this EDIT:
$data
$data[[1]]
Response [https://api.secure.com/v1/people/111322450]
Date: 2021-09-01 18:24
Status: 200
Content-Type: application/json; charset=utf-8
Size: 1.58 kB
{
"Id": 111322450,
"firstName": "Jackie",
"lastName": "Jormpjomp",
"middleName": null,
"suffix": null,
"title": null,
"contactMode": "Person",
"organizationContactCommonName": null,
"organizationContactOfficialName": null,
...
$data[[2]]
Response [https://api.secure.com/v1/people/findOrCreate/]
Date: 2021-09-01 18:24
Status: 201
Content-Type: application/json; charset=utf-8
Size: 58 B
{
"Id": 111323215,
"status": "UnmatchedStored"
$data[[3]]
Response [https://api.secure.com/v1/people/findOrCreate/]
Date: 2021-09-01 18:24
Status: 201
Content-Type: application/json; charset=utf-8
Size: 58 B
{
"Id": 111323216,
"status": "UnmatchedStored"
$args
$args[[1]]
[1] "map(jsonlite::fromJSON)"
attr(,"type")
[1] "map"
$args[[2]]
[1] "map(as.data.frame)"
attr(,"type")
[1] "map"
attr(,"class")
[1] "jqr"
The only data I need from each response is Id
EDIT #2
Running the following:
resContent=map(response , httr::content)
resTable <- map(resContent, ~.x$Id) %>% as.data.frame()
returns the following error:
Error in as.data.frame.default(.) :
cannot coerce class ‘"jqr"’ to a data.frame

httr::POST is not vectorized. You will need to loop through each one. You can use lapply or a tidy version like:
library(purrr)
response = map(data_upsert,
~httr::POST(url = 'https://www.google.com/',
httr::add_headers(.headers=headers_upsert),
body = .x))
See if these works. Edits:
resContent=map(response , httr::content)
resTable <- map(resContent, ~.x$Id) #%>% as.data.frame()

Related

Microsoft Graph query parameter in Karate test

I am trying to take follow Postman Get request to Microsoft Graph API and convert it into Karate test
https://graph.microsoft.com/v1.0/users/moo#moo.com/messages?$search="body:'979f13ea-5c87-45e3-98e2-7243d321b238'"
The issue I am having is how to handle the query parameters with the single quote inside the double quotes.
Try this:
* url 'https://httpbin.org/anything'
* param $search = `"body:'979f13ea-5c87-45e3-98e2-7243d321b238'"`
* method get
Actual request:
1 > GET https://httpbin.org/anything?%24search=%22body%3A%27979f13ea-5c87-45e3-98e2-7243d321b238%27%22
1 > Host: httpbin.org
1 > Connection: Keep-Alive
1 > User-Agent: Apache-HttpClient/4.5.14 (Java/17.0.5)
1 > Accept-Encoding: gzip,deflate
But, you can see from the server response that the data was encoded correctly:
1 < 200
1 < Date: Mon, 09 Jan 2023 18:52:15 GMT
1 < Content-Type: application/json
1 < Content-Length: 516
1 < Connection: keep-alive
1 < Server: gunicorn/19.9.0
1 < Access-Control-Allow-Origin: *
1 < Access-Control-Allow-Credentials: true
{
"args": {
"$search": "\"body:'979f13ea-5c87-45e3-98e2-7243d321b238'\""
},
"data": "",
"files": {},
"form": {},
"headers": {
"Accept-Encoding": "gzip,deflate",
"Host": "httpbin.org",
"User-Agent": "Apache-HttpClient/4.5.14 (Java/17.0.5)",
"X-Amzn-Trace-Id": "Root=1-63bc625f-36a4b2e92b1976b303454a8a"
},
"json": null,
"method": "GET",
"origin": "49.205.149.94",
"url": "https://httpbin.org/anything?%24search=\"body%3A'979f13ea-5c87-45e3-98e2-7243d321b238'\""
}
Using back-ticks gives you a nice option to dynamically change data:
* def id = '979f13ea-5c87-45e3-98e2-7243d321b238'
* param $search = `"body:'${id}'"`
Escaping the single-quote would also work:
* param $search = '"body:\'979f13ea-5c87-45e3-98e2-7243d321b238\'"'
Also see: https://stackoverflow.com/a/59977660/143475

How to write a POST request using httr:POST similar to one used in Postman-app

I have a similar problem as: How to translate a POST request from Postman to httr:POST in R. Below is a view of what I do in Postman:
Postman_image_post
In Postman, I receive a response with the requested data. However when I try to use the same URL and body in R using httr:post, I do not receive any data.
The body I want to parse to httr::post (which is identically to the one used in Postman) is
body <- '{ "table": "Fravær løbende år",
"time": [
{
"y1": "2021",
"m1": "07"
}
],
"control": [
"kom_regx",
"hko_krl",
"s_ym"
],
"data": [
"frav_fuldtid",
"frav_dagsv",
"frav_dage_pr_fuldtid"
],
"selection": [
{
"name": "Udvalgte population",
"filters": {
"lif": [
"10101",
"10103",
"10191",
"10102"
]
}
}
],
"options": {
"totals": false,
"outputFormat": "csv",
"actions": [],
"tableName": "Fravær - løbende år",
"subLimit": 5,
"modelName": "Fravær",
"noStkPlus": true,
"timeIncreasing": false,
"hideTimeline": 1
}
}'
I have tried different inputs to the httr::POST, none of it working. For example:
test <- httr::POST('https://www.krl.dk/sirka/sirkaApi/tableApi', body = body, encode = 'json')
When I use verbose() in the function I receive the following:
httr::POST('https://www.krl.dk/sirka/sirkaApi/tableApi', body = body, encode = 'json', verbose())
POST /sirka/sirkaApi/tableApi HTTP/1.1
-> Host: www.krl.dk
-> User-Agent: libcurl/7.64.1 r-curl/4.3 httr/1.4.2
-> Accept-Encoding: deflate, gzip
-> Cookie: sails.sid=s%3A4B90dPjNtz45qB_YM9O_QpAy4eXuFhvg.ML2mJi%2FSh8J%2FVy9hdukQmmyHjUT9OqUt1bvxLKJUO3o
-> Accept: application/json, text/xml, application/xml, */*
-> Content-Length: 689
->
>> { "table": "Fravær løbende år",
>> "time": [
>> {
>> "y1": "2021",
>> "m1": "07"
>> }
>> ],
>> "control": [
>> "kom_regx",
>> "hko_krl",
>> "s_ym"
>> ],
>> "data": [
>> "frav_fuldtid",
>> "frav_dagsv",
>> "frav_dage_pr_fuldtid"
>> ],
>> "selection": [
>> {
>> "name": "Udvalgte population",
>> "filters": {
>> "lif": [
>> "10101",
>> "10103",
>> "10191",
>> "10102"
>> ]
>> }
>> }
>> ],
>> "options": {
>> "totals": false,
>> "outputFormat": "csv",
>> "actions": [],
>> "tableName": "Fravær - løbende år",
>> "subLimit": 5,
>> "modelName": "Fravær",
>> "noStkPlus": true,
>> "timeIncreasing": false,
>> "hideTimeline": 1
>> }
>> }
<- HTTP/1.1 200 OK
<- Cache-Control: no-cache, no-store, must-revalidate
<- Pragma: no-cache
<- Expires: -1
<- Access-Control-Allow-Origin: https://www.krl.dk
<- Connection: close
<- Content-Type: text/html; charset=utf-8
<- Content-Length: 34
<- ETag: W/"22-ccYOpHnkItqR5AChAXo9C/1+jkk"
<- Vary: Accept-Encoding
<- Date: Thu, 04 Nov 2021 11:44:09 GMT
<-
Response [https://www.krl.dk/sirka/sirkaApi/tableApi]
Date: 2021-11-04 11:44
Status: 200
Content-Type: text/html; charset=utf-8
Size: 34 B
Can anyone tell me, what I'm doing wrong in httr:POST, when the URL and body is working fine in Postman-app?
/////////
EDIT
Headers from postman:

Converting a cURL command to HTTR in R

I have a cURL command that I would like to convert to R using the httr package. The cURL command looks something like this (I obviously can't post the full scope of it, given certain confidentiality restrictions):
curl --request POST \
--url https://api.cfb.com/v1/players/110960703/stats \
--header 'Accept: application/json' \
--header 'Authorization: Basic ThisIsWhereTheAPIKeyGoes==' \
--header 'Content-Type: application/json' \
--data '
{
"statsContext": {
"statsId": "1"
},
"responses": [
{
"type": "ReceiverStats",
"QuestionId": "466069",
"ResponseId": "1898226"
}
]
}
'
I would like to convert it to an HTTR script that looks like this:
ExportIdAPI <- POST(
'https://api.cfb.com/v1/players/110960703/stats',
accept_json(),
content_type_json(),
add_headers(Authorization = Basic ThisIsWhereTheAPIKeyGoes==),
encode = "json",
body=list(statsId = 1,
QuestionId = 466069,
ResponseId = 1898226))
The problem is, this script does not work. What does work is this script:
ExportIdAPI <- POST(
'https://api.cfb.com/v1/players/110960703/stats',
accept_json(),
content_type_json(),
add_headers(Authorization = Basic ThisIsWhereTheAPIKeyGoes==),
encode = "json",
body=list(statsId = 1))
Based on how the original curl request is structured, am I just not storing it right in the body=list statement?
EDIT: I should note that this script runs perfectly fine. I just don't want it to be structured that way and I would like it to better reflect the format I'm trying to achieve from above.
headers = c(
`Accept` = 'application/json',
`Authorization` = 'Basic ThisIsWhereTheAPIKeyGoes==',
`Content-Type` = 'application/json'
)
data = ' { "statsContext": { "statsId": "1" }, "responses": [ { "type": "SurveyResponse", "QuestionId": "466069", "ResponseId": "1898226" } ] } '
res <- httr::POST(url = 'https://api.cfb.com/v1/players/110960703/stats', httr::add_headers(.headers=headers), body = data)
EDIT 2: Based on MrFlick's response, I tried running:
ExportIdAPI <- POST(
'https://api.cfb.com/v1/players/110960703/stats',
accept_json(),
content_type_json(),
add_headers(Authorization = Basic ThisIsWhereTheAPIKeyGoes==),
encode = "json",
data = list(statsContext = list(statsId = 1), responses= list(list(type="SurveyResponse", QuestionId="466069", ResponseId="1898226")))
)
But I'm met with the following error:
Response [https://api.cfb.com/v1/players/110960703/stats]
Date: 2021-08-26 20:58
Status: 400
Content-Type: application/json; charset=utf-8
Size: 144 B
{
"errors": [
{
"code": "INVALID_PARAMETER",
"text": "The body of the request is null or cannot be parsed."
}
]
The httr documentation for the POST function provides an example for how to post a json.
In your case, you could really just copy the json of the cURL command to the body parameter of the POST function:
ExportIdAPI <- POST(
'https://api.cfb.com/v1/players/110960703/stats',
accept_json(),
content_type_json(),
add_headers(Authorization = 'Basic ThisIsWhereTheAPIKeyGoes=='),
body = '
{
"statsContext": {
"statsId": "1"
},
"responses": [
{
"type": "ReceiverStats",
"QuestionId": "466069",
"ResponseId": "1898226"
}
]
}', encode = "raw")
You can have the ROST() request format the JSON for you. As a general rule named lists are turned into JSON objects {} and unnamed list become arrays []. You can do
POST(
'https://api.cfb.com/v1/players/110960703/stats',
accept_json(),
content_type_json(),
add_headers(Authorization = "Basic ThisIsWhereTheAPIKeyGoes=="),
encode = "json",
body = list(
statsContext = list(
statsId = 1
),
responses= list(
list(
type="SurveyResponse",
QuestionId="466069",
ResponseId="1898226"
)
)
)
)

How to Post API in R having header & json body

How to call API Post in R
Request URL
https://westus.api.cognitive.microsoft.com/text/analytics/v2.0/sentiment
Request headers
Ocp-Apim-Subscription-Key = some value &
Content-Type = application/json
Body application/json
{
"documents": [
{
"language": "string",
"id": "string",
"text": "string"
}
]
}
Please help!!!
Here is the example -
request_body <- data.frame(
language = c("en","en"),
id = c("1","2"),
text = c("This is wasted! I'm angry","This is awesome! Good Job Team! appreciated")
)
Converting the Request body(Dataframe) to Request body(JSON)
require(jsonlite)
request_body_json <- toJSON(list(documents = request_body), auto_unbox = TRUE)
Below we are calling API (Adding Request headers using add_headers)
require(httr)
result <- POST("https://westus.api.cognitive.microsoft.com/text/analytics/v2.0/sentiment",
body = request_body_json,
add_headers(.headers = c("Content-Type"="application/json","Ocp-Apim-Subscription-Key"="my_subscrition_key")))
Output <- content(result)
Show Output
Output

how to parse nested key value pairs in R

i am trying to parse a log file which contains structure in the form of key value pairs.
log <- c("name:praveen,age:23,place:UP,address:,dob:, site: {site_name:something , site_url: http://something.com, description:}")
i am trying to parse this line i have done some work but i have two main problem here .
1: How can i parse "site" variable (shown above) because for site key there are multiple key:value pair ?
2: How to tackle with condition if separator comes as a string . like for key:value pair separator is colon (:) and in the "site" key there is a key:value pair site_url:http://something.com here url also contains colon (:) which gives the wrong answer.
this is my code it does not contains "site" key becuase i don't know how to parse it
log <- c("name:praveen,age:23,place:UP,address:,dob:")
names <- setNames(1:5,c("name","age","place","address","dob"))
assign <- function(x, names){
key_value <- sapply(x, function(i)if(length(i)==2L) i else c(i, "nothing"))
z <- rep(NA, length(names))
z[names[key_value[1, ]]] <- key_value[2, ]
z
}
split_by_comma <- strsplit(log,",")
split_by_colon <- lapply(split_by_comma,strsplit,":")
ret <- t(sapply(split_by_colon, assign, names))
colnames(ret) <- names(names)
ret
please help me to parse this file thank you
i have updated with actual log file format.
{
"username": "lavita",
"host": "10.105.22.32",
"event_source": "server",
"event_type": "/courses/IITB/CS101/2014_T1/xblock/i4x:;_;_IITB;_CS101;_video;_d333fa637a074b41996dc2fd5e675818/handler/xmodule_handler/save_user_state",
"context": {
"course_id": "IITB/CS101/2014_T1",
"course_user_tags": {},
"user_id": 42,
"org_id": "IITB"
},
"time": "2014-06-20T05:49:10.468638+00:00",
"ip": "127.0.0.1",
"event": "{\"POST\": {\"saved_video_position\": [\"00:02:10\"]}, \"GET\": {}}",
"agent": "Mozilla/5.0 (X11; Ubuntu; Linux i686; rv:18.0) Gecko/20100101 Firefox/18.0",
"page": null
}
{
"username": "raeha",
"host": "10.105.22.32",
"event_source": "server",
"event_type": "problem_check",
"context": {
"course_id": "IITB/CS101/2014_T1",
"course_user_tags": {},
"user_id": 40,
"org_id": "IITB",
"module": {
"display_name": ""
}
},
"time": "2014-06-20T06:43:52.716455+00:00",
"ip": "127.0.0.1",
"event": {
"submission": {
"i4x-IITB-CS101-problem-33e4aac93dc84f368c93b1d08fa984fc_2_1": {
"input_type": "choicegroup",
"question": "",
"response_type": "multiplechoiceresponse",
"answer": "MenuInflater.inflate()",
"variant": "",
"correct": true
}
},
"success": "correct",
"grade": 1,
"correct_map": {
"i4x-IITB-CS101-problem-33e4aac93dc84f368c93b1d08fa984fc_2_1": {
"hint": "",
"hintmode": null,
"correctness": "correct",
"npoints": null,
"msg": "",
"queuestate": null
}
},
"state": {
"student_answers": {},
"seed": 1,
"done": null,
"correct_map": {},
"input_state": {
"i4x-IITB-CS101-problem-33e4aac93dc84f368c93b1d08fa984fc_2_1": {}
}
},
"answers": {
"i4x-IITB-CS101-problem-33e4aac93dc84f368c93b1d08fa984fc_2_1": "choice_0"
},
"attempts": 1,
"max_grade": 1,
"problem_id": "i4x://IITB/CS101/problem/33e4aac93dc84f368c93b1d08fa984fc"
},
"agent": "Mozilla/5.0 (X11; Ubuntu; Linux i686; rv:29.0) Gecko/20100101 Firefox/29.0",
"page": "x_module"
}
{
"username": "tushars",
"host": "localhost",
"event_source": "server",
"event_type": "/courses/IITB/CS101/2014_T1/instructor_dashboard/api/list_instructor_tasks",
"context": {
"course_id": "IITB/CS101/2014_T1",
"course_user_tags": {},
"user_id": 6,
"org_id": "IITB"
},
"time": "2014-06-20T05:49:26.780244+00:00",
"ip": "127.0.0.1",
"event": "{\"POST\": {}, \"GET\": {}}",
"agent": "Mozilla/5.0 (X11; Ubuntu; Linux i686; rv:29.0) Gecko/20100101 Firefox/29.0",
"page": null
}
This is a pretty ugly format. True json would have quoted strings and non-empty values so it's not really a standard format. Here's a method that's equally as ugly, but it can handle multiple nested elements.
I'll use this as a test case
log <- paste0("name:{first:praveen,last:smith},age:23,place:UP,address:,",
"dob:, site: {site_name:something , site_url: http://something.com, ",
"description:{english:woot,spanish:wooto}}")
And here's the parser
parseString<-function(log) {
nested<-c()
#find {} blocks and replace
m<-regexec("\\{[^}{]+?\\}", log)
while(sapply(m, `[`, 1)!=-1) {
s <- gsub("^\\{|\\}$","",sapply(regmatches(log,m), `[`, 1))
regmatches(log,m)<-paste0("~~", length(nested)+seq_along(s), "~~")
nested<-c(nested,s)
m<-gregexpr("\\{([^}{]+)\\}", log)
}
nested<-c(nested, log)
#turn elements into list
nestedl<-vector("list", length(nested))
for(i in seq_along(nested)) {
kv<-strsplit(nested[i], "\\s*,\\s*")[[1]]
kv<-lapply(strsplit(kv, ":"), function(x)
c(x[1], paste(x[-1],collapse=":")))
names <- gsub("\\s+","", sapply(kv, `[`,1))
vals <- gsub("\\s+","", sapply(kv, `[`,2))
valsl <- setNames(as.list(vals), names)
m <- regexec("~~(\\d+)~~", vals)
for(j in which(sapply(m, `[`, 1) != -1)) {
valsl[[j]]<-nestedl[[as.numeric(regmatches(vals[j], m[j])[[1]][2])]]
}
nestedl[[i]]<-valsl
}
nestedl[[length(nestedl)]]
}
So the strategy is to find the "{}" blocks and collapse them down to a simple string we can find again later; in this case i use "~~1~~" where the number in the middle is a unique ID for each block. I do this till i only have a set of name value pairs. The I go back, look for all the "~~" values and merge the correct sublist back in. For this test data, I get
#parseString(log)
$name
$name$first
[1] "praveen"
$name$last
[1] "smith"
$age
[1] "23"
$place
[1] "UP"
$address
[1] ""
$dob
[1] ""
$site
$site$site_name
[1] "something"
$site$site_url
[1] "http://something.com"
$site$description
$site$description$english
[1] "woot"
$site$description$spanish
[1] "wooto"

Resources