I have this code to get a response from a Web API
require("httr")
r <- POST("https://example.com/API/Bulk/2.0/Contacts/exports/",
config = list("Authorization"="Basic abc==","Accept"="application/json",
"body"= "{ 'name':'JB Test - Contact Lead Score Export','fields':{'ID':'{{Contact.Field(ContactIDEXT)}}'}}"))
r
I get the following error message:
Response [example.com/API/Bulk/2.0/Contacts/exports/] Date: 2018-03-02 12:49 Status: 400 Content-Type: application/json; charset=utf-8 Size: 175 B { "failures":[{"field":"name","constraint":"Must be a string value, at least 1 character and at most 100 characters long."},{"field":"fields","constraint":"Is required."}]}
I am getting failure message.
Whats wrong in this code?
Related
I'm learning to use APIs with R's httr package and the US Data Census API as my API.
When I make the request by manually entering the URL in the GET(), the request works.
try_using_manual<- httr::GET("https://api.census.gov/data/2014/pep/natstprc?get=STNAME,POP&DATE=7&for=state:01&key=XXXX")
> try_using_manual
Response [https://api.census.gov/data/2014/pep/natstprc?get=STNAME,POP&DATE=7&for=state:01&key=7e3d28d2d9facb45c12a0b63f9d31b709d917dda]
Date: 2018-12-08 03:08
Status: 200
Content-Type: application/json;charset=utf-8
Size: 65 B
[["STNAME","POP","DATE","state"],
When I try to do the same request, but instead building arguments from the GET() function, my request fails.
> try_using_arg <- httr::GET(url = "https://api.census.gov/data/2014/pep/natstprc",
+ query = list("STNAME"=":*","POP"=":*","DATE"=7))
> try_using_arg
Response [https://api.census.gov/data/2014/pep/natstprc?STNAME=%3A%2A&POP=%3A%2A&DATE=7]
Date: 2018-12-08 03:20
Status: 400
Content-Type: text/plain;charset=ISO-8859-1
Size: 74 B
Any ideas on how to call those variables, STNAME,POP,DATE,state as seen in try_using_manual in try_using_arg?
Any and all help is appreciated!
I have a list that contains 4 objects of type Response, as in an API response:
Response [https://api.idealista.com/3.5/es/search?&operation= etc. etc.]
Date: 2018-06-04 12:27
Status: 200
Content-Type: application/json;charset=UTF-8
Size: 45 kB
Suppose the list is called holle, I can access the contents and reassign them to another list, revs, as follows:
library(httr)
library(rlist)
revs[[1]] <- content(holle[[1]])$elementList
This works perfectly fine and all is well. However, I would like to seq_along each element and access the contents. When I write a for/seq_along, I get this error message:
for (i in seq_along(content(holle)$elementList)){
revs[[i]] <- content(holle[[i]])$elementList
}
"Error in content(holle) : is.response(x) is not TRUE".
Why?
I seem to be having an issue with our riak-kv db v2.2.3, we are currently running 5 node cluster, out of nowhere last week suddenly requests to our db started returning 500 internal server error for all/almost every request (also might be valuable to mention our db has nearly doubled in size over the past 2 weeks.) At first, we thought the issue was in the code making the requests however after sshing into one of the nodes in the cluster and attempting a simple request to list buckets we saw this:
Command:
curl -i http://localhost:8098/buckets?buckets=true
Response:
HTTP/1.1 500 Internal Server Error
Vary: Accept-Encoding
Server: MochiWeb/1.1 WebMachine/1.10.9 (cafe not found)
Date: Sun, 10 Dec 2017 06:21:20 GMT
Content-Type: text/html
Content-Length: 1193
<html><head><title>500 Internal Server Error</title></head><body>
<h1>Internal Server Error</h1>The server encountered an error while
processing this request:<br><pre>{error,
{error,
{badmatch,{error,mailbox_overload}},
[{riak_kv_wm_buckets,produce_bucket_list,2,
[{file,"src/riak_kv_wm_buckets.erl"},{line,225}]},
{webmachine_resource,resource_call,3,
[{file,"src/webmachine_resource.erl"},{line,186}]},
{webmachine_resource,do,3,
[{file,"src/webmachine_resource.erl"},{line,142}]},
{webmachine_decision_core,resource_call,1,
[{file,"src/webmachine_decision_core.erl"},{line,48}]},
{webmachine_decision_core,decision,1,
[{file,"src/webmachine_decision_core.erl"},{line,562}]},
{webmachine_decision_core,handle_request,2,
[{file,"src/webmachine_decision_core.erl"},{line,33}]},
{webmachine_mochiweb,loop,2,
[{file,"src/webmachine_mochiweb.erl"},{line,72}]},
{mochiweb_http,headers,5,
[{file,"src/mochiweb_http.erl"},{line,105}]}]}}</pre><P><HR>
<ADDRESS>mochiweb+webmachine web server</ADDRESS></body></html>
After some more investigation into the issue, I pulled the logs on one of the riak nodes and saw this:
2017-12-10 03:54:58.654 [error]
<0.24342.271>#yz_solrq_helper:send_solr_ops_for_entries:301 Updating a
batch of Solr operations failed for index <<"attachment">> with error
{error,{other,{ok,"500",[{"Content-Type","application/json;
charset=UTF-8"},{"Transfer-Encoding","chunked"}],<<"
{\"responseHeader\":{\"status\":500,\"QTime\":1}, error": {
"msg ": "Exception writing document id 1*default*Attachment*07d8e24-dc32s-11e7-q9640-a7f8b4edb446*623 to the index; possible analysis error.",
"trace": ""org.apache.solr.common.So lrException: Exception writing document id 1*default*Attachment*07d8e24-dc32s-11e7-q9640-a7f8b4edb446*623 to the index; possible analysis error.
at org.apache.solr.update. DirectUpdateHandler2.addDoc(DirectUpdateHandler2.java:169)
at org.apache.solr.update.processor.RunUpdateProcessor.processAdd(RunUpdateProcessorFactory.java:69)
at or g.apache.solr.update.processor.UpdateRequestProcessor.processAdd(UpdateRequestProcessor.java:51)
at org.apache.solr.update.processor.DistributedUpdateProcessor.versionAdd (DistributedUpdateProcessor.java:952)
at org.apache.solr.update.processor.DistributedUpdateProcessor.processAdd(DistributedUpdateProcessor.java:692)
at org.apache.so lr.handler.loader.JsonLoader$SingleThreadedJsonLoader.processUpdate(JsonLoader.java:141)
at org.apache.solr.handler.loader.JsonLoader$SingleThreadedJsonLoader.load(JsonLo ader.java:106)
at org.apache.solr.handler.loader.JsonLoader.load(JsonLoader.java:68)
at org.apache.solr.handler.UpdateRequestHandler$1.load(UpdateRequestHandler.java :99)
at org.apache.solr.handler.ContentStreamHandlerBase.handleRequestBody(ContentStreamHandlerBase.java:74)
at org.apache.solr.handler.RequestHandlerBase.handleRequ est(RequestHandlerBase.java:135)
at org.apache.solr.core.SolrCore.execute(SolrCore.java:1976)
at org.apache.solr.servlet.SolrDispatchFilter.execute(SolrDispatchFilte r.java:777)
at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:418)
at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFi lter.java:207)
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1419)
at org.eclipse.jetty.servlet.ServletHandler.doHandle(Servle tHandler.java:455)
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:137)
at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHand ler.java:557)
at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:231)
at org.eclipse.jetty.server.handler.ContextHandler.doHandle(Contex tHandler.java:1075)
at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:384)
at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHa ndler.java:193)
at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1009)
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedH andler.java:135)
at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:255)
at org.eclipse.jetty.server.handler.HandlerCo llection.handle(HandlerCollection.java:154)
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:116)
at org.eclipse.jetty.server.Server.han dle(Server.java:368)
at org.eclipse.jetty.server.AbstractHttpConnection.handleRequest(AbstractHttpConnection.java:489)
at org.eclipse.jetty.server.BlockingHttpConnec tion.handleRequest(BlockingHttpConnection.java:53)
at org.eclipse.jetty.server.AbstractHttpConnection.content(AbstractHttpConnection.java:953)
at org.eclipse.jetty.s erver.AbstractHttpConnection$RequestHandler.content(AbstractHttpConnection.java:1014)
at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:861)
at org.ecli pse.jetty.http.HttpParser.parseAvailable(HttpParser.java:235)
at org.eclipse.jetty.server.BlockingHttpConnection.handle(BlockingHttpConnection.java:72)
at org.eclips e.jetty.server.bio.SocketConnector$ConnectorEndPoint.run(SocketConnector.java:264)
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java :608)
at org.eclip..." >>
}
}
}
After some googling some people said this error is
Caused by: java.lang.OutOfMemoryError: Java heap space
So I modified my riak config from:
search.solr.jvm_options = -d64 -Xms4g -Xmx4g -XX:+UseStringCache -XX:+UseCompressedOops
To:
search.solr.jvm_options = -d64 -Xms8g -Xmx8g -XX:+UseStringCache -XX:+UseCompressedOops
However I am still getting the same error any help is much appreciated.
I am trying to access a private repository on Github using httr. I am able to do so with no problem if I add my github token (stored as an environment variable in GITHUB_TOKEN):
httr::GET("https://api.github.com/repos/aammd/miniature-meme/releases/assets/2859674",
httr::write_disk("test.rds", overwrite = TRUE),
httr::progress("down"),
httr::add_headers(Authorization = paste("token", Sys.getenv("GITHUB_TOKEN"))))
However, if I try to specify another header, I get an error. In this case, I want to download the binary file associated with a release (the "asset", in github terminology):
httr::GET("https://api.github.com/repos/aammd/miniature-meme/releases/assets/2859674",
httr::write_disk("test.rds", overwrite = TRUE),
httr::progress("down"),
httr::add_headers(Authorization = paste("token", Sys.getenv("GITHUB_TOKEN"))),
httr::add_headers(Accept = "application/octet-stream"))
?xml version="1.0" encoding="UTF-8"?>
<Error><Code>InvalidArgument</Code><Message>Only one auth mechanism allowed; only the X-Amz-Algorithm query parameter, Signature query string parameter or the Authorization header should be specified</Message>
That's only part of the message (the rest includes my token).
Apparently my authorization is being sent twice! How can I prevent this? Is it related to httr::handle_pool()
EDIT -- connection info
It appears that the original request receives a reply, which contains a signature. This signature, along with my token is then sent back, causing an error. A similar thing happened to these people
-> GET /repos/aammd/miniature-meme/releases/assets/2859674 HTTP/1.1
-> Host: api.github.com
-> User-Agent: libcurl/7.43.0 r-curl/2.3 httr/1.2.1.9000
-> Accept-Encoding: gzip, deflate
-> Authorization: token tttttttt
-> Accept: application/octet-stream
->
<- HTTP/1.1 302 Found
<- Server: GitHub.com
<- Date: Tue, 17 Jan 2017 13:28:12 GMT
<- Content-Type: text/html;charset=utf-8
<- Content-Length: 0
<- Status: 302 Found
<- X-RateLimit-Limit: 5000
<- X-RateLimit-Remaining: 4984
<- X-RateLimit-Reset: 1484662101
<- location: https://github-cloud.s3.amazonaws.com/releases/76993567/aee5d0d6-c70a-11e6-9078-b5bee39f9fbc.RDS?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAISTNZFOVBIJMK3TQ%2F20170117%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20170117T132812Z&X-Amz-Expires=300&X-Amz-Signature=ssssssssss&X-Amz-SignedHeaders=host&actor_id=1198242&response-content-disposition=attachment%3B%20filename%3Dff.RDS&response-content-type=application%2Foctet-stream
<- Access-Control-Expose-Headers: ETag, Link, X-GitHub-OTP, X-RateLimit-Limit, X-RateLimit-Remaining, X-RateLimit-Reset, X-OAuth-Scopes, X-Accepted-OAuth-Scopes, X-Poll-Interval
<- Access-Control-Allow-Origin: *
<- Content-Security-Policy: default-src 'none'
<- Strict-Transport-Security: max-age=31536000; includeSubdomains; preload
<- X-Content-Type-Options: nosniff
<- X-Frame-Options: deny
<- X-XSS-Protection: 1; mode=block
<- Vary: Accept-Encoding
<- X-Served-By: 3e3b9690823fb031da84658eb58aa83b
<- X-GitHub-Request-Id: 82782802:6E1B:E9F0BE:587E1BEC
<-
-> GET /releases/76993567/aee5d0d6-c70a-11e6-9078-b5bee39f9fbc.RDS?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAISTNZFOVBIJMK3TQ%2F20170117%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20170117T132812Z&X-Amz-Expires=300&X-Amz-Signature=sssssssssssssss&X-Amz-SignedHeaders=host&actor_id=1198242&response-content-disposition=attachment%3B%20filename%3Dff.RDS&response-content-type=application%2Foctet-stream HTTP/1.1
-> Host: github-cloud.s3.amazonaws.com
-> User-Agent: libcurl/7.43.0 r-curl/2.3 httr/1.2.1.9000
-> Accept-Encoding: gzip, deflate
-> Authorization: token ttttttttttttt
-> Accept: application/octet-stream
->
<- HTTP/1.1 400 Bad Request
<- x-amz-request-id: FA56B3D23B468704
<- x-amz-id-2: 49X1mT5j5BrZ4HApeR/+wb7iVOWA8yn1obrgMoeOy44RH414bo/Ov8AAWSx2baEXO0H/WHX5jK0=
<- Content-Type: application/xml
<- Transfer-Encoding: chunked
<- Date: Tue, 17 Jan 2017 13:28:12 GMT
<- Connection: close
<- Server: AmazonS3
<-
gh doesn't work either
I created a public repo to test this idea out. the JSON can be returned from the API, but not the binary file:
# this works fine
gh::gh("https://api.github.com/repos/aammd/test_idea/releases/assets/2998763")
# this does not
gh::gh("https://api.github.com/repos/aammd/test_idea/releases/assets/2998763", .send_headers = c("Accept" = "application/octet-stream"))
wget might work, however
I've found a gist that shows how to do this with wget. The key component seems to be:
wget -q --auth-no-challenge --header='Accept:application/octet-stream' \
https://$TOKEN:#api.github.com/repos/$REPO/releases/assets/$asset_id \
-O $2
However if I try to replicate that in httr::GET I am not successful:
auth_url <- sprintf("https://%s:#api.github.com/repos/aammd/miniature-meme/releases/assets/2859674", Sys.getenv("GITHUB_TOKEN"))
httr::GET(auth_url,
httr::write_disk("test.rds", overwrite = TRUE),
httr::progress("down"),
httr::add_headers(Accept = "application/octet-stream"))
Calling wget from R DOES work, but this solution is not totally satisfying because I can't guarantee that all my users have wget installed (unless there is a way to do that?).
system(sprintf("wget --auth-no-challenge --header='Accept:application/octet-stream' %s -O testwget.rds", auth_url))
output of wget (note the absence of -q above) included here (again, tokens and signatures redacted, hopefully):
--2017-01-18 13:21:55-- https://ttttt:*password*#api.github.com/repos/aammd/miniature-meme/releases/assets/2859674
Resolving api.github.com... 192.30.253.117, 192.30.253.116
Connecting to api.github.com|192.30.253.117|:443... connected.
HTTP request sent, awaiting response... 302 Found
Location: https://github-cloud.s3.amazonaws.com/releases/76993567/aee5d0d6-c70a-11e6-9078-b5bee39f9fbc.RDS?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAISTNZFOVBIJMK3TQ%2F20170118%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20170118T122156Z&X-Amz-Expires=300&X-Amz-Signature=SSSSSSSS-Amz-SignedHeaders=host&actor_id=1198242&response-content-disposition=attachment%3B%20filename%3Dff.RDS&response-content-type=application%2Foctet-stream [following]
--2017-01-18 13:21:55-- https://github-cloud.s3.amazonaws.com/releases/76993567/aee5d0d6-c70a-11e6-9078-b5bee39f9fbc.RDS?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAISTNZFOVBIJMK3TQ%2F20170118%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20170118T122156Z&X-Amz-Expires=300&X-Amz-Signature=SSSSSSSSSSSS-Amz-SignedHeaders=host&actor_id=1198242&response-content-disposition=attachment%3B%20filename%3Dff.RDS&response-content-type=application%2Foctet-stream
Resolving github-cloud.s3.amazonaws.com... 52.216.226.120
Connecting to github-cloud.s3.amazonaws.com|52.216.226.120|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 682 [application/octet-stream]
Saving to: ‘testwget.rds’
0K 100% 15.5M=0s
2017-01-18 13:21:56 (15.5 MB/s) - ‘testwget.rds’ saved [682/682]
It turns out that there are two possible solutions to this problem!
solution the first: token as parameter
As suggested by #user7433058, we can indeed pass the token through as a parameter! note however that we have to use paste0. This is the approach suggested by Github themselves on their API documentation
## pass oauth in the url
httr::GET(paste0("https://api.github.com/repos/aammd/miniature-meme/releases/assets/2859674?access_token=", Sys.getenv("GITHUB_TOKEN")),
httr::write_disk("test.rds", overwrite = TRUE),
httr::progress("down"),
httr::add_headers(Accept = "application/octet-stream"))
tt <- readRDS("test.rds")
Solution the second: ask again
Another solution is to make the request the first time, then extract the URL and use it to make a second request. Since the problem is caused by sending Authorization information twice -- once in the URL, once in the header -- we can avoid the problem by only using the URL.
## alternatively, get the query url (containing signature) from the (failed) html request made the first time
firsttry <- httr::GET("https://api.github.com/repos/aammd/miniature-meme/releases/assets/2859674",
httr::add_headers(Authorization = paste("token", Sys.getenv("GITHUB_TOKEN")),
Accept = "application/octet-stream"))
httr::GET(firsttry$url, httr::write_disk("test.rds", overwrite = TRUE),
httr::write_disk("test2.rds", overwrite = TRUE),
httr::progress("down"),
httr::add_headers(Accept = "application/octet-stream"))
tt2 <- readRDS("test2.rds")
This is, I suppose, a bit less efficient (making 3 requests total instead of 2). However, since only the first request is to the actual github API, it only counts for 1 towards your rate-limiting step.
a small refinement: no redirect from httr
We can make only 2, not 3, http requests if you tell httr not to follow redirects. To do this use httr::config(followlocation = FALSE) in the first of the two requests (i.e. to get firsttry)
Try sending the auth token as a query param instead of an auth header. That way when GitHub's Oauth redirects you it'll strip the original token & the X-Amz-Algorithm param will be left to do it's job.
httr::GET(paste("https://api.github.com/repos/aammd/miniature-meme/releases/assets/2859674?access_token=", Sys.getenv("GITHUB_TOKEN")),
httr::write_disk("test.rds", overwrite = TRUE),
httr::progress("down"))
I am making an api call in response to which I get:
Response [https://reports.appsflyer.com/reports/v5/installs/id1059669903_installs_2016-07-11_2016-07-12_KVCZP]
Date: 2016-07-12 06:45
Status: 200
Content-Type: application~/vnd.ms-excel~; charset=UTF-8
Size: 64.7 kB
<BINARY BODY>
How do I read this in R?
This is how I made the api call:
installReports <- GET(url = installReportsUrl, path = installReportsPath)
print(installReports)
Please try below:
result<-content(installReports,type='text/csv',col_names=T,col_types=NULL)
This will represent the .csv report as a data frame.
Hope this helps!