Update:
Here is code that shows how to get an access token. I also use the test api here which is free (no credit card required).
The first api call to test.api.amadeus.com/v2/shopping/flight-offers is shown.
It is the second api call to test.api.amadeus.com/v1/shopping/flight-offers/pricing api that I don't know how to format.
My question remains, what is the correct way to call the second API using R?
R Script
library("tidyverse")
library("httr")
library("rjson")
amadeus_api_key_prod <- Sys.getenv("AMADEUS_API_KEY")
amadeus_api_secret_prod <- Sys.getenv("AMADEUS_SECRET")
# Initialize variables
tmp_origin <- NULL
tmp_dest <- NULL
tmp_avg_total_fare <- NULL
# Get Token
response <- POST("https://test.api.amadeus.com/v1/security/oauth2/token",
add_headers("Content-Type" = "application/x-www-form-urlencoded"),
body = list(
"grant_type" = "client_credentials",
"client_id" = amadeus_api_key_prod,
"client_secret" = amadeus_api_secret_prod),
encode = "form")
response
rsp_content <- content(response, as = "parsed", type = "application/json")
access_token <- paste0("Bearer ", rsp_content$access_token)
origin <- "JFK"
dest <- "LHR"
dep_date <- "2021-12-01"
return_date <- "2021-12-18"
max_num_flights <- 1
url <- paste0("https://test.api.amadeus.com/v2/shopping/flight-offers?originLocationCode=",
origin,
"&destinationLocationCode=",
dest,
"&departureDate=",
dep_date,
"&returnDate=",
return_date,
"&max=",
max_num_flights,
"&adults=1&nonStop=false&travelClass=ECONOMY&max=1¤cyCode=CAD")
# Get flight info
response <- GET(url,
add_headers("Content-Type" = "application/x-www-form-urlencoded",
"Authorization" = access_token),
encode = "form")
response
rsp_content <- content(response, as = "parsed", type = "application/json")
rsp_content
# Get current, more detailed flight info
# This is the part I do not know how to do
url2 <- "https://test.api.amadeus.com/v1/shopping/flight-offers/pricing"
flt_info <- toJSON(rsp_content[["data"]])
response2 <- GET(url2,
add_headers("Authorization" = access_token),
body = list(
"priceFlightOffersBody" = flt_info
),
encode = "form")
response2
rsp_content2 <- content(response2, as = "parsed", type = "application/json")
rsp_content2
Original Question
I am using the Amadeus flight info api to retrieve prices for flights.
My understanding is that to get complete price info requires two steps.
Call to https://api.amadeus.com/v2/shopping/flight-offers api
Call to https://api.amadeus.com/v1/shopping/flight-offers/pricing api
I can successfully perform the first api call with origin, destination, dates, etc.
The second call, confirms the pricing is still available and has a more detailed breakdown of the fare than does the first api call. It is this detailed breakdown that I am most interested in.
I am having trouble understanding what info that gets returned from the first api call needs to be passed to the second api call and in what format.
Below, I have included the data structure that gets returned from the first api call and my failed attempt to call the second api.
What is the correct way to call the second API using R?
Links to what I believe is the relevant documentation:
https://developers.amadeus.com/self-service/category/air/api-doc/flight-offers-price/api-reference
https://github.com/amadeus4dev/amadeus-code-examples/blob/master/flight_offers_price/v1/post/curl/flight_offers_price.sh
# Data structure returned from call to https://api.amadeus.com/v2/shopping/flight-offers
# For YYZ to YOW return, Dec 1-18 economy
rsp_content <- list(meta = list(count = 1L, links = list(self = "https://api.amadeus.com/v2/shopping/flight-offers?originLocationCode=YYZ&destinationLocationCode=YOW&departureDate=2021-12-01&returnDate=2021-12-18&max=1&adults=1&nonStop=false&travelClass=ECONOMY&max=1¤cyCode=CAD")),
data = list(list(type = "flight-offer", id = "1", source = "GDS",
instantTicketingRequired = FALSE, nonHomogeneous = FALSE,
oneWay = FALSE, lastTicketingDate = "2021-08-07", numberOfBookableSeats = 7L,
itineraries = list(list(duration = "PT1H10M", segments = list(
list(departure = list(iataCode = "YYZ", terminal = "3",
at = "2021-12-01T11:00:00"), arrival = list(iataCode = "YOW",
at = "2021-12-01T12:10:00"), carrierCode = "WS",
number = "3462", aircraft = list(code = "DH4"),
duration = "PT1H10M", id = "1", numberOfStops = 0L,
blacklistedInEU = FALSE))), list(duration = "PT1H21M",
segments = list(list(departure = list(iataCode = "YOW",
at = "2021-12-18T10:45:00"), arrival = list(iataCode = "YYZ",
terminal = "3", at = "2021-12-18T12:06:00"),
carrierCode = "WS", number = "3463", aircraft = list(
code = "DH4"), duration = "PT1H21M", id = "2",
numberOfStops = 0L, blacklistedInEU = FALSE)))),
price = list(currency = "CAD", total = "232.78", base = "125.00",
fees = list(list(amount = "0.00", type = "SUPPLIER"),
list(amount = "0.00", type = "TICKETING")), grandTotal = "232.78"),
pricingOptions = list(fareType = list("PUBLISHED"), includedCheckedBagsOnly = FALSE),
validatingAirlineCodes = list("WS"), travelerPricings = list(
list(travelerId = "1", fareOption = "STANDARD", travelerType = "ADULT",
price = list(currency = "CAD", total = "232.78",
base = "125.00"), fareDetailsBySegment = list(
list(segmentId = "1", cabin = "ECONOMY", fareBasis = "LAVD0TBJ",
brandedFare = "BASIC", class = "E", includedCheckedBags = list(
quantity = 0L)), list(segmentId = "2",
cabin = "ECONOMY", fareBasis = "LAVD0ZBI",
brandedFare = "BASIC", class = "E", includedCheckedBags = list(
quantity = 0L))))))), dictionaries = list(
locations = list(YOW = list(cityCode = "YOW", countryCode = "CA"),
YYZ = list(cityCode = "YTO", countryCode = "CA")),
aircraft = list(DH4 = "DE HAVILLAND DHC-8 400 SERIES"),
currencies = list(CAD = "CANADIAN DOLLAR"), carriers = list(
WS = "WESTJET")))
# Get full pricing info
url2 <- "https://api.amadeus.com/v1/shopping/flight-offers/pricing"
# Get pricing info
response2 <- GET(url2,
add_headers("Authorization" = access_token),
body = list(
"priceFlightOffersBody" = rsp_content[["data"]][[1]]
),
encode = "form")
response2
rsp_content2 <- content(response2, as = "parsed", type = "application/json")
rsp_content2
You can take a look at this blog article, it explains how the data needs to be passed between the 3 endpoints and it has a video showing it on Postman.
You can check some of the code samples the Amadeus for Developers team has built. Here for Flight Offers Price (in different programming languages but not R) and here for Flight Create Orders (that includes the previous steps of search and price).
They have a couple of demo applications as well that show you how to combine these endpoints to build a flight booking engine, one of them in Python that you can find here.
It seems to work with the following, for the second request.
Changes :
Use POST, not GET
While the documentation names the body priceFlightOffersBody, it should actually be just the data part, no need to encapsulate in another list.
The data part is not exactly the same as the result of the first request.
Pass the data part as an R list and use POST argument encode = "json".
url2 <- "https://test.api.amadeus.com/v1/shopping/flight-offers/pricing"
flt_info <- list(data = list(type = "flight-offers-pricing",
flightOffers = rsp_content$data))
response2 <- POST(url2,
add_headers(Authorization = access_token),
body = flt_info,
encode = "json")
rsp_content2 <- content(response2, as = "parsed", type = "application/json")
rsp_content2
Related
Is there smooth solution to query data from Google Admob API into R environment? (for e.g. similar to googleAnalyticsR package).
So if anyone ever is looking for an answer the solution that i worked out is to use library(httr) and library(jsonlite) which are general packages for managing API's
First set up your google app for admob and generate oauth credentials,
store them in as described below and authorize token.
Obtain Token:
options(admob.client_id = client.id)
options(admob.client_secret = key.secret)
# GETTING OAUTH TOKEN
token = oauth2.0_token(endpoint = oauth_endpoints("google"), # 'google is standard
app = oauth_app(appname = "google",
key = getOption('admob.client_id'),
secret = getOption("admob.client_secret")
),
scope = r"{https://www.googleapis.com/auth/admob.report}",
use_oob = TRUE,
cache = TRUE)
You can generate your body request in within avaible google documentation in here:
https://developers.google.com/admob/api/v1/reference/rest/v1/accounts.networkReport/generate
I'am passing mine as.list:
`json body` = toJSON(list(`reportSpec` = list(
`dateRange` = list(
`startDate` = list(
`year` = 2021,
`month` = 10,
`day` = 20),
`endDate` = list(
`year` = 2021,
`month` = 10,
`day` = 22
)),
`dimensions` = list("DATE"),
`metrics` = list("IMPRESSIONS")
)), auto_unbox = TRUE)
Query your request:
test = POST(url = 'https://admob.googleapis.com/v1/accounts/YOURPUBIDGOESINHERE/networkReport:generate',
add_headers(`authorization` = paste("Bearer",
token$credentials$access_token)),
add_headers(`content-type` = "application/json"),
body = `json body`
)
Finalize with some data cleansing and you are done.
jsonText = content(test, as = 'text')
df = fromJSON(jsonText, flatten = T)
df = na.omit(df[c('row.metricValues.IMPRESSIONS.integerValue',
'row.dimensionValues.DATE.value')]) # select only needed columns
row.names(df) = NULL # reindex rows
df
I've tried to get the translate service to work using the AzureCognitive R package, but I get a HTTP 400 error and I can't seem to fix it.
# Initial setup
# I've already did the create login
# az <- AzureRMR::create_azure_login(tenant = "<tenant>")
az <- AzureRMR::get_azure_login()
sub <- az$get_subscription(id = "<id>")
rg <- sub$get_resource_group("<resource-group>")
# retrieve it
cogsvc <- rg$get_cognitive_service(name = "<name>")
# getting the endpoint from the resource object
endp <- cogsvc$get_endpoint()
############################
# This is where it fails
############################
call_cognitive_endpoint(endpoint = endp,
operation = "translate",
options = list('api-version' = "3.0",
'from' = 'en',
'to' = 'dk'),
body = "[{'Text':'Hello, what is your name?'}]",
http_verb = "POST")
call_cognitive_endpoint(endpoint = endp,
operation = "translate",
options = list('api-version' = "3.0",
'from' = 'en',
'to' = 'dk'),
body = list(text = "Hello, what is your name"),
http_verb = "POST")
Can someone see if this is a bug or me doing something wrong?
As per this example, you can pass the body as body=list(list(text = "Hello, what is your name"")),
call_cognitive_endpoint(endpoint = endp,
operation = "translate",
options = list('api-version' = "3.0",
'from' = 'en',
'to' = 'dk'),
body = list(list(text = "Hello, what is your name")),
http_verb = "POST")
I'm pulling data from a public page through the rfacebook package. My code is below:
fb_oauth <- fbOAuth(app_id="###", app_secret="###", extended_permissions = FALSE,
legacy_permissions = FALSE) #id/secret is hidden
telekom <- getPage(page="TelekomMK", token=fb_oauth)
t_head <- head(telekom, n = 1)
This is the data I'm getting:
dput(t_head)
structure(list(from_id = "86689960994", from_name = "Telekom MK",
message = "<U+0421><U+043C><U+0430><U+0440><U+0442><U+0444><U+043E><U+043D> <U+0437><U+0430> <U+0441><U+0430><U+043C><U+043E> 1 <U+0434><U+0435><U+043D><U+0430><U+0440>! <U+041E><U+0434><U+0431><U+0435><U+0440><U+0435><U+0442><U+0435> <U+0433><U+043E> Huawei P Smart <U+0432><U+043E> Magenta 1L <U+0438> <U+0434><U+043E><U+0431><U+0438><U+0458><U+0442><U+0435> <U+0434><U+0432><U+043E><U+0458><U+043D><U+043E> <U+043F><U+043E><U+0432><U+0435><U+045C><U+0435> <U+043C><U+043E><U+0431><U+0438><U+043B><U+0435><U+043D> <U+0438><U+043D><U+0442><U+0435><U+0440><U+043D><U+0435><U+0442>. <ed><U+00A0><U+00BD><ed><U+00B3><U+00B2>",
created_time = "2018-03-09T12:00:00+0000", type = "photo",
link = "https://www.facebook.com/TelekomMK/photos/a.90789160994.98029.86689960994/10156117256280995/?type=3",
id = "86689960994_10156117256945995", story = NA_character_,
likes_count = 6, comments_count = 0, shares_count = 0), .Names = c("from_id",
"from_name", "message", "created_time", "type", "link", "id",
"story", "likes_count", "comments_count", "shares_count"), row.names = 1L, class = "data.frame")
What I don't understand is.. why is the text thats written on cyrlic returned as these unreadable characters? Is there a way I can fix this?
Many thanks
I just extracted the posts using
TelekomMK_posts <- getPage("TelekomMK", token = fboauth,
n=10000, since = '2009/01/01',
until = '2018/03/15')
It came out fine
Are you only interested in the particular post from March 9th?
I use FB to extract Cyrillic a lot - pm me if you have questions
Try encoding to UTF-8
I am attempting to scrape a Dynamic website Morningstar.com via XHR requests.
The exact site I am scraping is: http://performance.morningstar.com/funds/etf/total-returns.action?t=SPY®ion=USA&culture=en_US
What I am trying to scrape is the Quarterly performance number (1-month). The result should be 0.64 as of today.
try(res <- GET(url = "http://performance.morningstar.com/fund/performance-return.action",
query = list(
t="SPY",
region="usa",
culture="en-US"
)
))
tryCatch(x <- content(res) %>%
html_nodes(xpath = '//*[#id="tab-quar-end-content"]/table/tbody/tr[1]/td[1]') %>%
html_text() %>%
trimws() %>%
as.numeric()
, error = function(e) x <-NA)
However, the result is numeric(0)
Any idea what I am doing wrong?
Sody
Update:
I was able to get the html data with the following code:
try(res <- GET(url = "http://performance.morningstar.com/fund/performance-return.action",
query = list(
t = "SPY",
region = "usa",
culture = "en-US",
ops = "clear",
s = "0P0000J533",
ndec = "2",
ep = "true",
align = "q",
annlz = "true",
comparisonRemove = "false"
)
))
But I am still having problems pointing to the data using either the CSS selector or the xpath with rvest.
What do you guys use to find those data points? is SelectorGadget still the go to?
Cheers, Aaron
library(httr)
GET(
url = "http://performance.morningstar.com/perform/Performance/cef/trailing-total-returns.action",
add_headers(
Referer = "http://performance.morningstar.com/funds/etf/total-returns.action?t=SPY®ion=USA&culture=en_US",
`X-Requested-With` = "XMLHttpRequest"
),
query = list(
t = "ARCX:SPY", region = "usa", culture = "en-US",
cur = "", ops = "clear", s = "0P00001MK8", ndec = "2", ep = "true",
align = "q", annlz = "true", comparisonRemove = "false",
benchmarkSecId = "", benchmarktype = ""
),
verbose()
) -> res
You have to target the XHR directly.
The table is embedded using java script, not hard-coded. You won't be able to scrape this data.
Working in R, but that shouldn't really matter.
I want to gather all tweets after : https://twitter.com/ChrisChristie/status/663046613779156996
So Tweet ID : 663046613779156996
base = "https://ontributor_details = "contributor_details=true"
## include_rts
include_rts = "include_rts=true"
## exclude_replies
exclude_replies = "exclude_replies=false"api.twitter.com/1.1/statuses/user_timeline.json?"
queryName = "chrischristie"
query = paste("q=", queryName, sep="")
secondary_url = paste(query, count, contributor_details,include_rts,exclude_replies, sep="&")
final_url = paste(base, secondary_url, sep="")
timeline = GET(final_url, sig)
This (the above) works. There is no since_id. The URL comes out to be
"https://api.twitter.com/1.1/statuses/user_timeline.json?q=chrischristie&count=200&contributor_details=true&include_rts=true&exclude_replies=false"
The below does not, just by adding in the following
cur_since_id_url = "since_id=663046613779156996"
secondary_url = paste(query, count,
contributor_details,include_rts,exclude_replies,cur_since_id_url, sep="&")
final_url = paste(base, secondary_url, sep="")
timeline = GET(final_url, sig)
The url for the above there is
"https://api.twitter.com/1.1/statuses/user_timeline.json?q=chrischristie&count=200&contributor_details=true&include_rts=true&exclude_replies=false&since_id=663046613779156992"
This seems to work:
require(httr)
myapp <- oauth_app(
"twitter",
key = "......",
secret = ".......")
twitter_token <- oauth1.0_token(oauth_endpoints("twitter"), myapp)
req <- GET("https://api.twitter.com/1.1/statuses/user_timeline.json",
query = list(
screen_name="chrischristie",
count=10,
contributor_details=TRUE,
include_rts=TRUE,
exclude_replies=FALSE,
since_id=663046613779156992),
config(token = twitter_token))
content(req)
Have a look at GET statuses/user_timeline