I'm pulling data from a public page through the rfacebook package. My code is below:
fb_oauth <- fbOAuth(app_id="###", app_secret="###", extended_permissions = FALSE,
legacy_permissions = FALSE) #id/secret is hidden
telekom <- getPage(page="TelekomMK", token=fb_oauth)
t_head <- head(telekom, n = 1)
This is the data I'm getting:
dput(t_head)
structure(list(from_id = "86689960994", from_name = "Telekom MK",
message = "<U+0421><U+043C><U+0430><U+0440><U+0442><U+0444><U+043E><U+043D> <U+0437><U+0430> <U+0441><U+0430><U+043C><U+043E> 1 <U+0434><U+0435><U+043D><U+0430><U+0440>! <U+041E><U+0434><U+0431><U+0435><U+0440><U+0435><U+0442><U+0435> <U+0433><U+043E> Huawei P Smart <U+0432><U+043E> Magenta 1L <U+0438> <U+0434><U+043E><U+0431><U+0438><U+0458><U+0442><U+0435> <U+0434><U+0432><U+043E><U+0458><U+043D><U+043E> <U+043F><U+043E><U+0432><U+0435><U+045C><U+0435> <U+043C><U+043E><U+0431><U+0438><U+043B><U+0435><U+043D> <U+0438><U+043D><U+0442><U+0435><U+0440><U+043D><U+0435><U+0442>. <ed><U+00A0><U+00BD><ed><U+00B3><U+00B2>",
created_time = "2018-03-09T12:00:00+0000", type = "photo",
link = "https://www.facebook.com/TelekomMK/photos/a.90789160994.98029.86689960994/10156117256280995/?type=3",
id = "86689960994_10156117256945995", story = NA_character_,
likes_count = 6, comments_count = 0, shares_count = 0), .Names = c("from_id",
"from_name", "message", "created_time", "type", "link", "id",
"story", "likes_count", "comments_count", "shares_count"), row.names = 1L, class = "data.frame")
What I don't understand is.. why is the text thats written on cyrlic returned as these unreadable characters? Is there a way I can fix this?
Many thanks
I just extracted the posts using
TelekomMK_posts <- getPage("TelekomMK", token = fboauth,
n=10000, since = '2009/01/01',
until = '2018/03/15')
It came out fine
Are you only interested in the particular post from March 9th?
I use FB to extract Cyrillic a lot - pm me if you have questions
Try encoding to UTF-8
Related
Is there smooth solution to query data from Google Admob API into R environment? (for e.g. similar to googleAnalyticsR package).
So if anyone ever is looking for an answer the solution that i worked out is to use library(httr) and library(jsonlite) which are general packages for managing API's
First set up your google app for admob and generate oauth credentials,
store them in as described below and authorize token.
Obtain Token:
options(admob.client_id = client.id)
options(admob.client_secret = key.secret)
# GETTING OAUTH TOKEN
token = oauth2.0_token(endpoint = oauth_endpoints("google"), # 'google is standard
app = oauth_app(appname = "google",
key = getOption('admob.client_id'),
secret = getOption("admob.client_secret")
),
scope = r"{https://www.googleapis.com/auth/admob.report}",
use_oob = TRUE,
cache = TRUE)
You can generate your body request in within avaible google documentation in here:
https://developers.google.com/admob/api/v1/reference/rest/v1/accounts.networkReport/generate
I'am passing mine as.list:
`json body` = toJSON(list(`reportSpec` = list(
`dateRange` = list(
`startDate` = list(
`year` = 2021,
`month` = 10,
`day` = 20),
`endDate` = list(
`year` = 2021,
`month` = 10,
`day` = 22
)),
`dimensions` = list("DATE"),
`metrics` = list("IMPRESSIONS")
)), auto_unbox = TRUE)
Query your request:
test = POST(url = 'https://admob.googleapis.com/v1/accounts/YOURPUBIDGOESINHERE/networkReport:generate',
add_headers(`authorization` = paste("Bearer",
token$credentials$access_token)),
add_headers(`content-type` = "application/json"),
body = `json body`
)
Finalize with some data cleansing and you are done.
jsonText = content(test, as = 'text')
df = fromJSON(jsonText, flatten = T)
df = na.omit(df[c('row.metricValues.IMPRESSIONS.integerValue',
'row.dimensionValues.DATE.value')]) # select only needed columns
row.names(df) = NULL # reindex rows
df
Update:
Here is code that shows how to get an access token. I also use the test api here which is free (no credit card required).
The first api call to test.api.amadeus.com/v2/shopping/flight-offers is shown.
It is the second api call to test.api.amadeus.com/v1/shopping/flight-offers/pricing api that I don't know how to format.
My question remains, what is the correct way to call the second API using R?
R Script
library("tidyverse")
library("httr")
library("rjson")
amadeus_api_key_prod <- Sys.getenv("AMADEUS_API_KEY")
amadeus_api_secret_prod <- Sys.getenv("AMADEUS_SECRET")
# Initialize variables
tmp_origin <- NULL
tmp_dest <- NULL
tmp_avg_total_fare <- NULL
# Get Token
response <- POST("https://test.api.amadeus.com/v1/security/oauth2/token",
add_headers("Content-Type" = "application/x-www-form-urlencoded"),
body = list(
"grant_type" = "client_credentials",
"client_id" = amadeus_api_key_prod,
"client_secret" = amadeus_api_secret_prod),
encode = "form")
response
rsp_content <- content(response, as = "parsed", type = "application/json")
access_token <- paste0("Bearer ", rsp_content$access_token)
origin <- "JFK"
dest <- "LHR"
dep_date <- "2021-12-01"
return_date <- "2021-12-18"
max_num_flights <- 1
url <- paste0("https://test.api.amadeus.com/v2/shopping/flight-offers?originLocationCode=",
origin,
"&destinationLocationCode=",
dest,
"&departureDate=",
dep_date,
"&returnDate=",
return_date,
"&max=",
max_num_flights,
"&adults=1&nonStop=false&travelClass=ECONOMY&max=1¤cyCode=CAD")
# Get flight info
response <- GET(url,
add_headers("Content-Type" = "application/x-www-form-urlencoded",
"Authorization" = access_token),
encode = "form")
response
rsp_content <- content(response, as = "parsed", type = "application/json")
rsp_content
# Get current, more detailed flight info
# This is the part I do not know how to do
url2 <- "https://test.api.amadeus.com/v1/shopping/flight-offers/pricing"
flt_info <- toJSON(rsp_content[["data"]])
response2 <- GET(url2,
add_headers("Authorization" = access_token),
body = list(
"priceFlightOffersBody" = flt_info
),
encode = "form")
response2
rsp_content2 <- content(response2, as = "parsed", type = "application/json")
rsp_content2
Original Question
I am using the Amadeus flight info api to retrieve prices for flights.
My understanding is that to get complete price info requires two steps.
Call to https://api.amadeus.com/v2/shopping/flight-offers api
Call to https://api.amadeus.com/v1/shopping/flight-offers/pricing api
I can successfully perform the first api call with origin, destination, dates, etc.
The second call, confirms the pricing is still available and has a more detailed breakdown of the fare than does the first api call. It is this detailed breakdown that I am most interested in.
I am having trouble understanding what info that gets returned from the first api call needs to be passed to the second api call and in what format.
Below, I have included the data structure that gets returned from the first api call and my failed attempt to call the second api.
What is the correct way to call the second API using R?
Links to what I believe is the relevant documentation:
https://developers.amadeus.com/self-service/category/air/api-doc/flight-offers-price/api-reference
https://github.com/amadeus4dev/amadeus-code-examples/blob/master/flight_offers_price/v1/post/curl/flight_offers_price.sh
# Data structure returned from call to https://api.amadeus.com/v2/shopping/flight-offers
# For YYZ to YOW return, Dec 1-18 economy
rsp_content <- list(meta = list(count = 1L, links = list(self = "https://api.amadeus.com/v2/shopping/flight-offers?originLocationCode=YYZ&destinationLocationCode=YOW&departureDate=2021-12-01&returnDate=2021-12-18&max=1&adults=1&nonStop=false&travelClass=ECONOMY&max=1¤cyCode=CAD")),
data = list(list(type = "flight-offer", id = "1", source = "GDS",
instantTicketingRequired = FALSE, nonHomogeneous = FALSE,
oneWay = FALSE, lastTicketingDate = "2021-08-07", numberOfBookableSeats = 7L,
itineraries = list(list(duration = "PT1H10M", segments = list(
list(departure = list(iataCode = "YYZ", terminal = "3",
at = "2021-12-01T11:00:00"), arrival = list(iataCode = "YOW",
at = "2021-12-01T12:10:00"), carrierCode = "WS",
number = "3462", aircraft = list(code = "DH4"),
duration = "PT1H10M", id = "1", numberOfStops = 0L,
blacklistedInEU = FALSE))), list(duration = "PT1H21M",
segments = list(list(departure = list(iataCode = "YOW",
at = "2021-12-18T10:45:00"), arrival = list(iataCode = "YYZ",
terminal = "3", at = "2021-12-18T12:06:00"),
carrierCode = "WS", number = "3463", aircraft = list(
code = "DH4"), duration = "PT1H21M", id = "2",
numberOfStops = 0L, blacklistedInEU = FALSE)))),
price = list(currency = "CAD", total = "232.78", base = "125.00",
fees = list(list(amount = "0.00", type = "SUPPLIER"),
list(amount = "0.00", type = "TICKETING")), grandTotal = "232.78"),
pricingOptions = list(fareType = list("PUBLISHED"), includedCheckedBagsOnly = FALSE),
validatingAirlineCodes = list("WS"), travelerPricings = list(
list(travelerId = "1", fareOption = "STANDARD", travelerType = "ADULT",
price = list(currency = "CAD", total = "232.78",
base = "125.00"), fareDetailsBySegment = list(
list(segmentId = "1", cabin = "ECONOMY", fareBasis = "LAVD0TBJ",
brandedFare = "BASIC", class = "E", includedCheckedBags = list(
quantity = 0L)), list(segmentId = "2",
cabin = "ECONOMY", fareBasis = "LAVD0ZBI",
brandedFare = "BASIC", class = "E", includedCheckedBags = list(
quantity = 0L))))))), dictionaries = list(
locations = list(YOW = list(cityCode = "YOW", countryCode = "CA"),
YYZ = list(cityCode = "YTO", countryCode = "CA")),
aircraft = list(DH4 = "DE HAVILLAND DHC-8 400 SERIES"),
currencies = list(CAD = "CANADIAN DOLLAR"), carriers = list(
WS = "WESTJET")))
# Get full pricing info
url2 <- "https://api.amadeus.com/v1/shopping/flight-offers/pricing"
# Get pricing info
response2 <- GET(url2,
add_headers("Authorization" = access_token),
body = list(
"priceFlightOffersBody" = rsp_content[["data"]][[1]]
),
encode = "form")
response2
rsp_content2 <- content(response2, as = "parsed", type = "application/json")
rsp_content2
You can take a look at this blog article, it explains how the data needs to be passed between the 3 endpoints and it has a video showing it on Postman.
You can check some of the code samples the Amadeus for Developers team has built. Here for Flight Offers Price (in different programming languages but not R) and here for Flight Create Orders (that includes the previous steps of search and price).
They have a couple of demo applications as well that show you how to combine these endpoints to build a flight booking engine, one of them in Python that you can find here.
It seems to work with the following, for the second request.
Changes :
Use POST, not GET
While the documentation names the body priceFlightOffersBody, it should actually be just the data part, no need to encapsulate in another list.
The data part is not exactly the same as the result of the first request.
Pass the data part as an R list and use POST argument encode = "json".
url2 <- "https://test.api.amadeus.com/v1/shopping/flight-offers/pricing"
flt_info <- list(data = list(type = "flight-offers-pricing",
flightOffers = rsp_content$data))
response2 <- POST(url2,
add_headers(Authorization = access_token),
body = flt_info,
encode = "json")
rsp_content2 <- content(response2, as = "parsed", type = "application/json")
rsp_content2
My setup: I am using two selectizeInputs togehter with hcmap() in order to display a world map (this one custom/world-palestine-highres) where users can change the institute and the variable they want to look at. Hence, the map gets rendered often!
In this process, I downloaded the map multiple times via testing/rendering the app till I found the options(highcharter.download_map_data = FALSE) while browsing through your repository - the vignette on how to use download_map_data is not that clear tbh. However, now the map gets rendered without downloading itself again --> perfect!
Now to my question: Where does the map get stored on the server when downloaded the first time? Because with my current setup, no user will download the map again (which is what I want), but what if we have to restart the server or move files? I just want to make sure that I dont have to service the app all the time in regards to map downloads.
EDIT#1: Code samples for JBKunst, as the new code doesn't show any data
My hcmap code I used before:
hcmap("custom/world-palestine-highres", data = datapool_credit, joinBy = c("hc-a2", "Country"), value = capital_variable,
name = capital_variable,
dataLabels = list(enabled = TRUE, format = '{point.name}'),
borderColor = "#FAFAFA", borderWidth = 0.1,
tooltip = list(valueDecimals = 0, valuePrefix = "€", valueSuffix = "")) %>%
hc_mapNavigation(enabled = TRUE)
The new code as suggested by JBKunst:
highchart(type = "map") %>%
hc_add_series(
mapData = JS("Highcharts.maps['custom/world-palestine-highres']"),
data = datapool_credit, joinBy = c("hc-key"), joinBy = c("hc-a2", "Country"), value = capital_variable,
name = capital_variable,
dataLabels = list(enabled = TRUE, format = '{point.name}'),
borderColor = "#FAFAFA", borderWidth = 0.1,
tooltip = list(valueDecimals = 0, valuePrefix = "€", valueSuffix = "")) %>%
hc_mapNavigation(enabled = TRUE)
EDIT#2: structure of used data for the map (changed the numbers for data privacy reasons, but not the format)
> dput(head(datapool_credit))
structure(list(Country = c("AE", "AF", "AL", "AM", "AO", "AQ"
), PD = c(0.506010018321176, 64.350505, 8.94600128868667, 14.29401046096271,
25.0439479, 3.5626)), .Names = c("Country", "PD"), class = c("data.table",
"data.frame"), row.names = c(NA, -6L), .internal.selfref = <pointer: 0x244ccb8>)
Best regards,
Martin
you can use something like https://github.com/jbkunst/shiny-apps/tree/master/highcharter-maps-performance
You don't need to store the map. Instead the user will read from https://code.highcharts.com/mapdata/custom/world-palestine-highres.js one time per session just like the highcharts example:
In your ui.R you need to add the map
# ui.R
...
tags$script(src = "https://code.highcharts.com/mapdata/custom/world-palestine-highres.js")
...
Then in the server.R don't use the hcmap, instead use highchart():
# server.R
...
highchart(type = "map") %>%
hc_add_series(
mapData = JS("Highcharts.maps["custom/world-palestine-highres"]"),
data = data, joinBy = c("hc-key")
)
I have a data frame with two variables: classification and email id. The reproducible example of the data frame is as follows:
structure(list(Classification = structure(c(1L, 1L, 1L), .Label = "bills", class = "factor"),
FromMailId = structure(c(1L, 1L, 1L), .Label = "bimbilikkibimbi#yahoo.com", class = "factor")), .Names = c("Classification",
"FromMailId"), row.names = c(NA, 3L), class = "data.frame")
I am using library(mailR) to send mails. I have to customise the mail delivery basis the value in Classification variable. I was trying to loop through the data frame to pick up the email id depending on the value of the Classification. The following is the code I am using:
for(i in df1$Classification){
if(i == "bills"){
library(mailR)
send.mail(from = "UserName#gmail.com",
to = df1$FromMailId[i],
subject = paste("Subject: ", "This is a test message"),
body = "You have got bills",
smtp = list(host.name = "smtp.gmail.com", port = 465, user.name = "UserName", passwd = "PassWord", ssl = TRUE),
authenticate = TRUE,
send = TRUE)
}
}
The above code works perfectly if I change the 'to' email id to an actual email id. But when I reference the same field to a column in the data frame(FromMailId here), I get the following error:
Error in ls(envir = envir, all.names = private) :
invalid 'envir' argument
I gathered that I am referencing a specific column in the loop and referring to another column inside send.mail...but not sure how to resolve this.
You're looping over the elements of Classification and trying to use that as a subset index, which won't work. Try this instead:
library(mailR) #No need to have that in the loop
for(i in 1:nrow(df1)){
if(df1$Classification[i] == "bills"){
send.mail(from = "UserName#gmail.com",
to = df1$FromMailId[i],
subject = paste("Subject: ", "This is a test message"),
body = "You have got bills",
smtp = list(host.name = "smtp.gmail.com", port = 465, user.name = "UserName", passwd = "PassWord", ssl = TRUE),
authenticate = TRUE,
send = TRUE)
}
}
I believe the df1$FromMailId is a factor. So try converting it in to character.
Either convert the column using df1$FromMailId <- as.character(df1$FromMailId, stringAsFactors = F) or just edit the sendmail code by to = as.character(df1$FromMailId[i]).
sac[,treatment_days := as.character(seq(from = SACDPDAT, to = SACRTDAT, by = "1 day")), by = PACKID] I have data named sac with dput(sac[1:2,]) as follows:
structure(list(SUBJECT_Blinded = c(1201001, 1101001), LINE = c(8,
4), MODULE = c("SAC", "SAC"), CENTRE_Blinded = c(1201, 1201),
STUDYPER = c(7, 4), PACKID = c(10096, 10595), SACDPDAT = structure(c(1335304800,
1325545200), class = c("POSIXct", "POSIXt"), tzone = ""),
SACDP1 = c(35, 35), C_SACDP = c(NA_character_, NA_character_
), SACRTDAT = structure(c(1340316000, 1327964400), class = c("POSIXct",
"POSIXt"), tzone = ""), SACRT1 = c(0, 9), C_SACRT = c(NA_character_,
NA_character_)), .Names = c("SUBJECT_Blinded", "LINE", "MODULE",
"CENTRE_Blinded", "STUDYPER", "PACKID", "SACDPDAT", "SACDP1",
"C_SACDP", "SACRTDAT", "SACRT1", "C_SACRT"), sorted = c("SUBJECT_Blinded",
"PACKID"), class = c("data.table", "data.frame"), row.names = c(NA,
-2L))
When I running the code:
sac[,treatment_days := list(format(seq(from = SACDPDAT, to = SACRTDAT, by = "1 day"),"%Y-%m-%d")), by = PACKID]
RStudio crushes and returns info:
Problem signature:
Problem Event Name: APPCRASH
Application Name: rsession.exe
Application Version: 0.98.501.0
Application Timestamp: 52e8371d
Fault Module Name: R.dll
Fault Module Version: 3.3.65126.0
Fault Module Timestamp: 53185fd3
Exception Code: c0000005
Exception Offset: 0000000000028c36
OS Version: 6.1.7601.2.1.0.256.48
Locale ID: 1045
Additional Information 1: 4fc0
Additional Information 2: 4fc0e6e5b53a870c89fb6e37a38d7e6b
Additional Information 3: 9d6e
Additional Information 4: 9d6e8f79167930945e5a5d06afac680e
It's the same with pure R. Any ideas how to do it another way?
There's a couple of problems with your new code:
"1 day" is incorrect, if you run seq on a date object, the number you pass to by will be interpreted as days, so:
seq(from = SACDPDAT, to = SACRTDAT, by = 1)
You also cannot create a new column from this sequence, because there can only be one value for each row. Instead, you can generate the sequence of days by PACKID, and then join this onto the old data.table
So try:
setkey(sac, PACKID)
sac <- sac[sac[,seq(from = SACDPDAT, to = SACRTDAT, by = 1), by=PACKID]]