I am interested in the data provided by the link: https://stats.bis.org/api/v1/data/WS_XRU_D/D.RU../all?detail=full
My code for retrieving the data on the daily exchange rate history so far has developed into just this basic lines, while it seems I am stuck with realization that I am not able to extract the core daily data I am interested in:
u <- "https://stats.bis.org/api/v1/data/WS_XRU_D/D.RU../all?detail=full"
d <- xml2::read_xml(u)
d
{xml_document}
<StructureSpecificData
xmlns:ss="http://www.sdmx.org/resources/sdmxml/schemas/v2_1/data/structurespecific
...
[1] <message:Header>\n <message:ID>IDREF85e8b4cf-d7d2-4506-81e6-adf668cd841b</message:ID>\n <message:Test>fa ...
[2] <message:DataSet ss:dataScope="DataStructure" xsi:type="ns1:DataSetType" ss:structureRef="BIS_WS_XRU_D_1_0 ...
I will appreciate very much for any suggestion on how to proceed correctly with xml data retreiving!
You started on the correct track, it is just a matter of extracting the correct nodes and obtaining the attribute values:
library(xml2)
library(dplyr)
#read the page
url <- "https://stats.bis.org/api/v1/data/WS_XRU_D/D.RU../all?detail=full"
page <- xml2::read_xml(url)
#extract the OBS nodes
#<Obs TIME_PERIOD="1992-07-01" OBS_VALUE="0.12526" OBS_STATUS="A" OBS_CONF="F"/>
observations <- xml_find_all(page, ".//Obs")
#extract the attribute vales from each node
date <- observations %>% xml_attr("TIME_PERIOD") %>% as.Date("%Y-%m-%d")
value <- observations %>% xml_attr("OBS_VALUE") %>% as.numeric()
status <- observations %>% xml_attr("OBS_STATUS")
conf <- observations %>% xml_attr("OBS_CONF")
answer <- data.frame(date, value, status, conf)
plot(answer$date, answer$value)
Related
I am an R beginner. I am using the {TopGO} R Package for performing enrichment analysis for gene ontology.
In the last part of the script (#create topGO object), I get this error:
Error in .local(.Object, ...) : allGenes must be a factor with 2 levels
Could you please help me with this? I really appreciate your help. Thanks
Below I have reported the R script I have used. I would like to upload also my starting dataset but I don't know how to to it.
library(tidyverse)
library(data.table)
# if (!require("BiocManager", quietly = TRUE))
# install.packages("BiocManager")
# BiocManager::install("topGO")
library(topGO)
# make table from meta data holding grouping information for each sample
# in this case landrace, cultivar ...
getwd()
setwd("C:/Users/iaia-/Desktop/")
background <- fread('anno.variable.out.txt') %>%
as_tibble() %>%
#the ID is the actual GO term
mutate(id = as.numeric(id)) %>%
na.omit() %>%
#according to pannzer2 ARGOT is the best scoring algorithm
filter(str_detect(type ,'ARGOT'),
#choose a PPV value
#according to the pannzer2 manual, there is no 'best' option
# Philipp advised to use 0.4, 0.6, 0.8
# maybe I write a loop to check differences later
PPV >=0.6 ) %>%
#create a ontology column to have the 3 ontology options, which topGO supports
# 'BP' - biological process, 'MF' - molecular function, 'CC' - cellular component
mutate(ontology=sapply(strsplit(type,'_'),'[',1)) %>%
#select the 3 columns we need
dplyr::select(id,ontology, qpid) %>%
#add GO: to the GO ids
dplyr::mutate(id=paste0('GO:',id))
foreground <- fread('LIST.txt',header=F)
colnames(foreground) <- c("rowname")
#rename type to group to prevent confusion
#dplyr::rename(group=type)
#for all groups together
all_results <- tibble()
for (o in unique(background$ontology)){
#filter background for a certain ontology
ont_background <- filter(background, ontology==o)} #BIOLOGICAL PROCESS
annAT <- split(ont_background$qpid,ont_background$id)
#filter foreground for a group
fg_genes <- foreground %>% pull(rowname)
ont_background <- ont_background %>%
mutate(present=as.factor(ifelse(qpid %in% fg_genes,1,0))) %>%
dplyr::select(-id) %>% distinct() %>%
pull(present, name = qpid)
#create topGO object
GOdata <-new("topGOdata", ontology = o, allGenes = ont_background, nodeSize = 5,annot=annFUN.GO2genes,GO2genes=annAT)
weight01.fisher <- runTest(GOdata, statistic = "fisher")
results <- GenTable(GOdata, classicFisher=weight01.fisher,topNodes=ifelse(length(GOdata#graph#nodes) < 30,length(GOdata#graph#nodes),30)) %>%
dplyr::rename(pvalue=6) %>%
mutate(ontology=o,
pvalue=as.numeric(pvalue))
all_results <- bind_rows(all_results,results)
}
#all_results %>% pull(GO.ID) %>% writeClipboard()
resultspvalue<- all_results %>% dplyr::select(GO.ID,pvalue)
write_tsv(resultspvalue, "GOTermsrep_pvalue.txt")
It's tough to say for sure without knowing what ont_background looked like before you modified it, but based on these threads I think that there's a problem with the final ont_background object you are using to make the topGO object in your final "#create topGO object" chunk.
It looks like the allGenes argument (ont_background) needs to be a vector of pvalues with names() assigned as the gene names in whatever format you are using for other arguments.
I'm scraping the ASN database (http://aviation-safety.net/database/). I've written code to paginate through each of the years (1919-2019) and scrape all relevant nodes except fatalities (represented as "fat."). Selector Gadget tells me the fatalities node is called "'#contentcolumnfull :nth-child(5)'". For some reason ".list:nth-child(5)" doesn't work.
When I scrape #contentcolumnfull :nth-child(5), the first element is blank, represented as "".
How can I write a function to delete the first empty element for every year/page that's scraped? It's simple to delete the first element when I scrape a single page on its own:
fat <- html_nodes(webpage, '#contentcolumnfull :nth-child(5)')
fat <- html_text(fat)
fat <- fat[-1]
but I'm finding it difficult to write into a function.
I also have a second question regarding date-time and formatting. My days data are represented as day-month-year. Several element days and months are missing (ex: ??-??-1985, JAN-??-2004). Ideally, I'd like to transform the dates into a lubridate object, but I can't with missing data or if I only keep the years.
At this point, I've used gsub() and regex to clean the data (delete "??" and floating dashes), so I have a mixed bag of data formats. However, this makes it difficult to visualize the data. Thoughts on best practice?
# Load libraries
library(tidyverse)
library(rvest)
library(xml2)
library(httr)
years <- seq(1919, 2019, by=1)
pages <- c("http://aviation-safety.net/database/dblist.php?Year=") %>%
paste0(years)
# Leaving out the category, location, operator, etc. nodes for sake of brevity
read_date <- function(url){
az <- read_html(url)
date <- az %>%
html_nodes(".list:nth-child(1)") %>%
html_text() %>%
as_tibble()
}
read_type <- function(url){
az <- read_html(url)
type <- az %>%
html_nodes(".list:nth-child(2)") %>%
html_text() %>%
as_tibble()
}
date <- bind_rows(lapply(pages, read_date))
type <- bind_rows(lapply(pages, read_type))
# Writing to dataframe
aviation_df <- cbind(type, date)
aviation_df <- data.frame(aviation_df)
# Excluding data cleaning
It is bad practice to ping the same page more than once in order to extract the requested information. You should read the page, extract all of the desired information and then move to the next page.
In this case the individual nodes are all store in one master table. rvest's html_table() function is handy to convert a html table into a data frame.
library(rvest)
library(dplyr)
years <- seq(2010, 2015, by=1)
pages <- c("http://aviation-safety.net/database/dblist.php?Year=") %>%
paste0(years)
# Leaving out the category, location, operator, etc. nodes for sake of brevity
read_table <- function(url){
#add delay so that one is not attacking the host server (be polite)
Sys.sleep(0.5)
#read page
page <- read_html(url)
#extract the table out (the data frame is stored in the first element of the list)
answer<-(page %>% html_nodes("table") %>% html_table())[[1]]
#convert the falatities column to character to make a standardize column type
answer$fat. <-as.character(answer$fat.)
answer
}
# Writing to dataframe
aviation_df <- bind_rows(lapply(pages, read_table))
The are a few extra columns which will need clean-up
This question already has an answer here:
Webscraping with Yahoo Finance
(1 answer)
Closed last year.
Unfortunately, I am not an experienced scraper yet. However, I need to scrape key statistics of multiple stocks from Yahoo Finance with R.
I am somewhat familiar with scraping data directly from html using read_html, html_nodes(), and html_text() from the rvest package. However, this web page MSFT key stats is a bit complicated, I am not sure if all the stats are kept in XHR, JS, or Doc. I am guessing the data is stored in JSON.
If anyone knows a good way to extract and parse data for this web page with R, kindly answer my question, great thanks in advance!
Or if there is a more convenient way to extract these metrics via quantmod or Quandl, kindly let me know, that would be a extremely good solution!
The goal is to have tickets/symbols as rownames/rowlabels whereas the statistics are identified as columns. A illustration of my needs can be found at this Finviz link:
https://finviz.com/screener.ashx
The reason I would like to scrape Yahoo Finance data is because Yahoo also considers Enterprise, EBITDA key stats..
EDIT:
I meant to refer to the key statistics page.. For example.. : https://finance.yahoo.com/quote/MSFT/key-statistics/ . The code should lead to one data frame rows of stock symbols and columns of key stats.
Code
library(rvest)
library(tidyverse)
# Define stock name
stock <- "MSFT"
# Extract and transform data
df <- paste0("https://finance.yahoo.com/quote/", stock, "/financials?p=", stock) %>%
read_html() %>%
html_table() %>%
map_df(bind_cols) %>%
# Transpose
t() %>%
as_tibble()
# Set first row as column names
colnames(df) <- df[1,]
# Remove first row
df <- df[-1,]
# Add stock name column
df$Stock_Name <- stock
Result
Revenue `Total Revenue` `Cost of Revenu… `Gross Profit`
<chr> <chr> <chr> <chr>
1 6/30/2… 110,360,000 38,353,000 72,007,000
2 6/30/2… 96,571,000 33,850,000 62,721,000
3 6/30/2… 91,154,000 32,780,000 58,374,000
4 6/30/2… 93,580,000 33,038,000 60,542,000
# ... with 25 more variables: ...
edit:
Or, for convenience, as a function:
get_yahoo <- function(stock){
# Extract and transform data
x <- paste0("https://finance.yahoo.com/quote/", stock, "/financials?p=", stock) %>%
read_html() %>%
html_table() %>%
map_df(bind_cols) %>%
# Transpose
t() %>%
as_tibble()
# Set first row as column names
colnames(x) <- x[1,]
# Remove first row
x <- x[-1,]
# Add stock name column
x$Stock_Name <- stock
return(x)
}
Usage: get_yahoo(stock)
I hope that this is what are you looking for:
library(quantmod)
library(plyr)
what_metrics <- yahooQF(c("Price/Sales",
"P/E Ratio",
"Price/EPS Estimate Next Year",
"PEG Ratio",
"Dividend Yield",
"Market Capitalization"))
Symbols<-c("XOM","MSFT","JNJ","GE","CVX","WFC","PG","JPM","VZ","PFE","T","IBM","MRK","BAC","DIS","ORCL","PM","INTC","SLB")
metrics <- getQuote(paste(Symbols, sep="", collapse=";"), what=what_metrics)
to get the list of metrics
yahooQF()
you can use lapply to get more than one pirce
library(quantmod)
Symbols<-c("XOM","MSFT","JNJ","GE","CVX","WFC","PG","JPM","VZ","PFE","T","IBM","MRK","BAC","DIS","ORCL","PM","INTC","SLB")
StartDate <- as.Date('2015-01-01')
Stocks <- lapply(Symbols, function(sym) {
Cl(na.omit(getSymbols(sym, from=StartDate, auto.assign=FALSE)))
})
Stocks <- do.call(merge, Stocks)
in this case i get the closing price look in function Cl()
I have a table from a website I am trying to download, and it appears to be made of a bunch of tables. Right now I am using rvest to bring the table in as text, but it is bringing in a bunch of other tables I am not interested in and then I am coercing the data into a better format, but it's not a repeatable process. Here is my code:
library(rvest)
library(tidyr)
#Auto Download Data
#reads the url of the race
race_url <- read_html("http://racing-reference.info/race/2016_Folds_of_Honor_QuikTrip_500/W")
#reads in the tables, in this code there are too many
race_results <- race_url %>%
html_nodes(".col") %>%
html_text()
race_results <- data.table(race_results) #turns from a factor to a DT
f <- nrow(race_results) #counts the number of rows in the data
#eliminates all rows after 496 (11*45 + 1) since there are never more than 43 racers
race_results <- race_results[-c(496:f)]
#puts the data into a format with 1:11 numbering for unstacking
testDT <- data.frame(X = race_results$race_results, ind = rep(1:11, nrow(race_results)/11))
testDT <- unstack(testDT, X~ind) #unstacking data into 11 columns
colnames(testDT) <- testDT[1, ] #changing the top column into the header
I commented everything so you would know what I am trying to do. If you go to the URL, there is a top table with driver results, which is what I am trying to scrape, but it is pulling the bottom ones too, as I can't seem to get a different html_nodes to work other than with ".col". I also tried html_table() in place of the html_text() but it didn't work. I suppose this can be done either by identifying the table in the css (I can't figure this out) or by using a different type of call or the XML library (which I also can't figure out). Any help or direction is appreciated.
UPDATE:
From the comments below, the correct code to pull this data is as follows:
library(rvest)
library(tidyr)
#Auto Download Data
race_url <- read_html("http://racing-reference.info/race/2016_Folds_of_Honor_QuikTrip_500/W") #reads the url of the race
race_results <- race_url %>% html_nodes("table") #returns a DF with all of the tables on the page
race_results <- race_results[7] %>% html_table()
race_results <- data.frame(race_results) #turns from a factor to a DT
I am new to R, and I have come upon a problem I can't solve. I would like to scrape Swedish election data at electoral district level. They are structured as can be found here http://www.val.se/val/val2014/slutresultat/K/valdistrikt/25/82/0134/personroster.html
I get the data I want by using this code:
library(rvest)
district.data <- read_html("http://www.val.se/val/val2014/slutresultat/K/kommun/25/82/0134/personroster.html")
prost <- district.data %>%
html_nodes("table") %>%
.[[2]] %>%
html_table()
But that is just for one district out of 6,227 districts. The districts are identified by the html address. In the website mentioned above it is identified by "25/82/0134". I can find the identities of all districts here http://www.val.se/val/val2014/statistik/2014_riksdagsval_per_valdistrikt.skv
And I read this semi-colon separated file into R by using this code:
valres <- read_csv2("http://www.val.se/val/val2014/statistik/2014_riksdagsval_per_valdistrikt.skv" )
(as a side note, how can I change the encoding so that the Swedish letters (e.g. å, ä, ö) are imported correctly? I manage to do that with read.csv and specifying encoding='utf-8' but not with read_csv)
In this data frame, the columns LAN, KOM and VALDIST give the identities of the districts (note that VALDIST sometimes just have 2 characters). Hence the addresses have the following structure http://www.val.se/val/val2014/slutresultat/K/kommun/LAN/KOM/VALDIST/personroster.html
So, I would like to use the combination in each row to get the identity of district, scrape the information into R, add a column with the district identity (i.e. LAN, KOM and VALDIST combined into one string), and do so over all 6,227 districts and append the information from each of those districts into one single data frame. I assume I need to use some kind of loop or some of those apply functions, to iterate over the data frame, but I have not figured out how.
UPDATE:
After the help I received (thank you!) in the answer below, the code now is as follows. My remaining problem is that I want to add the district identity (i.e. paste0(LAN, KOM, VALDIST)) for each website that is scraped to a column in the final data frame. Can someone help me with this final step?
# Read the indentities of the districts (w Swedish letters)
districts_url <- "http://www.val.se/val/val2014/statistik/2014_riksdagsval_per_valdistrikt.skv"
valres <- read_csv2(districts_url, locale=locale("sv",encoding="ISO-8859-1", asciify=FALSE))
# Add a variabel to separate the two types of electoral districts
valres$typ <- "valdistrikt"
valres$typ [nchar(small_valres$VALDIST) == 2] <- "onsdagsdistrikt"
# Create a vector w all the web addresses to the district data
base_url <- "http://www.val.se/val/val2014/slutresultat/K/%s/%s/%s/%s/personroster.html"
urls <- with(small_valres, sprintf(base_url, typ, LAN, KOM, VALDIST))
# Scrape the data
pb <- progress_estimated(length(urls))
map_df(urls, function(x) {
pb$tick()$print()
# Maybe add Sys.sleep(1)
read_html(x) %>%
html_nodes("table") %>%
.[[2]] %>%
html_table()
}) -> df
Any help would be greatly appreciated!
All the best,
Richard
You can use sprintf() to do positional substitution and then use purrr::map_df() to iterate over a vector of URLs and generate a data frame:
library(rvest)
library(readr)
library(purrr)
library(dplyr)
districts_url <- "http://www.val.se/val/val2014/statistik/2014_riksdagsval_per_valdistrikt.skv"
valres <- read_csv2(districts_url, locale=locale("sv",encoding="UTF-8", asciify=FALSE))
base_url <- "http://www.val.se/val/val2014/slutresultat/K/valdistrikt/%s/%s/%s/personroster.html"
urls <- with(valres, sprintf(base_url, LAN, KOM, VALDIST))
pb <- progress_estimated(length(urls))
map_df(urls, function(x) {
pb$tick()$print()
read_html(x) %>%
html_nodes("table") %>%
.[[2]] %>%
html_table()
}) -> df
HOWEVER, you should add a randomized delay to avoid being blocked as a bot and should look at wrapping read_html() with purrr::safely() since not all those LAN/KOM/VALDIST combinations are valid URLs (at least in my testing).
That code also provides a progress bar since it's going to take a while (prbly an hour on a moderately decent connection).