Scraping from more than one aspx pages with R - r

Question

I am a linguistics student doing experiments in R. I have been looking at other questions and got a lot of help, but I am stuck at the moment as I cannot implement example functions to my case and would love to have some help.
First, I would like to go through every semester from here: http://registration.boun.edu.tr/schedule.htm, and every department here: http://registration.boun.edu.tr/scripts/schdepsel.asp
It is actually a bit easy to generate the list of it as the final link is something like this: http://registration.boun.edu.tr/scripts/sch.asp?donem=2017/2018-3&kisaadi=ATA&bolum=ATATURK+INSTITUTE+FOR+MODERN+TURKISH+HISTORY
Secondly, I need to select the code, name, days and hours of the course and tag the semester, which I did. (probably, I did it extremely poorly, but I did it nevertheless, yay!)
library("rvest")
library("dplyr")
library("magrittr")
# define the html
reg <- read_html("http://registration.boun.edu.tr/scripts/sch.asp?donem=2017/2018-3&kisaadi=ATA&bolum=ATATURK+INSTITUTE+FOR+MODERN+TURKISH+HISTORY")
# make the html a list of tables
regtable <- reg %>% html_table(fill = TRUE)
# tag their year
regtable[[4]][ ,15] <- regtable[[1]][1,2]
regtable[[4]][1,15] <- "Semester"
# Change the Days and Hours to sth usable, but how and to what?
# parse the dates, T and Th problem?
# parse the hour 10th hour problem?
# get the necessary info
regtable <- regtable %>% .[4] %>% as.data.frame() %>% select( . , X1 , X3 , X8 , X9 , V15)
# correct the names
names(regtable) <- regtable[1,]
regtable <- regtable[-1,]
View(regtable)
But the problem is that I want to write a function where I can do this for more than 20 semester and more than 50 departments. Any help would be great! I am doing this so that I can work on optimization for class hours for my department.
I guess I can do this better with XML Package, but I could not understand how to use it.
Thanks for any help,
Utku

Answer 1

Here is an answer building upon what you have already done. There are likely more efficient solutions, but this should be a good start. You also don't state how you would like to store the data, so currently what I have made will assign each combination of semester and department to its own data frame, which creates a huge amount for the number of departments. It is not ideal but I don't know how you plan to use the data after collection.
library("rvest")
library("dplyr")
library("magrittr")
# Create a Department list
dep_list <- read_html("http://registration.boun.edu.tr/scripts/schdepsel.asp")
# Take the read html and identify all objects of class menu2 and extract the
# href which will give you the final part of the url
dep_list <- dep_list %>%
html_nodes(xpath = '//*[#class="menu2"]') %>%
xml_attr("href")
department_list <- gsub("/scripts/sch.asp?donem=", "", dep_list, fixed = TRUE)
# Create a list for all of the semesters
sem_list <- read_html("http://registration.boun.edu.tr/schedule.htm")
sem_list <- sem_list %>% html_table(fill = TRUE)
# Extract the table from the list needed
semester_df <- sem_list[[2]]
# The website uses a table for the dropdown but the values are all in the second cell
# of the second column as a string
semester_list <- semester_df$X2[2]
# Separate the string into a list at the space characters
semester_list <- unlist(strsplit(semester_list, "\\s+"))
# Loop through the list of departments and within each department loop through the
# list of semesters to get the data you want
for(dep in department_list){
for(sem in semester_list){
url <- paste("http://registration.boun.edu.tr/scripts/sch.asp?donem=", sem, dep, sep = "")
reg <- read_html(url)
# make the html a list of tables
regtable <- reg %>% html_table(fill = TRUE)
# The data we want is in the 4th portion of the created list so extract that
regtable <- regtable[[4]]
# Rename the column headers to the values in the first row and remove the
# first row
regtable <- setNames(regtable[-1, ], regtable[1, ])
# Create semester column and select the variables we want
regtable <- regtable %>%
mutate(Semester = sem) %>%
select(Code.Sec, Name, Days, Hours, Semester)
# Assign the created table to a dataframe
# Could also save the file here instead
assign(paste("table", sem, gsub(" ", "_", dep), sep = "_"), regtable)
}
}

Answer 2

Thanks to #Amanda I was able to achieve what I wanted to do. Only thing left is to scraping shortnames list, match them and do the whole thing, but I can do what I want with creating a list. Any further comments to do this more elegantly are appreciated!
library("rvest")
library("dplyr")
library("magrittr")
# Create a Department list
dep_list <- read_html("http://registration.boun.edu.tr/scripts/schdepsel.asp")
dep_list <- dep_list %>% html_table(fill = TRUE)
# Select the table from the html that contains the data we want
department_df <- dep_list[[2]]
# Rename the columns with the value of the first row and remove row
department_df <- setNames(department_df[-1, ], department_df[1, ])
# Combine the two columns into a list
department_list <- c(department_df[, 1], department_df[, 2])
# Edit the department list
# We can choose accordingly.
department_list <- department_list[c(7,8,16,20,26,33,36,37,38,39)]
# Create a list for all of the semesters
sem_list <- read_html("http://registration.boun.edu.tr/schedule.htm")
sem_list <- sem_list %>% html_table(fill = TRUE)
# Extract the table from the list needed
semester_df <- sem_list[[2]]
# The website uses a table for the dropdown but the values are all in the second cell
# of the second column as a string
semester_list <- semester_df$X2[2]
# Separate the string into a list at the space characters
semester_list <- unlist(strsplit(semester_list, "\\s+"))
# Shortnames string
# We can add whichever we want.
shortname_list <- c("FLED", "HIST" , "PSY", "LL" , "PA" , "PHIL" , "YADYOK" , "SOC" , "TR" , "TKL" )
# Length
L = length(department_list)
# the function to get the schedule for the selected departments
for( i in 1:L){
for(sem in semester_list){tryCatch({
dep <- department_list[i]
sn <- shortname_list[i]
url_second_part <- interaction("&kisaadi=" , sn, "&bolum=", gsub(" ", "+", (gsub("&" , "%26", dep))), sep = "", lex.order = TRUE)
url <- paste("http://registration.boun.edu.tr/scripts/sch.asp?donem=", sem, url_second_part, sep = "")
reg <- read_html(url)
# make the html a list of tables
regtable <- reg %>% html_table(fill = TRUE)
# The data we want is in the 4th portion of the created list so extract that
regtable <- regtable[[4]]
# Rename the column headers to the values in the first row and remove the
# first row
regtable <- setNames(regtable[-1, ], regtable[1, ])
# Create semester column and select the variables we want
regtable <- regtable %>%
mutate(Semester = sem) %>%
select(Code.Sec, Name, Days, Hours, Semester)
# Assign the created table to a dataframe
# Could also save the file here instead
assign(paste("table", sem, gsub(" ", "_", dep), sep = "_"), regtable)
}, error = function(e){cat("ERROR : No information on this" , url , "\n" )})
}
}
### Maybe make Errors another dataset or list too.