Unlist Function Is Turning Numeric Values Into Characters. R - r

Question

The code is scraping a website for stock data and returns a 1x18 dataframe for each stock. I am trying to convert the dataframe into a vector without turning the numeric columns into factors which is what is happening. I have also attempted to try and turn the dataframe into a matrix, but the numeric columns are still being converted into factors. In conclusion, I would like to keep characters as characters and numeric as numeric all in a vector. Thank you.
#get.dates is a function I created to scrape
data = get.dates("AAPL")
class(data)
[1] "data.frame"
class(data$surprise)
[1] "numeric"
dput(data)
structure(list(date = "2019-05-07T00:00:00", company = "Apple",
ticker = "AAPL", periodEnding = "Mar 2019", eps = "2.37",
reportedEPS = NA_character_, lastEps = "2.73", consensus = 4L,
bpConsensus = 4L, ratingsAndPT = structure(list(priceTarget = 177.34,
numBuys = 17L, numHolds = 18L, numSells = 0L), class = "data.frame", row.names = c(NA,
-1L)), bpRatingsAndPT = structure(list(priceTarget = 176.88,
numBuys = 14L, numHolds = 14L, numSells = 0L), class = "data.frame", row.names = c(NA,
-1L)), marketCap = 827573630900, sector = 18731L, stockId = 7624L,
stockTypeId = 1L, surprise = NA_real_, timeOfDay = 4L, isConfirmed = FALSE), class = "data.frame", row.names = c(NA,
-1L))
data = unlist(data)
class(data)
[1] "character"
So the final output is to rbind each of the outputs into a single data.frame.
I think I have to convert each 1x18 dataframe into a vector to rbind because I am getting an error when trying to rbind the columns using the foreach package.
tickers = c("AAPL", "PEP", "KO")
system.time({
data = foreach(r = tickers, .packages = c("jsonlite", "dplyr"), .combine = rbind) %dopar% {get.dates(r)}
})
error calling combine function:
<simpleError in `.rowNamesDF<-`(x, value = value): duplicate 'row.names' are not allowed>
user system elapsed
0.02 0.00 0.56
Warning message:
non-unique value when setting 'row.names': ‘1’
print(data)
NULL
#I will do the same thing outside of the foreach loop to give some more context
data = lapply(tickers, get.dates)
do.call(rbind, data)
Error in `.rowNamesDF<-`(x, value = value) :
duplicate 'row.names' are not allowed
In addition: Warning message:
non-unique value when setting 'row.names': ‘1’
dput(data)
list(structure(list(date = "2019-05-07T00:00:00", company = "Apple",
ticker = "AAPL", periodEnding = "Mar 2019", eps = "2.37",
reportedEPS = NA_character_, lastEps = "2.73", consensus = 4L,
bpConsensus = 4L, ratingsAndPT = structure(list(priceTarget = 177.34,
numBuys = 17L, numHolds = 18L, numSells = 0L), class = "data.frame", row.names = c(NA,
-1L)), bpRatingsAndPT = structure(list(priceTarget = 176.88,
numBuys = 14L, numHolds = 14L, numSells = 0L), class = "data.frame", row.names = c(NA,
-1L)), marketCap = 827573630900, sector = 18731L, stockId = 7624L,
stockTypeId = 1L, surprise = NA_real_, timeOfDay = 4L, isConfirmed = FALSE), class = "data.frame", row.names = c(NA,
-1L)), structure(list(date = "2019-04-23T00:00:00", company = "Coca-Cola",
ticker = "KO", periodEnding = "Mar 2019", eps = "0.46", reportedEPS = NA_character_,
lastEps = "0.47", consensus = 4L, bpConsensus = 5L, ratingsAndPT = structure(list(
priceTarget = 50.89, numBuys = 4L, numHolds = 5L, numSells = 0L), class = "data.frame", row.names = c(NA,
-1L)), bpRatingsAndPT = structure(list(priceTarget = 51.25,
numBuys = 3L, numHolds = 1L, numSells = 0L), class = "data.frame", row.names = c(NA,
-1L)), marketCap = 193681840000, sector = 18731L, stockId = 8359L,
stockTypeId = 1L, surprise = NA_real_, timeOfDay = 4L, isConfirmed = FALSE), class = "data.frame", row.names = c(NA,
-1L)), structure(list(date = "2019-04-25T00:00:00", company = "PepsiCo",
ticker = "PEP", periodEnding = "Mar 2019", eps = "0.92",
reportedEPS = NA_character_, lastEps = "0.96", consensus = 4L,
bpConsensus = 4L, ratingsAndPT = structure(list(priceTarget = 123.67,
numBuys = 4L, numHolds = 3L, numSells = 0L), class = "data.frame", row.names = c(NA,
-1L)), bpRatingsAndPT = structure(list(priceTarget = 126,
numBuys = 1L, numHolds = 1L, numSells = 0L), class = "data.frame", row.names = c(NA,
-1L)), marketCap = 163697620000, sector = 18731L, stockId = 10962L,
stockTypeId = 1L, surprise = NA_real_, timeOfDay = 4L, isConfirmed = FALSE), class = "data.frame", row.names = c(NA,
-1L)))
Here is what I would like the output to look like. Thank you!!

Answer 1

You basically have to do your own list flattening here, which is not desirable. It's easier to do this when you get the json data originally. https://rdrr.io/cran/jsonlite/man/flatten.html
The below solution users purrr but you can do it with a for-loop or apply functions if you prefer. There are two main ideas here:
1. Bind together the dataframe-type columns with the part of the dataframe that doesn't have any nested columns. In your example, we bind together 3 separate pieces: 1 original dataframe with df_cols removed, and the other two dataframe columns. You can do this with bind_cols. It helps to prepend the original column names to avoid duplicates.
2. Collapse all the rows together with rbind or the like.
flatten_df_cols <- function(df) {
df_cols <- map_lgl(df, is.data.frame)
imap_dfc(df[, df_cols], ~setNames(.x, paste0(.y, ".", names(.x)))) %>%
bind_cols(list(df[, !df_cols]), .)
}
map_dfr(data, flatten_df_cols)
Observations: 3
Variables: 24
$ date <chr> "2019-05-07T00:00:00", "2019-04...
$ company <chr> "Apple", "Coca-Cola", "PepsiCo"
$ ticker <chr> "AAPL", "KO", "PEP"
$ periodEnding <chr> "Mar 2019", "Mar 2019", "Mar 2019"
$ eps <chr> "2.37", "0.46", "0.92"
$ reportedEPS <chr> NA, NA, NA
$ lastEps <chr> "2.73", "0.47", "0.96"
$ consensus <int> 4, 4, 4
$ bpConsensus <int> 4, 5, 4
$ marketCap <dbl> 827573630900, 193681840000, 163...
$ sector <int> 18731, 18731, 18731
$ stockId <int> 7624, 8359, 10962
$ stockTypeId <int> 1, 1, 1
$ surprise <dbl> NA, NA, NA
$ timeOfDay <int> 4, 4, 4
$ isConfirmed <lgl> FALSE, FALSE, FALSE
$ ratingsAndPT.priceTarget <dbl> 177.34, 50.89, 123.67
$ ratingsAndPT.numBuys <int> 17, 4, 4
$ ratingsAndPT.numHolds <int> 18, 5, 3
$ ratingsAndPT.numSells <int> 0, 0, 0
$ bpRatingsAndPT.priceTarget <dbl> 176.88, 51.25, 126.00
$ bpRatingsAndPT.numBuys <int> 14, 3, 1
$ bpRatingsAndPT.numHolds <int> 14, 1, 1
$ bpRatingsAndPT.numSells <int> 0, 0, 0