R: How to read selected columns from a RDS files? - r

Question

How to read part of the data from very large files?
The sample data is generated as:
set.seed(123)
df <- data.frame(replicate(10, sample(0:2000, 15 * 10^5, rep = TRUE)),
replicate(10, stringi::stri_rand_strings(1000, 5)))
head(df)
# X1 X2 X3 X4 X5 X6 X7 X8 X9 X10 X1.1 X2.1 X3.1 X4.1 X5.1 X6.1 X7.1 X8.1 X9.1 X10.1
# 1 575 1843 1854 883 592 1362 1075 210 1526 1365 Qk8NP Xvw9z OYRa1 8BGIV bejiv CCoIE XDKJN HR7zc 2kKNY 1I5h8
# 2 1577 390 1861 912 277 636 758 1461 1978 1865 ZaHFl QLsli E7lbs YGq8u DgUAW c6JQ0 RAZFn Sc0Zt mif8I 3Ys6U
# 3 818 1076 147 1221 257 1115 759 1959 1088 1292 jM5Uw ctM3y 0HiXR hjOHK BZDOP ULQWm Ei8qS BVneZ rkKNL 728gf
# 4 1766 884 1331 1144 1260 768 1620 1231 1428 1193 r4ZCI eCymC 19SwO Ht1O0 repPw YdlSW NRgfL RX4ta iAtVn Hzm0q
# 5 1881 1851 1324 1930 1584 1318 940 1796 830 15 w8d1B qK1b0 CeB8u SlNll DxndB vaufY ZtlEM tDa0o SEMUX V7tLQ
# 6 91 264 1563 414 914 1507 1935 1970 287 409 gsY1u FxIgu 2XqS4 8kreA ymngX h0hkK reIsn tKgQY ssR7g W3v6c
saveRDS is used to save the file.
saveRDS(df, 'df.rds')
The file size is looked using the below commands:
file.info('df.rds')$size
# [1] 29935125
utils:::format.object_size(29935125, "auto")
# [1] "28.5 Mb"
The saved file is read using the below function.
readRDS('df.rds')
However, some of my files are in GBs and would need only few columns for certain processing. Is it possible to read selected columns from RDS files?
Note: I already have RDS files, generated after considerably large amounts of processing. Now, I want to know the best possible way to read selected columns from the existing RDS files.

Answer 1

I don't think you can read only a portion of an rds or rda file.
An alternative would be to use feather. As an example, using a large-ish feather I'm working with:
library(feather)
file.info("../feathers/C1.feather")["size"]
# size
# ../feathers/C1.feather 498782328
system.time( c1whole <- read_feather("../feathers/C1.feather") )
# user system elapsed
# 0.860 0.856 5.540
system.time( c1dyn <- feather("../feathers/C1.feather") )
# user system elapsed
# 0 0 0
ls.objects()
# Type Size PrettySize Dim
# c1dyn feather 3232 3.2 Kb 2886147 x 36
# c1whole tbl_df 554158688 528.5 Mb 2886147 x 36
You can react with both variables as full data.frames: though c1whole is already in memory (so may be a little faster), accessing c1dyn is still quite speedy.
NB: some functions (e.g., several within dplyr) do not work on feather as they do on data.frame or tbl_df. If your intent is solely to pick-and-choose specific columns, then you'll be fine.

Answer 2

SQLite also could be a common way to store tabular/matrix/dataframe data on your hard drive using an SQLite database. This also allows the use of standard SQL commands or DPLYR to interrogate the data. Just be warned that SQLite does not have a date format so any dates need to be converted to character before writing them to the database.
set.seed(123)
df <- data.frame(replicate(10, sample(0:2000, 15 * 10^5, rep = TRUE)),
replicate(10, stringi::stri_rand_strings(1000, 5)))
library(RSQLite)
conn <- dbConnect(RSQLite::SQLite(), dbname="myDB")
dbWriteTable(conn,"mytable",df)
alltables <- dbListTables(conn)
# Use sql queries to query data...
oneColumn <- dbGetQuery(conn,"SELECT X1 FROM mytable")
library(dplyr)
library(dbplyr)
my_db <- tbl(conn, "mytable")
my_db
# Use dplyr functions to query data...
my_db %>% select(X1)