I am very new to R and would like to use some code to run various batch code on all of the data that I have available. It should be clear what I'm trying to do:
# library(PerformanceAnalytics)
# mydata <- mtcars[, c('mpg', 'cyl', 'disp', 'hp', 'carb')];
# chart.Correlation(mydata, histogram=TRUE, pch=19)
library(MASS)
M_names = data(package = "MASS")$result[, "Item"]
for (i in 1:length(M_names)) {
eval(paste("MASS::", M_names[i], sep=""));
}
The commented part is some code I found that I haven't been able to integrate yet. The Correlation is a very cool correlation matrix, which I'm attempting to funnel every single dataset I have access to into so I can quickly review them instead of doing it all manually. I guess I will need to save them all to PNGs to have practical workflow around that, as it's clear there's no way to coax the X windows to appear or stay put when running R code as a script.
The behavior I observe as I execute this on my Mac is:
> library(MASS)
> M_names = data(package = "MASS")$result[, "Item"]
> for (i in 1:length(M_names)) {
+ eval(paste("MASS::", M_names[i], sep=""));
+ }
>
>
I don't know for sure what the silent + indicator means, but I'm pretty sure it just means that code line is inside the for loop scope. But the eval is swallowing the command I assembled. I'm just trying to get it to print out the content of the data at each iteration of the loop for now.
I also noticed this:
> eval("MASS::ships")
[1] "MASS::ships"
It just prints it when I try to eval it.
I also hope there is a way to programmatically print individual datasets. I'm already hacking really hard at this, and there is no way that what I am doing here is a good idea.
If you have the package dataset names in a vector the key to accessing them
by their character names is the get function:
library(MASS)
M_names = data(package = "MASS")$result[, "Item"]
head(get(M_names[1]), 1)
# state sex diag death status T.categ age
# 1 NSW M 10905 11081 D hs 35
You can then loop through the vector of names
for (DATA in M_names) print(summary(get(DATA)))
Another options is to use the envir argument of the data function to load the datasets into a specific environment. It may be worth adding the data to a new environment instead of polluting your workspace.You can do that with
data(list=M_names, package="MASS", envir = list_of_datafames<- new.env())
You can then look through the list_of_datafames as you would with an other list object:
lapply(list_of_datafames, summary)
I am analyzing FCS files from a CyTOF experiment using Flowcore package
. When I import and export my FCS files using read.FCS and write.FCS, I find that these functions have corrupted my FCS file and all channels are affected and the data looks like the tSNE in the picture below (not what is expected or meaningful).
I'm using R (ver.3.6), Rstudio (1.2.1335), and flowcore ver.3.9.
Here is the code I have used:
library(flowCore)
#Import FCS file
myfilename<-"export_MIX_NT_Ungated_viSNE.fcs"
myfile_fcs<-read.FCS(myfilename,
transformation="linearize", which.lines=NULL,
alter.names=FALSE, column.pattern=NULL)
#I plan to do some data analysis here in the final version before exporting below
#export the fcs file and rename it to T_+filename
write.FCS(myfile_fcs,paste("T_",keyword(myfile_fcs)$"$FIL",sep=""), what="numeric")
and this is what the original file looks like before import into R
and this is what the exported result looks like after export
Here is the file that we have used for this code: dropbox link for the example file
I've looked into your problem and at first I was skeptical about the transformation of read.fcs. Looking into your example file, I also see that there are already columns for your original (full plot) tsne plot, so I'm assuming flowjo is rewriting the tsne values after you read/write it into R. Since Flowcore is generally more targeted towards flow data and not cytof, I took a few pieces of this Bioc2017 walkthough and recreated the transformations, which seems to work better although I'm not sure how flowjo will handle the data now. If you were going to do more work on the data though, we now have it at an accessible low level so you can basically do whatever you want. Here's my code.
fcs_raw <- read.flowSet("~/Downloads/export_MIX_NT_Ungated_viSNE.fcs", transformation = FALSE,
truncate_max_range = FALSE)
fcs <- fsApply(fcs_raw, function(x, cofactor = 5){
expr <- exprs(x)
expr <- asinh(expr[,] / cofactor)
exprs(x) <- expr
x
})
expr <- fsApply(fcs, exprs)
library(matrixStats)
rng <- colQuantiles(expr, probs = c(0.01, 0.99))
expr01 <- t((t(expr) - rng[, 1]) / (rng[, 2] - rng[, 1]))
expr01[expr01 < 0] <- 0
expr01[expr01 > 1] <- 1
expr01
summary(expr01)
Be aware that this does mess up your original tSNE column numbers, so if these were important to you, I would read the flowset, make a copy of those columns, and move on with the data analysis in the code. If you have future questions or analysis with flow data feel free to contact me directly.
#csugai, thanks for your answer. The truncate_max_range = FALSE argument in the read.flowSet function caught my eyes so I included that into my read.FCS function and that fixed the problem! Although I didn't really understand other parts of your code that resulted in a binned data.
The R package vtreat provides a handy way of creating "one-hot encoders" for the categorical variables (see a relevant post at the Win-Vector blog). Is there any way to save the treatment plan tplan object for further use (e.g., equivalent mechanism of pickle in Python).
tplan <- vtreat::designTreatmentsZ(dTrain, vars)
oneHotEncoded <- as.matrix(vtreat::prepare(tplan, dTrain, varRestriction = vars))
I would like to transform whatever data I will get with this particular treatment plan (which was computed on the dTrain), in a situation where the dTrain is no longer available. That is, I cannot re-use dTrain the next time I will call the script.
P.s. the solution should not necessary be confined to using vtreat
Base R provides the general functions save() and load() for such purposes.
Here is a reproducible example using code snippets from the post you have linked to:
library(titanic)
library(vtreat)
data(titanic_train)
outcome <- 'Survived'
target <- 1
shouldBeCategorical <- c('PassengerId', 'Pclass', 'Parch')
for(v in shouldBeCategorical) {
titanic_train[[v]] <- as.factor(titanic_train[[v]])
}
tooDetailed <- c("Ticket", "Cabin", "Name", "PassengerId")
vars <- setdiff(colnames(titanic_train), c(outcome, tooDetailed))
dTrain <- titanic_train
set.seed(4623762)
tplan <- vtreat::designTreatmentsZ(dTrain, vars,
minFraction= 0,
verbose=FALSE)
save(tplan, file='tplan.RData')
The file tplan.RData will be saved in your current working directory; afterwards, in a new R session, when you ask for
load('tplan.RData')
you will get your tplan variable back.
Alternatively, base R functions saveRDS and loadRDS will also do the job; their usage is exactly similar, and they seem to be preferable.
I have multiple time series (each in a seperate file), which I need to adjust seasonally using the season package in R and store the adjusted series each in a seperate file again in a different directory.
The Code works for a single county.
So I tried to use a for Loop but R is unable to use the read.dta with a wildcard.
I'm new to R and using usually Stata so the question is maybe quite stupid and my code quite messy.
Sorry and Thanks in advance
Nathan
for(i in 1:402)
{
alo[i] <- read.dta("/Users/nathanrhauke/Desktop/MA_NH/Data/ALO/SEASONAL_ADJUSTMENT/SINGLE_SERIES/County[i]")
alo_ts[i] <-ts(alo[i], freq = 12, start = 2007)
m[i] <- seas(alo_ts[i])
original[i]<-as.data.frame(original(m[i]))
adjusted[i]<-as.data.frame(final(m[i]))
trend[i]<-as.data.frame(trend(m[i]))
irregular[i]<-as.data.frame(irregular(m[i]))
County[i] <- data.frame(cbind(adjusted[i],original[i],trend[i],irregular[i], deparse.level =1))
write.dta(County[i], "/Users/nathanrhauke/Desktop/MA_NH/Data/ALO/SEASONAL_ADJUSTMENT/ADJUSTED_SERIES/County[i].dta")
}
This is a good place to use a function and the *apply family. As noted in a comment, your main problem is likely to be that you're using Stata-like character string construction that will not work in R. You need to use paste (or paste0, as here) rather than just passing the indexing variable directly in the string like in Stata. Here's some code:
f <- function(i) {
d <- read.dta(paste0("/Users/nathanrhauke/Desktop/MA_NH/Data/ALO/SEASONAL_ADJUSTMENT/SINGLE_SERIES/County",i,".dta"))
alo_ts <- ts(d, freq = 12, start = 2007)
m <- seas(alo_ts)
original <- as.data.frame(original(m))
adjusted <- as.data.frame(final(m))
trend <- as.data.frame(trend(m))
irregular <- as.data.frame(irregular(m))
County <- cbind(adjusted,original,trend,irregular, deparse.level = 1)
write.dta(County, paste0("/Users/nathanrhauke/Desktop/MA_NH/Data/ALO/SEASONAL_ADJUSTMENT/ADJUSTED_SERIES/County",i,".dta"))
invisible(County)
}
# return a list of all of the resulting datasets
lapply(1:402, f)
It would probably also be a good idea to take advantage of relative directories by first setting your working directory:
setwd("/Users/nathanrhauke/Desktop/MA_NH/Data/ALO/SEASONAL_ADJUSTMENT/")
Then you can simply the above paths to:
d <- read.dta(paste0("./SINGLE_SERIES/County",i,".dta"))
and
write.dta(County, paste0("./ADJUSTED_SERIES/County",i,".dta"))
which will make your code more readable and reproducible should, for example, someone ever run it on another computer.
I know I can use ls() and rm() to see and remove objects that exist in my environment.
However, when dealing with "old" .RData file, one needs to sometimes pick an environment a part to find what to keep and what to leave out.
What I would like to do, is to have a GUI like interface to allow me to see the objects, sort them (for example, by there size), and remove the ones I don't need (for example, by a check-box interface). Since I imagine such a system is not currently implemented in R, what ways do exist? What do you use for cleaning old .RData files?
Thanks,
Tal
I never create .RData files. If you are practicing reproducible research (and you should be!) you should be able to source in R files to go from input data files to all outputs.
When you have operations that take a long time it makes sense to cache them. If often use a construct like:
if (file.exists("cache.rdata")) {
load("cache.rdata")
} else {
# do stuff ...
save(..., file = "cache.rdata")
}
This allows you to work quickly from cached files, and when you need to recalculate from scratch you can just delete all the rdata files in your working directory.
Basic solution is to load your data, remove what you don't want and save as new, clean data.
Another way to handle this situation is to control loaded RData by loading it to own environment
sandbox <- new.env()
load("some_old.RData", sandbox)
Now you can see what is inside
ls(sandbox)
sapply(ls(sandbox), function(x) object.size(get(x,sandbox)))
Then you have several posibilities:
write what you want to new RData: save(A, B, file="clean.RData", envir=sandbox)
remove what you don't want from environment rm(x, z, u, envir=sandbox)
make copy of variables you want in global workspace and remove sandbox
I usually do something similar to third option. Load my data, do some checks, transformation, copy final data to global workspace and remove environments.
You could always implement what you want. So
Load the data
vars <- load("some_old.RData")
Get sizes
vars_size <- sapply(vars, function(x) object.size(get(x)))
Order them
vars <- vars[order(vars_size, decreasing=TRUE)]
vars_size <- vars_size [order(vars_size, decreasing=TRUE)]
Make dialog box (depends on OS, here is Windows)
vars_with_size <- paste(vars,vars_size)
vars_to_save <- select.list(vars_with_size, multiple=TRUE)
Remove what you don't want
rm(vars[!vars_with_size%in%vars_to_save])
To nice form of object size I use solution based on getAnywhere(print.object_size)
pretty_size <- function(x) {
ifelse(x >= 1024^3, paste(round(x/1024^3, 1L), "Gb"),
ifelse(x >= 1024^2, paste(round(x/1024^2, 1L), "Mb"),
ifelse(x >= 1024 , paste(round(x/1024, 1L), "Kb"),
paste(x, "bytes")
)))
}
Then in 4. one can use paste(vars, pretty_size(vars_size))
You may want to check out the RGtk2 package.
You can very easily create an interface with Glade Interface Designer and then attach whatever R commands you want to it.
If you want a good starting point where to "steal" ideas on how to use RGtk2, install the rattle package and run rattle();. Then look at the source code and start making your own interface :)
I may have a go at it and see if I can come out with something simple.
EDIT: this is a quick and dirty piece of code that you can play with. The big problem with it is that for whatever reason the rm instruction does not get executed, but I'm not sure why... I know that it is the central instruction, but at least the interface works! :D
TODO:
Make rm work
I put all the variables in the remObjEnv environment. It should not be listed in the current variable and it should be removed when the window is closed
The list will only show objects in the global environment, anything inside other environment won't be shown, but that's easy enough to implement
probably there's some other bug I haven't thought of :D
Enjoy
# Our environment
remObjEnv <<- new.env()
# Various required libraries
require("RGtk2")
remObjEnv$createModel <- function()
{
# create the array of data and fill it in
remObjEnv$objList <- NULL
objs <- objects(globalenv())
for (o in objs)
remObjEnv$objList[[length(remObjEnv$objList)+1]] <- list(object = o,
type = typeof(get(o)),
size = object.size(get(o)))
# create list store
model <- gtkListStoreNew("gchararray", "gchararray", "gint")
# add items
for (i in 1:length(remObjEnv$objList))
{
iter <- model$append()$iter
model$set(iter,
0, remObjEnv$objList[[i]]$object,
1, remObjEnv$objList[[i]]$type,
2, remObjEnv$objList[[i]]$size)
}
return(model)
}
remObjEnv$addColumns <- function(treeview)
{
colNames <- c("Name", "Type", "Size (bytes)")
model <- treeview$getModel()
for (n in 1:length(colNames))
{
renderer <- gtkCellRendererTextNew()
renderer$setData("column", n-1)
treeview$insertColumnWithAttributes(-1, colNames[n], renderer, text=n-1)
}
}
# Builds the list.
# I seem to have some problems in correctly build treeviews from glade files
# so we'll just do it by hand :)
remObjEnv$buildTreeView <- function()
{
# create model
model <- remObjEnv$createModel()
# create tree view
remObjEnv$treeview <- gtkTreeViewNewWithModel(model)
remObjEnv$treeview$setRulesHint(TRUE)
remObjEnv$treeview$getSelection()$setMode("single")
remObjEnv$addColumns(remObjEnv$treeview)
remObjEnv$vbox$packStart(remObjEnv$treeview, TRUE, TRUE, 0)
}
remObjEnv$delObj <- function(widget, treeview)
{
model <- treeview$getModel()
selection <- treeview$getSelection()
selected <- selection$getSelected()
if (selected[[1]])
{
iter <- selected$iter
path <- model$getPath(iter)
i <- path$getIndices()[[1]]
model$remove(iter)
}
obj <- as.character(remObjEnv$objList[[i+1]]$object)
rm(obj)
}
# The list of the current objects
remObjEnv$objList <- NULL
# Create the GUI.
remObjEnv$window <- gtkWindowNew("toplevel", show = FALSE)
gtkWindowSetTitle(remObjEnv$window, "R Object Remover")
gtkWindowSetDefaultSize(remObjEnv$window, 500, 300)
remObjEnv$vbox <- gtkVBoxNew(FALSE, 5)
remObjEnv$window$add(remObjEnv$vbox)
# Build the treeview
remObjEnv$buildTreeView()
remObjEnv$button <- gtkButtonNewWithLabel("Delete selected object")
gSignalConnect(remObjEnv$button, "clicked", remObjEnv$delObj, remObjEnv$treeview)
remObjEnv$vbox$packStart(remObjEnv$button, TRUE, TRUE, 0)
remObjEnv$window$showAll()
Once you've figured out what you want to keep, you can use the function -keep- from package gdata does what its name suggests.
a <- 1
b <- 2
library(gdata)
keep(a, all = TRUE, sure = TRUE)
See help(keep) for details on the -all- and -sure- options.
all: whether hidden objects (beginning with a .) should be removed, unless explicitly kept.
sure: whether to perform the removal, otherwise return names of objects that would have been removed.
This function is so useful that I'm surprised it isn't part of R itself.
The OS X gui does have such a thing, it's called the Workspace Browser. Quite handy.
I've also wished for an interface that shows the session dependency between objects, i.e. if I start from a plot() and work backwards to find all the objects that were used to create it. This would require parsing the history.
It doesn't have checkboxes to delete with, rather you select the file(s) then click delete. However, the solution below is pretty easy to implement:
library(gWidgets)
options(guiToolkit="RGtk2")
## make data frame with files
out <- lapply((x <- list.files()), file.info)
out <- do.call("rbind", out)
out <- data.frame(name=x, size=as.integer(out$size), ## more attributes?
stringsAsFactors=FALSE)
## set up GUI
w <- gwindow("Browse directory")
g <- ggroup(cont=w, horizontal=FALSE)
tbl <- gtable(out, cont=g, multiple=TRUE)
size(tbl) <- c(400,400)
deleteThem <- gbutton("delete", cont=g)
enabled(deleteThem) <- FALSE
## add handlers
addHandlerClicked(tbl, handler=function(h,...) {
enabled(deleteThem) <- (length(svalue(h$obj, index=TRUE)) > 0)
})
addHandlerClicked(deleteThem, handler=function(h,...) {
inds <- svalue(tbl, index=TRUE)
files <- tbl[inds,1]
print(files) # replace with rm?
})
The poor guy answer could be :
ls()
# spot the rank of the variables you want to remove, for example 10 to 25
rm(list= ls()[[10:25]])
# repeat until satisfied
To clean the complete environment you can try:
rm(list(ls())