I am working with a dataframe, where one of the columns is a multivalued variable, which I've implemented as a list as column.
Here's a reproducible example:
df <- data.frame(title=c('one','two','three'), subjects=I(list(c('A'), c('A','B','C','D'), c('B','D','E'))))
The general idea being that I can attach as many subjects as I'd like without using too much space.
Now the set of possible subjects isn't that large, so if it were a simple column, I'd turn it into a factor. But if I do that here, R stores the levels attribute separately for each list item, (i.e. each row), once again needing a huge amount of storage.
Does anyone know of a way to store a list of factors, with the levels of these factors as a shared attribute?
The only thing I could think of was to do it myself, store the values as integers and create a separate lookup-table, but that doesn't look very efficient.
Related
I am going to do repeated measures ANOVA on my data, but to this point, my data is wide. Two independent (categorical) variables are spread across single responsive variable.
See the image: https://imgur.com/1eTWSIM
I want to create two categorical variables that take values from the different parts of the columns (circled on the screenshot). Subject numbers should be kept as a category. So after using gather() function, the data should look something like this:
https://imgur.com/SGM2N69
I've seen in a tutorial (that I can't find anymore) that you can create two columns from a single function, using different parts of the colnames (using "_" as a separator), but I can't exactly remember how it was done.
Any help would be appreciated and ask if anythings is not clear in my explanation.
I have solved the problem by using 'gather()' function first and then 'separate()' to separate it into two new columns. So I guess, if you want to make two key columns, first you have to make a single column containing both values and later separate it into two.
At least that is how I did it.
I have a large, multidimensional list as a result of a statistic's project. The list holds different objects, holding objects by themselves. There are also plots, matrices etc. It's a heterogeneous mix of a lot of different types and different dimensionalities.
Now I have to change the name of one variable completely. Every occurence has to be overriden. Is there a way to do this?
Here is a little example. There's no use in solving this example explicitely, as my list is much larger.
a <- list(entry1=list("a","b","c","xx",p=c(3,4,"xx")),
entry2=list(matrix(c(1,2,"xx",4), nrow = 2),xx=list(6,7,8,"xx")),
xx=list(1,2,3,4,"xx"))
How can I change the xx to yy? Thanks in advance!
I need to subset my data depending on the content of one factor variable.
I tried to do it with subset:
new <- subset(data, original$Group1=="SALAD")
data is already a subset from a bigger data frame, in original I have the factor variable which should identify the wanted rows.
This works perfectly for one level of the factor variable, but (and I really don´t understand why!!) when I do it with the other factor level "BREAD" it creates the data frame but says "no data available" - so it is empty. I´ve imported the data from SPSS, if this matters. I´ve already checked the factor levels, but the naming should be right!
Would be really grateful for help, I spent 3 hours on this problem and wasn´t able to find a solution.
I´ve also tried other ways to subset my data (e.g. split), but I want a data frame as output.
Do you have advice in general, what is the best way to subset a data frame if I want e.g. 3 columns of this data frame and these should be extracted depending on the level of a factor (most Code examples are only for one or all columns..)
The entire point of the subset function (as I understand it) is to look inside the data frame for the right variable - so you can type
subset(data, var1 == "value")
instead of
data[data$var1 == "value,]
Please correct me anyone if that is incorrect.
Now, in you're case, you are explicitly taking Group1 from the data frame original and using that to subset data - which you say is a subset of original. Based on this, I see no reason to believe (and every reason not to believe) that the elements of original$Group1 will align with the rows of data. If Group1 is defined within data, why not just use the copy defined there - which is aligned correctly? If not, you need to be very explicit about what you are trying to accomplish, so that you can ensure that things are aligned correctly.
I am working on a data set in R having dimensions
dim(adData)
[1] 15844717 11
Out of 11 features,
one feature is having 273596(random integers used as id) unique values out of 15844717.
second feature is having 884353(random integers used as id) unique values out of 15844717.
My confusion is whether to convert them into factors or not because categorical variables with large number of levels will create a problem at the time of modelling or please suggest how to treat them.
I am new to Data Science and never worked on large data sets before.
~300k categories for one variable is sure to cause computational issues. I would first take a step back and examine the nature of this variable and its relevance to the prediction at hand. Without knowing the source of the data, it is hard to give specific advice.
If it is truly a categorical variable, it would be silly to leave the ids as numeric variables since the scale and order of the ids are likely meaningless.
Is it possible to group the levels into fewer but still meaningful categories?
Example 1: If the ids were zipcodes in the United States, there are potentially 40,000 unique values. These can be grouped into states or regions, reducing the number of levels to 50 or fewer.
Example 2: If the ids were product ids from an e-commerce site, they could be grouped by product category or sub-category. There would be much fewer distinct values to work with.
Another option is to examine the relative frequency of each category. If there are a few very common categories, with thousands of rare categories, you leave the common levels in tact and group the rare levels into an 'other' category.
I have this csv with 4000+ entries and I am trying to create a histogram of one of the variables. Because of the way the data was collected, there was a possibility that if data was uncollectable for that entry, it was coded as a period (.). I still want to create a histogram and just ignore that specific entry.
What would be the best or easiest way to go about this?
I tried making it so that the histogram would only use the data for every entry except the one with the period by doing
newlist <- data1$var[1:3722]+data1$var[3724:4282]
where 3723 is the entry with the period, but R said that + is not meaningful for factors. I'm not sure if I went about this the right way, my intention was to create a vector or list or table conjoining those two subsets above into one bigger list called newlist.
Your problem is deeper that you realize. When R read in the data and saw the lone . it interpreted that column as a factor (categorical variable).
You need to either convert the factor back to a numeric variable (this is FAQ 7.10) or reread the data forcing it to read that column as numeric, if you are using read.table or one of the functions that calls read.table then you can set the colClasses argument to specify a numeric column.
Once the column of data is a numeric variable then a negative subscript or !is.na will work (or some functions will automatically ignore the missing value).