I have an H2O frame R object like this
h2odf
A | B | C | D
--|---|---|---
1 | NA| 2 | 0
2 | 1 | 2 | 0
3 | NA| 2 | 0
4 | 3 | 2 | 0
I want to remove all those rows where B is NA (1st and 3rd row). I have tried
na <- is.na(h2odf[,"b"])
h2odf <- h2odf[!na,]
and
h2odf <- h2odf[!is.na(h2odf$B),]
and
h2odf <- subset(h2odf, B!=NA)
This works for R Dataframe but not H2O. Giving this error:
Error in .h2o.doSafeREST(h2oRestApiVersion = h2oRestApiVersion, urlSuffix = page, :
ERROR MESSAGE:
DistributedException from localhost/127.0.0.1:54321: 'Cannot set illegal UUID value'
Desired output is
h2odf
A | B | C | D
--|---|---|---
2 | 1 | 2 | 0
4 | 3 | 2 | 0
One option I have is to convert it into R Dataframe, remove rows and convert it back to H2O frame. But that is taking long time because input file size is close to 4.5 GB. Is it possible to do this in H2O frame hex object itself?
I am running Rstudio on aws cluster.
> class(h2odf)
[1] "H2OFrame"
> h2odf
A B C D
1 1 NA 2 0
2 2 1 2 0
3 3 NA 2 0
4 4 3 2 0
[4 rows x 4 columns]
> h2odf[!is.na(as.numeric(as.character(h2odf$B))),]
A B C D
1 2 1 2 0
2 4 3 2 0
[2 rows x 4 columns]
Related
I have a data frame with the following columns Entity, Customer Class, Month and other
|CClass |Entity |Month| Sales volume|
|-------|--------|-----|-------------|
|Bakery | 1 | 1 |100 |
|Bakery | 1 | 2 |106 |
|Bakery | 1 | 3 |103 |
|Bakery | 1 | 5 |135 |
|Bakery | 1 | 6 |121 |
|Bakery | 1 | 7 |176 |
|Bakery | 1 | 10 |133 |
|Bakery | 1 | 11 |100 |
|Bakery | 1 | 12 |112 |
|Bakery | 2 | 1 |136 |
|Bakery | 2 | 3 |123 |
|Bakery | 2 | 4 |108 |
|Bakery | 2 | 5 |101 |
|Bakery | 2 | 7 |105 |
|Bakery | 3 | 10 |103 |
|Bakery | 3 | 11 |106 |
|Bakery | 3 | 12 |110 |
|Grocery| 1 | 1 |120 |
|Grocery| 1 | 2 |150 |
When I try to populate the missing Month to each Customer Class using the complete() function:
DF <- complete(DF, nesting(Entity, CClass), Month)
I got the Error message "! object 'Entity' not found"
st <- complete(ST, nesting(Entity, CClass), SBMONTH)
Error in dplyr::summarise():
! Problem while computing ..1 = complete(data = dplyr::cur_data(), ..., fill = fill, explicit = explicit).
i The error occurred in group 1: CClass = "Bagel Shop", End Market = "Food Service", Entity = 1.
Caused by error:
! object 'Entity' not found
Run rlang::last_error() to see where the error occurred.
But with the testing samples this function works.
Please advise
I can't reproduce the error. Starting from a fresh R session and using this data:
DF = read.table(text = 'CClass Entity Month Sales_volume
Bakery 1 1 100
Bakery 1 2 106
Bakery 1 3 103
Bakery 1 5 135
Bakery 1 6 121
Bakery 1 7 176
Bakery 1 10 133
Bakery 1 11 100
Bakery 1 12 112
Bakery 2 1 136
Bakery 2 3 123
Bakery 2 4 108
Bakery 2 5 101
Bakery 2 7 105
Bakery 3 10 103
Bakery 3 11 106
Bakery 3 12 110
Grocery 1 1 120
Grocery 1 2 150', header = T)
I load tidyr and run the complete command you have in your question and get reasonable-looking output:
library(tidyr)
complete(DF, nesting(Entity, CClass), Month)
# # A tibble: 40 × 4
# Entity CClass Month Sales_volume
# <int> <chr> <int> <int>
# 1 1 Bakery 1 100
# 2 1 Bakery 2 106
# 3 1 Bakery 3 103
# 4 1 Bakery 4 NA
# 5 1 Bakery 5 135
# 6 1 Bakery 6 121
# 7 1 Bakery 7 176
# 8 1 Bakery 10 133
# 9 1 Bakery 11 100
# 10 1 Bakery 12 112
# # … with 30 more rows
# # ℹ Use `print(n = ...)` to see more rows
Some ideas: make sure you are using tidyr::complete, if you loaded another package with a complete function that might be masking the correct version. You can check conflicts() and see if complete is listed, and if so specify tidyr::complete to get the correct version. Also check names(DF) and make sure your column names are exactly what you think they are--no extra whitespace, capitalized correctly, etc. Also check the class(DF) and make sure it is data.frame or tbl_df, and maybe have a look at str(DF) to make sure the columns are appropriate classes. Since you didn't use dput() to share your data, we can't be sure what the class of the data or the columns are.
If you still have trouble, please try to find a sample of data that reproduces the problem and please use dput to share it.
I am trying to do a Friedman's test and yes my data is repeated measures but nonparametric.
The data is organized like this from the csv and used Rstudio's import dataset function so it is a table in Rstudio:
score| treatment | day
10 | 1 | 1
20 | 1 | 1
40 | 1 | 1
7 | 2 | 1
100| 2 | 1
58 | 2 | 1
98 | 3 | 1
89 | 3 | 1
40 | 3 | 1
70 | 4 | 1
10 | 4 | 1
28 | 4 | 1
86 | 5 | 1
200| 5 | 1
40 | 5 | 1
77 | 1 | 2
100| 1 | 2
90 | 1 | 2
33 | 2 | 2
15 | 2 | 2
25 | 2 | 2
23 | 3 | 2
54 | 3 | 2
67 | 3 | 2
1 | 4 | 2
2 | 4 | 2
400| 4 | 2
16 | 5 | 2
10 | 5 | 2
90 | 5 | 2
library(readr)
sample_data$treatment <- as.factor(sample_data$treatment) #setting treatment as categorical independent variable
sample_data$day <- as.factor(sample_data$day) #setting day as categorical independent variable
summary(sample_data)
#attach(sample_data) #not sure if this should be used only because according to https://www.sheffield.ac.uk/polopoly_fs/1.714578!/file/stcp-marquier-FriedmanR.pdf it says to use attach for R to use the variables directly
friedman3 <- friedman.test(y = sample_data$score, groups = sample_data$treatment, blocks = sample_data$day)
summary(friedman3)
I am interested in day and score using Friedman's.
this is the error I get:
>Error in friedman.test.default(y = sample_data$score, groups = sample_data$treatment, blocks = sample_data$day, :
not an unreplicated complete block design
Not sure what is wrong.
Prior to writing the Friedman part of the code, I only specified day and treatment as categorical using as.factor
I have two data.tables like so:
tests
id | test | score
=================
1 | 1 | 90
1 | 2 | 100
2 | 1 | 70
2 | 2 | 80
3 | 1 | 100
3 | 2 | 95
cheaters
id | test | score
=================
1 | 2 | 100
3 | 1 | 100
3 | 2 | 95
Say I now want to include a boolean column in all_scores to tell whether that particular test was cheated on, so the output would be like this:
tests
id | test | score | cheat
=========================
1 | 1 | 90 | FALSE
1 | 2 | 100 | TRUE
2 | 1 | 70 | FALSE
2 | 2 | 80 | FALSE
3 | 1 | 100 | TRUE
3 | 2 | 95 | TRUE
Is there an easy way to do this? The tables are keyed on id and test.
Create the cheat column with initial value of FALSE, then join with cheaters, and update cheat column to TRUE when there's match:
library(data.table)
setkey(setDT(tests), id, test)
setkey(setDT(cheaters), id, test)
tests[, cheat := FALSE][cheaters, cheat := TRUE]
tests
# id test score cheat
#1: 1 1 90 FALSE
#2: 1 2 100 TRUE
#3: 2 1 70 FALSE
#4: 2 2 80 FALSE
#5: 3 1 100 TRUE
#6: 3 2 95 TRUE
Or without setting the keys, use on parameter to specify the columns to join on:
setDT(tests)
setDT(cheaters)
tests[, cheat := FALSE][cheaters, cheat := TRUE, on = .(id, test)]
tests
# id test score cheat
#1: 1 1 90 FALSE
#2: 1 2 100 TRUE
#3: 2 1 70 FALSE
#4: 2 2 80 FALSE
#5: 3 1 100 TRUE
#6: 3 2 95 TRUE
I have a table like
+------+---------+---------+---------+----------+---------+
| Code | Display | Synonym | Synonym | Synonym | Synonym |
+------+---------+---------+---------+----------+---------+
| 1 | A | Cat | Dog | Lion | |
| 2 | B | Horse | Penguin | | |
| 3 | C | Donkey | Giraffe | Mongoose | Rabbit |
+------+---------+---------+---------+----------+---------+
I want to output a table like
+------+---------+----------+
| Code | Display | Synonym |
+------+---------+----------+
| 1 | A | Cat |
| 1 | A | Dog |
| 1 | A | Lion |
| 2 | B | Horse |
| 2 | B | Penguin |
| 3 | C | Donkey |
| 3 | C | Giraffe |
| 3 | C | Mongoose |
| 3 | C | Rabbit |
+------+---------+----------+
In other words, I want to pair off Code and Display with each Synonym that is presented, and each Code can have 1 to several synonyms. I've seen examples of reshape used in other contexts, but haven't been able to figure out how to apply it here.
You can use standard reshaping on a ragged array - with melt() from reshape2, you can use the na.rm argument to remove NAs as you go, otherwise you can do it afterward:
library(reshape2)
dat.m <- melt(dat, id.vars = c("Code", "Display"), value.name = "Synonym", na.rm = TRUE)
# Code Display variable Synonym
#1 1 A Synonym Cat
#2 2 B Synonym Horse
#3 3 C Synonym Donkey
#4 1 A Synonym.1 Dog
#5 2 B Synonym.1 Penguin
#6 3 C Synonym.1 Giraffe
#7 1 A Synonym.2 Lion
#9 3 C Synonym.2 Mongoose
#12 3 C Synonym.3 Rabbit
You can drop the variable column if you like:
dat.m$variable <- NULL
Here are two base R approaches.
stack
cbind(mydf[1:2], stack(lapply(mydf[-c(1:2)], as.character)))
# Code Display values ind
# 1 1 A Cat Synonym
# 2 2 B Horse Synonym
# 3 3 C Donkey Synonym
# 4 1 A Dog Synonym.1
# 5 2 B Penguin Synonym.1
# 6 3 C Giraffe Synonym.1
# 7 1 A Lion Synonym.2
# 8 2 B Synonym.2
# 9 3 C Mongoose Synonym.2
# 10 1 A Synonym.3
# 11 2 B Synonym.3
# 12 3 C Rabbit Synonym.3
reshape
Make life easier by renaming your columns first to a pattern like "Synonym_1", "Synonym_2" and so on. Actually, R likes "Synonym.1", "Synonym.2" and so on better....
A <- grep("Synonym", names(mydf))
names(mydf)[A] <- paste0("Synonym_", seq_along(A))
Now, reshape...
reshape(mydf, direction = "long", varying = A, sep = "_")
# Code Display time Synonym id
# 1.1 1 A 1 Cat 1
# 2.1 2 B 1 Horse 2
# 3.1 3 C 1 Donkey 3
# 1.2 1 A 2 Dog 1
# 2.2 2 B 2 Penguin 2
# 3.2 3 C 2 Giraffe 3
# 1.3 1 A 3 Lion 1
# 2.3 2 B 3 2
# 3.3 3 C 3 Mongoose 3
# 1.4 1 A 4 1
# 2.4 2 B 4 2
# 3.4 3 C 4 Rabbit 3
I figured out a maybe indirect way to do this shortly after asking the question:
allergies_output <- reshape(allergies_input,varying=list(grep('Synonym',names(allergies_input),value=TRUE)),direction='long',idvar=c('Code','Display'),v.names='Synonym',names(allergies_input))
This gives some wonky results, but nothing that can't be fixed by dropping some column names.
I have a series of numbers in two columns, with the titles "a" and "b".
I want to get R to change the values in column "b" if the difference between a value in column "a" is greater than 10 from its neighboring cells.
For example:
a | b
-----------
1 | 1
2 | 1
3 | 1
4 | 1
21 | 1
22 | 1
23 | 1
24 | 1
... | ...
Then I would like R to change the values in column "b" to
a | b
-----------
1 | 1
2 | 1
3 | 1
4 | 0
21 | 0
22 | 1
23 | 1
24 | 1
... | ...
Because the values 4 and 21 in the a-column are greater than 10 from each other.
Any help would be greatly appreciated.
df <- data.frame(a = c(1:4, 21:24), b = 1)
# check whether differences are greater than 10
diffs <- diff(df$a) > 10
# create `b`
df$b <- as.integer(!(c(FALSE, diffs) | c(diffs, FALSE)))
The result:
a b
1 1 1
2 2 1
3 3 1
4 4 0
5 21 0
6 22 1
7 23 1
8 24 1
Some alternative.
df <- data.frame(a = c(1:4, 21:24), b = 1L)
local({
w10 <- with(df, which(diff(a) > 10)))
df$b[c(w10, w10+1)] <<- 0L
})