R Group by exactly inverted columns for large data - r

I'm trying to group my data by a very specific condition. Consider below data.frame:
from <- c("a", "b", "a", "b")
to <- c("b", "a", "b", "a")
give <- c("x", "y", "y", "x")
take <- c("y", "x", "x", "y")
amount <- c(1, 2, 3, 4)
df <- data.frame(from, to, give, take, amount)
which creates something like:
| from | to | give | take | amount
---------------------------------------
1 | a | b | x | y | 1
2 | b | a | y | x | 2
3 | a | b | y | x | 3
4 | b | a | x | y | 4
To provide some background: consider some user in the 'from' column giving something (in column 'give') to the user in 'to' column and taking something in return (in column 'take'). As you might see, rows 1 & 2 are the same in that way, because they describe the same scenario, just form another perspective. Therefore, I want these to belong to the same group. (You could also consider them as duplicates, which involves the same task, i.e. identifying them as similar.) The same holds for rows 3 & 4. The amount is some value to be summed up per group, to make the example clear.
My desired result for grouping them is as follows.
| user1 | user2 | given_by_user1 | taken_by_user1 | amount
-----------------------------------------------------------
| a | b | x | y | 3 # contains former rows 1&2
| a | b | y | x | 7 # contains former rows 3&4
Note that both from&to and give&take need to by inverted, i.e. taking the values from two columns, sorting their values and considering them equal on that basis is not what I need. This would lead to all four rows in above example being considered equal. That kind of solution was proposed in similar posts, e.g.:
Remove duplicates where values are swapped across 2 columns in R
I've read many similar solutions and found one which actually does the trick:
match two columns with two other columns
However, the proposed solution creates an outer product of two columns, which is not feasible in my case, because my data has millions of rows and at least thousands of unique values within each column.
(Any solution that either groups the rows directly, or gets the indices of rows belonging to the same group would be great!)
Many thanks for any suggestions!

Related

How to make a dataset with rows representing the no. of clusters while columns are two variables with different value for clusters

Im quiet confused.
I have 50 clusters each with a different size, and I have two variables "Year" and "Income level".
The data set I have right now has 10,000 rows where each row represents a single individual.
What I want to do is to form a new dataset from this dataframe where each row represents the number of clusters (50) and the columns be the two variables + the cluster variable. The problem is these two variables (that we call the study level covariates) do not have a unique value for clusters.
How would I put them in one cell for each cluster then?
X1<-c(1,1,1,2,2,2,2,2,3,3,4,4,4,4,4,4) #Clusters
X2<c(1,2,3,1,1,1,1,1,1,2,3,3,1,1,2,2,2) #Covariate1
X3<-c(1991,2001,2002,1998,2014,2015,1990,
2002,2004,2006,2006,2006,2005,2003,2003,2000) #Covariate2
data<-data.frame(X1,X2,X3)
My desire output should be something like this:
|Clusters|Covariet1|Covariate2|
|--------|---------|----------|
|1 | ? |? |
|2 | ? |? |
|3 | ? |? |
|4 | ? |? |
Meanening that instead of a data frame with 16 rows, a dataframe with 4 rows

Here is how to aggreagate the data using the average of the covariate per cluster:
df <- data.frame(X1 = c(1,1,1,2,2,2,2,2,3,3,4,4,4,4,4,4),
X2 = c(1,2,3,1,1,1,1,1,1,2,3,3,1,1,2,2),
X3 = c(1991,2001,2002,1998,2014,2015,1990,2002,2004,2006,2006,2006,2005,2003,2003,2000)
)
library(tidyverse)
df %>% group_by(X1) %>% summarise(mean_cov1 = mean(X2))
# A tibble: 4 x 2
X1 mean_cov1
* <dbl> <dbl>
1 1 2
2 2 1
3 3 1.5
4 4 2
For the case you are working on, you have to decide what the most relevant aggreagation is. You can probably also create multiple at once.

How can i apply conditional formatting on a column by column basis in R on my data using ggplot?

I have a datframe like the following:
group | amount_food | amount_finance | amount_clothes
A | 30 | 40 | 50
B | 34 | 43 | 53
C | 50 | 86 | 90
I would like to colour the contents of the cells depending on the value (a gradient of sorts where e.g. red would indicate higher and blue would indicate lower values etc). Similar to conditional formatting in excel. Ideally would like this done on a column by column basis, s i know which group has the highest amount_food etc.
How can i achieve this in R?
df <- read.csv("shopspend.csv")
new to R so any pointers helpful.

Sparklyr : separate rows on 2 columns

I am using sparklyr for a project. I have a Spark Dataframe with lists in some of the columns and I'd like to separate them into multiple rows, i.e. have one value in each row, exactly like separate_rows does in dplyr.
So basically my dataframe is like this
| x | y
1| [a,b] | [c,d]
And I'd like to have something like this in the end :
| x | y
1| a | c
2| b | d
Like suggested in this post, explode is a good start, but it can do the job for only one column at once ; and if I use it twice, I will end up with 4 rows here instead of the 2 I want. In this very simple example, I could manage my way to keep only the rows that I want, but things can get a bit messier if there are more than two elements in the lists...
Something I thought about would be to do :
Merge the columns x and y into a single column which would contain [[a,c] , [b,d]]
Then use explode to have [a,c] and then [b,d]
Then explode but in columns (rather that in rows).
Only I don't know how to do 1) and 3).
Thank you for the help !
Here is a reproducible example obtained with collect and dput :
structure(list(ref_amount = list(list(967.66, 1592.56), list(
967.66, 1592.56)), ref_theta = list(list(5.26977034898459,
5.16119062369122), list(5.26977034898459, 5.16119062369122))), .Names = c("ref_amount",
"ref_theta"), row.names = c(NA, -2L), class = c("tbl_df", "tbl",
"data.frame"))

Calculate autocorrelation in panel data?

I have a large panel data set in the form:
ID | Time| X-VALUE
---| ----|-----
1 | 1 |x
1 | 2 |x
1 | 3 |x
2 | 1 |x
2 | 2 |x
2 | 3 |x
3 | 1 |x
3 | 2 |x
3 | 3 |x
. | . |.
. | . |.
More specifically, I have dataset of a large set of individual stock returns over a period of 30 years. I would like to calculate the "stock-specific" first (lag 1) autocorrelation in returns for all stocks individually.
I suspect that by applying the code: acf(pdata$return, lag.max = 1, plot = FALSE) I'll only get som kind of "average" autocorrelation value, is that correct?
Thank you

You can split the data frame and do the acf on each subset. There are tons of ways to do this in R. For example
by(pdata$return, pdata$ID, function(i) { acf(i, lag.max = 1, plot = FALSE) })
You may need to change variable and data frame names to match your own data.

This is not exactly what was requested, but a real autocorrelation function for panel data in R is collapse::psacf, it works by first standardizing data in each group, and then computing the autocovariance on the group-standardized panel-series using proper panel-lagging. Implementation is in C++ and very fast.

Creating a dictionary for tabular data in Julia

I have a tabular data like:
+---+----+----+
| | a | b |
+---+----+----+
| P | 1 | 2 |
| Q | 10 | 20 |
+---+----+----+
and I want to represent this using a Dict.
With the column and row names:
x = ["a", "b"]
y = ["P", "Q"]
and data
data = [ 1 2 ;
10 20 ]
how may I create a dictionary object d, so that d["a", "P"] = 1 and so on? Is there a way like
d = Dict(zip(x,y,data))
?

Your code works with a minor change to use Iterators.product:
d = Dict(zip(Iterators.product(x, y), data.'))
To do this you need to add a line using Iterators to your project, and might need to Pkg.add("Iterators"). Because Julia matrices are column-major (elements are stored in order within columns, and columns are stored in order within the matrix), we needed to transpose the data matrix using the transpose operator .'.
This is a literal answer to your question. I don't recommend doing that. If you have tabular data, it's probably better to use a DataFrame. These are not two dimensional (rows have no names) but that can be fixed by adding an additional column, and using select.

Resources