How to compare changes in data profiles over time? - r

Question

Say , I have datasets
df1=
structure(list(date = c("17.02.2021", "04.11.2020", "14.11.2020",
"24.11.2020", "29.11.2020", "04.12.2020", "09.12.2020"), x1 = c(0L,
0L, 7L, 0L, 0L, 0L, 0L), x2 = c(674L, 632L, 1036L, 656L, 736L,
762L, 698L), x3 = c(698L, 712L, 1140L, 704L, 784L, 786L, 722L
), x4 = c(522L, 472L, 988L, 464L, 608L, 578L, 514L), x5 = c(2408L,
3256L, 2840L, 2840L, 2888L, 2632L, 2648L), x6 = c(1952L, 2336L,
2480L, 2208L, 2208L, 2144L, 2016L), x7 = c(1056L, 1120L, 1504L,
1056L, 1184L, 1184L, 1120L), x8 = c(1984L, 2464L, 2400L, 2144L,
2208L, 2144L, 2080L), x9 = c(2336L, 2976L, 2784L, 2464L, 2784L,
2528L, 2400L), x10 = c(2528L, 3232L, 3104L, 2848L, 2912L, 2592L,
2656L), x11 = c(1248L, 1312L, 1504L, 1312L, 1312L, 1312L, 1248L
)), class = "data.frame", row.names = c(NA, -7L))
each row it is date. for the first day data profile here
The second day has data profiles
and so on.
Here reference dataset
df2=structure(list(date = c("06.11.2019", "01.12.2019", "25.01.2020",
"04.02.2020", "09.02.2020", "14.02.2020"), x1 = c(12L, 0L, 1L,
6L, 23L, 1L), x2 = c(1272L, 1046L, 688L, 572L, 592L, 328L), x3 = c(1032L,
974L, 736L, 780L, 800L, 568L), x4 = c(792L, 862L, 496L, 476L,
592L, 296L), x5 = c(2232L, 1496L, 1784L, 2792L, 3064L, 3544L),
x6 = c(2976L, 1904L, 1632L, 1760L, 1376L, 1440L), x7 = c(1568L,
1248L, 1008L, 1120L, 992L, 800L), x8 = c(1888L, 1376L, 1632L,
2400L, 2464L, 2720L), x9 = c(2080L, 1504L, 1760L, 2848L,
2912L, 3296L), x10 = c(2400L, 1552L, 1824L, 2848L, 2928L,
3360L), x11 = c(2400L, 1504L, 1120L, 1040L, 784L, 736L)), class = "data.frame", row.names = c(NA,
-6L))
Is there a way or method that would compare the profile of each row of data in df1 with the reference dataset df2, if the profile is similar, then 1 otherwise 0
The date in both dataset can be different, the main problem is detect is profiles are similar or not.
My desired output. The Peter's code is good, but is it possible calculate The difference between profiles by variables for example

Answer 1

This code allows you to visually compare the reference and df1 profiles. As you can see that none of the profiles match exactly. Some profiles are similar, but without a definition of "similar" as pointed out by #user2974951 it's difficult to move this closer to an answer.
library(dplyr)
library(tidyr)
library(ggplot2)
# restructure the data to allow comparison between the datasets
df <-
expand.grid("date_ref" = df2$date, "date_df1" = df1$date) %>%
left_join(df2, by = c("date_ref" = "date")) %>%
left_join(df1, by = c("date_df1" = "date")) %>%
pivot_longer(starts_with("x"), names_to = c("var", "df"), names_sep = "\\.") %>%
mutate(df = if_else(df == "x", "ref", "df1"),
var = factor(var, paste0("x", 1:11)))
# now you can plot the data to compare profiles; had to add some formatting to make the graph readable.
ggplot(df, aes(var, value, group = df, colour = df))+
geom_line()+
facet_grid(date_ref~date_df1)+
labs(colour = "Dataset")+
theme_classic()+
theme(legend.position = "bottom",
axis.text.x = element_text(size = 6, angle = 90),
axis.text.y = element_text(size = 6),
strip.text = element_text(size = 6))
Created on 2021-04-07 by the reprex package (v1.0.0)

Answer 2

What you need to define first is what criteria of similarity you want to use and what your threshold level of similarity is (how similar the datasets need to be to be considered equivalent). Also the important factor is the nature of your data. For example whether you consider your x1..x11 to be independent or just different samples of the same set.
Depending on the answers it can be anything from comparing each df1[i,2:12] to df2[i,2:12] exactly (if they are just duplicates or not) to comparing both of them to NA and checking if they are both NA or both a known value. Something in between would be checking if the differences of each parameter for each line of the datasets are not greater then 0.05 of the minimal value for example and marking the line equivalent if all the parameters are OK or using something like Pearson's correlation coefficient (cor(x,y) function has it enabled by default) for each line and comparing its value to 0.5 for example (both 0.05 and 0.5 are just arbitrary numbers of course and they probably need to be adjusted somewhat). Or maybe the amount of matching points (compared exactly as integers or just similar to some degree) is a better indication for you. There are also known standard tests for sample group dissimilarities, time series dissimilarities, or other statistical hypothesis. Many of them are available in R from bundled packages and if you fancy something else then it is most likely already available in one of the extra packages you can easily download and install.