Transition probabilities for entire table - r

Question

I have df with the following structure:
sid step1 step2 step3 . . . . . step30
The sid is an id and the steps are steps through a webpage where
sids with a minimum of two steps
sids with a maximum of thirty steps
no duplicate sequential pages (ie page refreshes)
the steps are all string object types
I want to essentially create a total transition probability where for every unique page— I get a table/matrix which has a transition probability for every single possible page.
I have around ~3k unique pages so I don't know if this will be computationally feasible.
I would be okay with also passing a few pages as an argument for the matrix, so its not a 3000x3000 matrix and maybe a 1x3000 or 5x3000. In fact, I would prefer to start with this and scale up until it crashes lol.

Answer 1

Starting with the concept
To build a transition matrix, it is often easy to first build a matrix of counts. The counts can then be divided to produce transition probabilities.
To produce something like:
| to_site_A | to_site_B | ...
----------- +-----------+-----------+-----
from_site_A |
from_site_B | counts
from_site_C |
...
It might be simpler to first produce:
from | to | count
-------+--------+-------
site_A | site_B |
site_A | site_C |
...
This is the same information, just arranged differently.
And to do this, it is probably easier if you rearrange your current data into a structure like this:
from | to
-------+-------
site_A | site_B
site_A | site_C
...
So
Step 1: get data into long-thin structure of transitions
Step 2: count all pairwise transitions
Step 3: pivot or rearrange counts into square matrix
Step 1, rearrange data to long thin
You probably want something like this:
df_from_1_to_2 = df %>%
select(from = step1, to = step2) %>%
filter(!is.na(to))
df_from_2_to_3 = df %>%
select(from = step2, to = step3) %>%
filter(!is.na(to))
...
df_from_29_to_30 = df %>%
select(from = step29, to = step30) %>%
filter(!is.na(to))
long_list = rbind(df_from_1_to_2,
df_from_2_to_3,
...
df_from_29_to_30)
No this is not the most efficient way to approach this (by code or memory management) but we shall focus on the approach.
Step 2, count all pairwise transitions
This is now straightforward:
pairwise_count = long_list %>%
group_by(from, to) %>%
summarise(count = n())
Step 3, pivot or rearrange counts into square matrix
This step is just changing how the data is presented, and may not even be necessary depending on your application.
For rearranging this type of data, I suggest pivot_wider from the tidyr package:
count_matrix = pivot_wider(
data = pairwise_count,
names_from = to,
names_prefix = "to",
names_sep = "_",
values_from = count,
)
Edit: getting probabilities instead of counts
There are multiple points at which you could swap from counts to probabilities, one place to do it would be during step 2:
pairwise_count = long_list %>%
group_by(from, to) %>%
summarise(count = n())
pairwise_prob = pairwise_count %>%
group_by(from) %>%
mutate(from_count = sum(count)) %>%
mutate(prob = count / from_count) %>%
select(from, to, prob)
You can then use pairwise_prob in step 3 rather than pairwise_count.