cumulative distinct count in hive - count

Question

Here is some sample data from table daily_user. Each row represents an active user on a specific day, the revenue is based on the money generated by the user on that day. The earliest date in this table is 1/1.
date user_id group revenue
1/1 1 a 1
1/1 2 b 0
1/1 3 a 0
1/2 2 b 10
1/2 3 a 0
1/3 3 a 1
The output I want (Basically, each row tells me for each group, from 1/1 to each observation date, how many users have ever paid. For example, the last row means from 1/1-1/3, for group b, in total we have 1 user who paid us):
end_date group # users who ever paid
1/1 a 1
1/1 b 0
1/2 a 1
1/2 b 1
1/3 a 2
1/3 b 1
There seems to be some UDFs to do cumulative sum, but I am not sure if there is any cumulative distinct count function that I can leverage here. Is there anyway to struct a hive query to implement this?

Answer 1

I think the solution is to actually 'collect_set' the users ( collect unique values) and take the size of the array, for small numbers of user ( ie. which would fit in memory)
SELECT size( collect_set( user_id ) ) as uniques
end_date, group
FROM daily_user
GROUP BY end_date, group;
For large numbers of uniques, you'll need a probabilistic data structure, like sketch sets or hyperloglogs, available as UDF's from the Brickhouse library ( http://github.com/klout/brickhouse ). This will give you an estimate which is close, but not the exact number of uniques
SELECT estimated_reach( sketch_set( user_id )) as uniques_est,
end_date, group
FROM daily_user
GROUP BY end_date, group;
You can also merge these, so that can merge pre-calculated collections/sketches from previous days. Something like :
SELECT size(combine_unique( unique_set ) ) as uniques,
group
FROM daily_uniques
WHERE end_date > date_add( today, -30 )
GROUP BY group;
or
SELECT estimated_reach( union_sketch( unique_sketch) ) as uniques,
group
FROM daily_uniques
WHERE end_date > date_add( today, -30 )
GROUP BY group;

Answer 2

The function if(revenue=0,1,0) will have value 1 if the revenue is 0, and will have value 0 otherwise. Summing this function will give you the total number of people who had revenue of 0:
select
date as end_date,
group,
sum(if(revenue=0,1,0)) as number_of_users_who_never_paid
from
daily_user
group by
date,
group

Answer 3

The simplest way of doing this, without writing a custom UDF, would be to do some sort of cartesian join:
select
date as end_date,
group,
sum(if(mon.user_id is not null AND mon.date <= du.date,1,0)) as cumulative_spenders
from
daily_user du
LEFT OUTER JOIN
(
select
distinct
user_id,
date,
group
from
daily_user
where
revenue > 0
) mon
ON
(du.user_id=mon.user_id and du.group=mon.group)
group by
date,
group
This will generate a row per spending transaction per entry in the original table, then aggregate from there.