You also often need to scan the whole data set in order to sample properly. Supp...

yummyfajitas on Nov 9, 2012 | parent | context | favorite | on: How I came to love big data

You also often need to scan the whole data set in order to sample properly.

Suppose you are interested in user behavior. You want sample(group_by(user_ident, all_user_events)), not group_by(user_ident, sample(all_user_events)). This involves running a group_by on the full data set.

tel on Nov 9, 2012 [–]

Or counting the number of events for each user as they come in.