Replay tools charge by session. Legal teams flinch at full capture. Sampling is reasonable. What is not reasonable is quoting “we watched 40 sessions and onboarding is fine” when the sample excluded every user who hit the known crash on older Android builds.
In Replay as Evidence we treat a recording as a citation. A citation needs a sampling note: who was eligible, what was excluded, whether errors were oversampled, whether consenting users differ from the rest of the base. Without that note, the clip is theatre.
A pattern that keeps showing up
Teams oversample “angry” sessions from a rage-tap detector and then conclude the UI is broadly confusing. Rage-tap detectors are not a census. They are a magnet. The atelier response is to pair them with a boring random sample of the same surface, even if that second sample is tiny.
The other failure is under-sampling logged-out users because the identity join is messy. That quietly makes replay a study of people who already have accounts. Fine if you say so. Unfine if the slide title is “new user experience.”
We still will not help you turn replay into a warehouse of faces. If your brief is full capture “just in case,” the programme will refuse it in the same week we teach the citation habit.