[EPIC] Accurate metrics when client-side sampling is on

[danielkhan]

Problem

Our customers frequently use client-side sampling to control costs, but we also don’t feel confident suggesting a 100% client-side sampling rate on all platforms because of performance concerns. At the same time, we tend to assume a 100% client-side sampling rate when we talk about the completeness of metrics.

Accurate metrics must be a key goal regardless of client-side or server-side sampling rates.

User Stories

• As a user, I want my metrics to be accurate, even if I sample on the client.

Possible Solutions

• Improve our SDKs to sample 100% in all cases (does not solve cost control by customers).
• Always extract and aggregate complete metrics in our SDK regardless of the sampling rate (will create cost on our end, complicated to maintain on all SDKs)
• Report the client-side sample rate with metrics sent to the backend and extrapolate them based on this rate.

Clarifications

• While it's a technical requirement to strive for, we will never be able to assume that any given client samples with 100% and as such, tuning our SDKs for this, won't solve the problem completely.
• Whichever approach we choose, this will increase COGS for us.
• We need to re-validate all SDKs and implement better backpressure management to guarantee even if customers set 100% sampling, especially in a very high throughput environment, we never negatively impact the customers infrastructure