↓ Skip to main content
jdhao’s digital space
  1. Posts/

Opentelemetry Trace Sampling in Action

·252 words·2 mins·
Python OpenTelemetry
Table of Contents

In this post, I talk briefly about trace sampling in Opentelemetry.

Why do we need sampling
#

For distributed system with high load, it is not practical to trace every request: this would introduce latency and also incur a lot of trace-related cost. Usually only a proportion of all requests are traced. This is called sampling in OpenTelemetry, which includes both head and tail sampling.

If your system use head sampling, usually the service at the forefront will decide whether to sample a request.

For tail sampling, a collector will collect the spans for a trace and then decicde whether to keep the trace or discard it. This is a more complex sampling strategy, which I will not elaborate here.

How to do head sampling inside your service
#

In head sampling, the sampling flag is propagated from upstream services through the HTTP traceparent header. Inside your service, you can then use ParentBased sampler for traces sampling. For example:

  • Traceparent: 00-4d502c03384abc99124806be91afdfe7-a7e7f36238457189-00, sampled: ❌
  • Traceparent: 00-f5793ff3bab328e635d9484d4053492d-17737305481aefd8-01, sampled: ✅︎

If you want to control sampling independent of the upstream service, you can use ratio-based sampler and set up a sampling rate.

from opentelemetry.sdk.resources import Resource
from opentelemetry.sdk.trace.sampling import ALWAYS_ON, ParentBased, Sampler, TraceIdRatioBased


resource = Resource.create(
    attributes={
        "service.name": "my-demo-service",
    },
)

use_parent_sampling = True
sampling_rate = 0.1

if use_parent_sampling:
    sampler = ParentBased(ALWAYS_ON)
else:
    sampler = TraceIdRatioBased(sampling_rate)

# create a trace provider
provider = TracerProvider(
    resource=resource,
    sampler=sampler,
)

Check the above sampling module for more details about different samplers.

references
#

Related