Sample Traces from OTel

Abstract

Production services can emit more traces than a team needs to retain. Keeping every trace creates unnecessary volume, but improper sampling can hide failures and slow requests.

In this course, you will configure tail sampling in an OpenTelemetry (OTel) Collector. You'll build policies that retain high-value traces, and verify the resulting trace mix in Datadog.

Learning Objectives

By the end of this course, you will be able to:

Explain the benefits of calculating trace metrics before sampling.
Configure a 5% probabilistic sampling baseline and evaluate its limitations for investigations.
Combine policies that retain errors and slow traces, sample routine traffic, and drop successful health checks.
Verify trace retention and sampling policy decisions in Datadog.

Primary Audience

This intermediate course is for SREs, platform engineers, and developers who operate OTel-instrumented services. This course is most useful for those looking to reduce large, noisy trace volumes.

Prerequisites

Before taking this course, you should understand:

OTel signals and basic Collector components, including receivers, processors, exporters, and pipelines. These concepts are covered in Understanding OpenTelemetry.
How to use the Learning Environment.

Technical Requirements

In order to complete the course, you will need:

Google Chrome or Firefox
Third-party cookies must be enabled to access labs

Course Navigation

At the bottom of each lesson, click MARK LESSON COMPLETE AND CONTINUE button so that you are marked complete for each lesson and can receive the certificate at the end of the course.

Course Enrollment Period

Please note that your enrollment in this course ends after 30 days. You can re-enroll at any time and pick up where you left off.

Learning Center