In this post, I will share how to integrate opentelemetry tracing with GCP logging.
Why is this useful?#
In a highly-concurrent environments, your service is receiving a lot of requests. Each request will generate logs related to that request, and these logs are usually mixed together in GCP logging explorer.
If you want to examine only the logs related to a single request, it will be difficult without the trace id.
The trace integration means that we will add trace and span id info to each log entry, so the log info for a single trace can be automatically linked together.
How to do it#
The naive way is to add trace id to each log message, the python logging cookbook provides one such example. However, adding trace id to the log message is a naive idea and does not work well:
- It is not easy for filtering: to filter logs for a single request, you need to manually write the logging query
- It will not work well with the GCP trace explorer together
The recommended way is to use structured logging, and populate the trace and spanId field with the trace and span id.
See the LogEntry for the format of trace value.
If you are not using the GCP logging client, you need to do this manually: extracting trace and span id from current oTel context,
and populate the relevant fields.
If you are using the GCP logging client together with the standard python logging package (see [here][https://jdhao.github.io/2024/09/20/python_logging_in_gcp/] for the setup), the GCP logging client already integrates with OpenTelemetry. It will automatically extract the trace id and span id from the current active opentelemtry context. The logging client will collect the info, create the LogEntry and send it to GCP logging. You do not have to set this up manually.
If everything is set up correctly, in the GCP logging explorer, you will see the little trace icon for each log. You can click the trace icon to check trace details or filter logs matching a trace. Both of these are incredibly useful to gain insights about the application and debug production issues.