Configure MCP gateway OpenTelemetry from observability.otlp and actions/setup trace IDs#24697
Open
Configure MCP gateway OpenTelemetry from observability.otlp and actions/setup trace IDs#24697
Conversation
…ns/setup trace IDs When observability.otlp.endpoint is set in the workflow frontmatter, the MCP gateway config now includes an opentelemetry section (spec §4.1.3.6) with: - endpoint from OTEL_EXPORTER_OTLP_ENDPOINT env var - headers parsed from OTEL_EXPORTER_OTLP_HEADERS (comma-separated key=value) - traceId from GITHUB_AW_OTEL_TRACE_ID (set by actions/setup step) - spanId from GITHUB_AW_OTEL_PARENT_SPAN_ID (set by actions/setup step) Agent-Logs-Url: https://github.com/github/gh-aw/sessions/7b2ea1c3-184d-4a30-9426-2ac47eceaf54 Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>
Copilot created this pull request from a session on behalf of
pelikhan
April 5, 2026 09:27
View session
pelikhan
reviewed
Apr 5, 2026
pkg/workflow/mcp_gateway_config.go
Outdated
| // Detect OTLP configuration from observability.otlp frontmatter. | ||
| // When configured, the gateway will emit distributed traces to the OTLP collector | ||
| // using the trace/span IDs from the actions/setup step for cross-job correlation. | ||
| otlpEndpoint, _ := extractOTLPConfigFromRaw(workflowData.RawFrontmatter) |
Collaborator
There was a problem hiding this comment.
@copilot must operate on full resolved front matter with imports
Contributor
Author
There was a problem hiding this comment.
Fixed in b8188e6. buildMCPGatewayConfig now uses workflowData.OTLPEndpoint, which is the fully resolved endpoint set by injectOTLPConfig after merging observability from both the top-level frontmatter and any imported shared workflows (e.g. shared/observability-otlp.md). The observability field is now extracted from imports using a first-wins strategy (same pattern as activationGitHubToken), merged into RawFrontmatter before injectOTLPConfig runs, making it the single source of truth for all downstream code.
pkg/workflow/mcp_setup_generator.go
Outdated
| // - traceId: from GITHUB_AW_OTEL_TRACE_ID (set by actions/setup via GITHUB_ENV) | ||
| // - spanId: from GITHUB_AW_OTEL_PARENT_SPAN_ID (set by actions/setup via GITHUB_ENV) | ||
| // The variable is consumed by ${GH_AW_GATEWAY_OTEL} in the MCP config heredoc. | ||
| otlpEndpoint, _ := extractOTLPConfigFromRaw(workflowData.RawFrontmatter) |
Collaborator
There was a problem hiding this comment.
@copilot must operate on fully resolved front matter with imports
Contributor
Author
There was a problem hiding this comment.
Fixed in b8188e6. mcp_setup_generator.go now checks workflowData.OTLPEndpoint != "" instead of re-reading raw frontmatter. OTLPEndpoint is the fully resolved endpoint set by injectOTLPConfig, which now also picks up observability defined in imported shared workflows via the new MergedObservability field in ImportsResult.
- Add observability extraction from imported shared workflows in the imports system (first-wins strategy, same as activationGitHubToken) - Before calling injectOTLPConfig, merge imported observability into RawFrontmatter when the top-level frontmatter has no observability section - Store the resolved OTLP endpoint in WorkflowData.OTLPEndpoint (set by injectOTLPConfig as the single source of truth including imports) - buildMCPGatewayConfig and mcp_setup_generator.go now use workflowData.OTLPEndpoint instead of independently re-reading raw frontmatter Agent-Logs-Url: https://github.com/github/gh-aw/sessions/257796c2-5f84-437a-957f-65ea15d03d61 Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>
Contributor
Author
|
Just as a heads up, I was blocked by some firewall rules while working on your feedback. Expand below for details. Warning Firewall rules blocked me from connecting to one or more addresses (expand for details)I tried to connect to the following addresses, but was blocked by firewall rules:
If you need me to access, download, or install something from one of these locations, you can either:
|
Collaborator
|
@copilot review mcp gateway specification for proper implementation. There is no change needed with awf, it's all in the mcp config file |
…riable
Per spec §4.1.3.6, the opentelemetry configuration belongs in the MCP
config file using ${VARIABLE_NAME} expressions (the same pattern used
for domain, apiKey, etc.). Remove the shell-built GH_AW_GATEWAY_OTEL
variable and instead render the opentelemetry object statically with
${OTEL_EXPORTER_OTLP_ENDPOINT}, ${GITHUB_AW_OTEL_TRACE_ID}, and
${GITHUB_AW_OTEL_PARENT_SPAN_ID} references. The gateway expands these
at config-load time from env vars passed via -e flags.
Agent-Logs-Url: https://github.com/github/gh-aw/sessions/80f8d9fa-8b38-4a3e-8a2c-e49ef408b516
Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>
Contributor
Author
Done in e4529e0. The "opentelemetry": {
"endpoint": "${OTEL_EXPORTER_OTLP_ENDPOINT}",
"traceId": "${GITHUB_AW_OTEL_TRACE_ID}",
"spanId": "${GITHUB_AW_OTEL_PARENT_SPAN_ID}"
}The shell code that built Warning Firewall rules blocked me from connecting to one or more addresses (expand for details)I tried to connect to the following addresses, but was blocked by firewall rules:
If you need me to access, download, or install something from one of these locations, you can either:
|
Collaborator
|
@copilot do not use env variable. Write the values in tbe config file. |
Endpoint and headers are available at compile time from the frontmatter
so write them as literal values in the opentelemetry config block. Only
traceId and spanId remain as ${VARIABLE_NAME} expressions since they are
set at runtime via GITHUB_ENV by actions/setup and cannot be known at
compile time. Remove OTEL_EXPORTER_OTLP_ENDPOINT and
OTEL_EXPORTER_OTLP_HEADERS from the gateway container -e flags since the
gateway reads those values directly from its config file now.
Agent-Logs-Url: https://github.com/github/gh-aw/sessions/0ecb18c0-a493-4c1d-b078-4c19df61dd9b
Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>
Contributor
Author
Done in 321d01f. The endpoint and parsed headers are now written as literal values directly in the config. Only "opentelemetry": {
"endpoint": "https://un5kxfz9zumx6nj4wv9ya7zq.julianrbryant.com:4318/v1/traces",
"headers": {"api-key": "myApiKey"},
"traceId": "${GITHUB_AW_OTEL_TRACE_ID}",
"spanId": "${GITHUB_AW_OTEL_PARENT_SPAN_ID}"
}
Warning Firewall rules blocked me from connecting to one or more addresses (expand for details)I tried to connect to the following addresses, but was blocked by firewall rules:
If you need me to access, download, or install something from one of these locations, you can either:
|
pelikhan
reviewed
Apr 5, 2026
| "payloadDir": "${MCP_GATEWAY_PAYLOAD_DIR}" | ||
| "payloadDir": "${MCP_GATEWAY_PAYLOAD_DIR}", | ||
| "opentelemetry": { | ||
| "endpoint": "${{ secrets.GH_AW_OTEL_ENDPOINT }}", |
Contributor
Author
There was a problem hiding this comment.
Done in eca8a58. Headers are now written as a raw string literal directly in the config (same treatment as endpoint), so ${{ secrets.GH_AW_OTEL_HEADERS }} is preserved and expanded by GitHub Actions at runtime:
"opentelemetry": {
"endpoint": "${{ secrets.GH_AW_OTEL_ENDPOINT }}",
"headers": "${{ secrets.GH_AW_OTEL_HEADERS }}",
"traceId": "${GITHUB_AW_OTEL_TRACE_ID}",
"spanId": "${GITHUB_AW_OTEL_PARENT_SPAN_ID}"
}Switch OTLPHeaders from map[string]string to string so that GitHub
Actions expressions (${{ secrets.X }}) are preserved and written
verbatim in the config just like the endpoint. Remove the now-unused
parseOTLPHeaders helper and its tests.
Agent-Logs-Url: https://github.com/github/gh-aw/sessions/3de8b920-c862-4a7c-982d-ea8bbd6d9a51
Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>
Contributor
Author
|
Just as a heads up, I was blocked by some firewall rules while working on your feedback. Expand below for details. Warning Firewall rules blocked me from connecting to one or more addresses (expand for details)I tried to connect to the following addresses, but was blocked by firewall rules:
If you need me to access, download, or install something from one of these locations, you can either:
|
This comment has been minimized.
This comment has been minimized.
Contributor
|
🎉 Great work on this PR! The OTLP tracing integration for the MCP gateway is a well-scoped, high-quality contribution. A few things that stand out:
This looks ready for maintainer review. 👀
|
Collaborator
|
@lpcox more lenient headers parsing |
Collaborator
|
@copilot merge main and recompile |
…ntelemetry-mcp-gateway # Conflicts: # pkg/parser/import_field_extractor.go # pkg/parser/import_processor.go # pkg/workflow/compiler_orchestrator_workflow.go Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>
Contributor
Author
Done in 38071d3. Merged Warning Firewall rules blocked me from connecting to one or more addresses (expand for details)I tried to connect to the following addresses, but was blocked by firewall rules:
If you need me to access, download, or install something from one of these locations, you can either:
|
Contributor
|
🎬 THE END — Smoke Claude MISSION: ACCOMPLISHED! The hero saves the day! ✨ |
Contributor
|
📰 BREAKING: Smoke Copilot is now investigating this pull request. Sources say the story is developing... |
Contributor
|
✅ All tools validated successfully! Agent Container Smoke Test confirms agent container is ready. |
Contributor
|
🌑 The shadows whisper... Smoke Codex failed. The oracle requires further meditation... |
Contributor
Agent Container Tool Check
Result: 12/12 tools available ✅ Overall Status: PASS
|
Contributor
|
Smoke Test PR #24697 Results
Overall: PASS (Playwright has known container permission issue) Author:
|
Contributor
There was a problem hiding this comment.
Smoke test PR review: This PR correctly propagates OTLP configuration through WorkflowData.OTLPEndpoint, using it as the single resolved source of truth rather than repeatedly parsing raw frontmatter. The headers handling via bash variable expansion is functional but asymmetric with the endpoint approach — minor documentation opportunity. Overall the approach is sound.
📰 BREAKING: Report filed by Smoke Copilot · ● 1.8M
| SecretMasking *SecretMaskingConfig // secret masking configuration | ||
| ParsedFrontmatter *FrontmatterConfig // cached parsed frontmatter configuration (for performance optimization) | ||
| RawFrontmatter map[string]any // raw parsed frontmatter map (for passing to hash functions without re-parsing) | ||
| OTLPEndpoint string // resolved OTLP endpoint (from observability.otlp.endpoint, including imports; set by injectOTLPConfig) |
Contributor
There was a problem hiding this comment.
✅ Good addition — OTLPEndpoint as a resolved field on WorkflowData is the right approach. This avoids repeated raw frontmatter lookups downstream. The comment is clear and helpful. One minor note: consider also adding OTLPHeaders alongside OTLPEndpoint so both resolved values travel together through the pipeline.
| if options.GatewayConfig.KeepaliveInterval != 0 { | ||
| fmt.Fprintf(&configBuilder, ",\n \"keepaliveInterval\": %d", options.GatewayConfig.KeepaliveInterval) | ||
| } | ||
| // When OTLP tracing is configured, add the opentelemetry section directly to the |
Contributor
There was a problem hiding this comment.
The OTLP opentelemetry section injection looks correct. Note that the \$\{_GH_AW_OTLP_HEADERS_JSON} bash variable expansion approach for headers is a bit different from the direct literal write used for the endpoint — this asymmetry could be worth documenting more explicitly so future maintainers understand why headers need special treatment.
Contributor
|
📰 VERDICT: Smoke Copilot has concluded. All systems operational. This is a developing story. 🎤 |
Contributor
Smoke Test Run §24007522566 — PARTIALCore (#1–12): ✅✅✅✅✅❌✅✅✅✅✅✅ (Tavily rate-limited) Overall: PARTIAL
|
Contributor
There was a problem hiding this comment.
💥 Automated smoke test review - all systems nominal!
💥 [THE END] — Illustrated by Smoke Claude · ● 293K
|
|
||
| export GH_AW_ENGINE="copilot" | ||
| export MCP_GATEWAY_DOCKER_COMMAND='docker run -i --rm --network host -v /var/run/docker.sock:/var/run/docker.sock -e MCP_GATEWAY_PORT -e MCP_GATEWAY_DOMAIN -e MCP_GATEWAY_API_KEY -e MCP_GATEWAY_PAYLOAD_DIR -e MCP_GATEWAY_PAYLOAD_SIZE_THRESHOLD -e DEBUG -e MCP_GATEWAY_LOG_DIR -e GH_AW_MCP_LOG_DIR -e GH_AW_SAFE_OUTPUTS -e GH_AW_SAFE_OUTPUTS_CONFIG_PATH -e GH_AW_SAFE_OUTPUTS_TOOLS_PATH -e GH_AW_ASSETS_BRANCH -e GH_AW_ASSETS_MAX_SIZE_KB -e GH_AW_ASSETS_ALLOWED_EXTS -e DEFAULT_BRANCH -e GITHUB_MCP_SERVER_TOKEN -e GITHUB_MCP_GUARD_MIN_INTEGRITY -e GITHUB_MCP_GUARD_REPOS -e GITHUB_REPOSITORY -e GITHUB_SERVER_URL -e GITHUB_SHA -e GITHUB_WORKSPACE -e GITHUB_TOKEN -e GITHUB_RUN_ID -e GITHUB_RUN_NUMBER -e GITHUB_RUN_ATTEMPT -e GITHUB_JOB -e GITHUB_ACTION -e GITHUB_EVENT_NAME -e GITHUB_EVENT_PATH -e GITHUB_ACTOR -e GITHUB_ACTOR_ID -e GITHUB_TRIGGERING_ACTOR -e GITHUB_WORKFLOW -e GITHUB_WORKFLOW_REF -e GITHUB_WORKFLOW_SHA -e GITHUB_REF -e GITHUB_REF_NAME -e GITHUB_REF_TYPE -e GITHUB_HEAD_REF -e GITHUB_BASE_REF -e GH_AW_SAFE_OUTPUTS_PORT -e GH_AW_SAFE_OUTPUTS_API_KEY -v /tmp/gh-aw/mcp-payloads:/tmp/gh-aw/mcp-payloads:rw -v /opt:/opt:ro -v /tmp:/tmp:rw -v '"${GITHUB_WORKSPACE}"':'"${GITHUB_WORKSPACE}"':rw ghcr.io/github/gh-aw-mcpg:v0.2.14' | ||
| export MCP_GATEWAY_DOCKER_COMMAND='docker run -i --rm --network host -v /var/run/docker.sock:/var/run/docker.sock -e MCP_GATEWAY_PORT -e MCP_GATEWAY_DOMAIN -e MCP_GATEWAY_API_KEY -e MCP_GATEWAY_PAYLOAD_DIR -e MCP_GATEWAY_PAYLOAD_SIZE_THRESHOLD -e DEBUG -e MCP_GATEWAY_LOG_DIR -e GH_AW_MCP_LOG_DIR -e GH_AW_SAFE_OUTPUTS -e GH_AW_SAFE_OUTPUTS_CONFIG_PATH -e GH_AW_SAFE_OUTPUTS_TOOLS_PATH -e GH_AW_ASSETS_BRANCH -e GH_AW_ASSETS_MAX_SIZE_KB -e GH_AW_ASSETS_ALLOWED_EXTS -e DEFAULT_BRANCH -e GITHUB_MCP_SERVER_TOKEN -e GITHUB_MCP_GUARD_MIN_INTEGRITY -e GITHUB_MCP_GUARD_REPOS -e GITHUB_REPOSITORY -e GITHUB_SERVER_URL -e GITHUB_SHA -e GITHUB_WORKSPACE -e GITHUB_TOKEN -e GITHUB_RUN_ID -e GITHUB_RUN_NUMBER -e GITHUB_RUN_ATTEMPT -e GITHUB_JOB -e GITHUB_ACTION -e GITHUB_EVENT_NAME -e GITHUB_EVENT_PATH -e GITHUB_ACTOR -e GITHUB_ACTOR_ID -e GITHUB_TRIGGERING_ACTOR -e GITHUB_WORKFLOW -e GITHUB_WORKFLOW_REF -e GITHUB_WORKFLOW_SHA -e GITHUB_REF -e GITHUB_REF_NAME -e GITHUB_REF_TYPE -e GITHUB_HEAD_REF -e GITHUB_BASE_REF -e GH_AW_SAFE_OUTPUTS_PORT -e GH_AW_SAFE_OUTPUTS_API_KEY -e GITHUB_AW_OTEL_TRACE_ID -e GITHUB_AW_OTEL_PARENT_SPAN_ID -v /tmp/gh-aw/mcp-payloads:/tmp/gh-aw/mcp-payloads:rw -v /opt:/opt:ro -v /tmp:/tmp:rw -v '"${GITHUB_WORKSPACE}"':'"${GITHUB_WORKSPACE}"':rw ghcr.io/github/gh-aw-mcpg:v0.2.14' |
Contributor
There was a problem hiding this comment.
The OpenTelemetry trace ID environment variable GITHUB_AW_OTEL_TRACE_ID is now passed to the MCP gateway docker command. Good addition for distributed tracing support.
| @@ -551,9 +551,14 @@ jobs: | |||
| export DEBUG="*" | |||
Contributor
There was a problem hiding this comment.
The OTLP headers parsing logic converts comma-separated key=value pairs to JSON using Node.js inline. This is a clever approach for passing OpenTelemetry headers to the MCP gateway without modifying the gateway startup script interface.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
gh-aw-mcpg v0.2.14 added support for OpenTelemetry OTLP tracing (spec §4.1.3.6). This wires up the gateway's
opentelemetryconfig block automatically whenobservability.otlpis present in workflow frontmatter (including via imported shared workflows), correlating gateway traces under the same trace/span started by theactions/setupstep.Changes
pkg/parser/import_field_extractor.go— Extractobservabilityfrom imported workflows using a first-wins strategy (same pattern asactivationGitHubToken)pkg/parser/import_processor.go— AddMergedObservability stringtoImportsResultpkg/workflow/compiler_types.go— AddOTLPEndpoint stringtoWorkflowData(the fully-resolved OTLP endpoint, set byinjectOTLPConfig)pkg/workflow/compiler_orchestrator_workflow.go— Before callinginjectOTLPConfig, merge imported observability intoRawFrontmatterwhen the top-level workflow has noobservabilitysection (enables shared imports likeshared/observability-otlp.md)pkg/workflow/observability_otlp.go— Store the resolved endpoint inworkflowData.OTLPEndpointafter injection — single source of truth for all downstream codepkg/workflow/tools_types.go— ReplaceOTLPEnabled boolwithOTLPEndpoint stringandOTLPHeaders stringonMCPGatewayRuntimeConfig(yaml:"-", compiler-derived, not user-settable)pkg/workflow/mcp_gateway_config.go— PopulateOTLPEndpointandOTLPHeadersfrom the fully-resolved frontmatter (including imports)pkg/workflow/mcp_renderer.go— Write theopentelemetryblock directly into the gateway config JSON. The endpoint is written as a literal value. Headers require a JSON object per the gateway schema, so a bash preamble is generated before the heredoc that convertsOTEL_EXPORTER_OTLP_HEADERS(already a job-level env var frominjectOTLPConfig) fromkey=value,...OTLP string format to a JSON object vianode; the heredoc uses the unquoted bash variable${_GH_AW_OTLP_HEADERS_JSON}so bash expands it to the actual object.traceIdandspanIduse${GITHUB_AW_OTEL_TRACE_ID}/${GITHUB_AW_OTEL_PARENT_SPAN_ID}which bash expands from the GITHUB_ENV values set byactions/setup.pkg/workflow/mcp_setup_generator.go— Pass onlyGITHUB_AW_OTEL_TRACE_IDandGITHUB_AW_OTEL_PARENT_SPAN_IDto the gateway container via-e;OTEL_EXPORTER_OTLP_ENDPOINTandOTEL_EXPORTER_OTLP_HEADERSare no longer forwarded since their values are written directly in the configRuntime gateway config produced
The endpoint is taken directly from the
observability.otlpfrontmatter and written as a literal value — GitHub Actions expressions (e.g.${{ secrets.X }}) are preserved verbatim and expanded by the runner at execution time. Headers are converted at runtime from the OTLPkey=value,...string format to a proper JSON object (as required by the gateway schema) using a generated bash preamble.GITHUB_AW_OTEL_TRACE_IDandGITHUB_AW_OTEL_PARENT_SPAN_IDare written toGITHUB_ENVbyactions/setupviaaction_setup_otlp.cjs, making the gateway spans children of the setup span and correlating all MCP tool call traces under the workflow's root trace.The
observability.otlpconfig is now fully resolved including imports: a shared workflow (e.g..github/workflows/shared/observability-otlp.md) imported by multiple workflows will correctly enable gateway OTLP tracing in each of them.No lock files change for workflows without
observability.otlpconfigured.✨ PR Review Safe Output Test - Run 24007522566