Service Onboarding Guide

How to instrument your service to send traces, logs, and metrics to the mpac-obs observability stack.

OTLP Endpoints

Your service sends all telemetry (traces, metrics, logs) to Alloy via OTLP.

Environment	gRPC Endpoint	HTTP Endpoint
Local	localhost:4317	localhost:4318
Dev (AWS)	alloy.mpac-obs.dev.local:4317	alloy.mpac-obs.dev.local:4318
Staging (AWS)	alloy.mpac-obs.staging.local:4317	alloy.mpac-obs.staging.local:4318
Production (AWS)	alloy.mpac-obs.prod.local:4317	alloy.mpac-obs.prod.local:4318

Authentication

All OTLP endpoints require a bearer token:

OTEL_EXPORTER_OTLP_HEADERS="Authorization=Bearer dev-otlp-key-2025"

Service Naming Convention

Use the following format for service.name:

{system}.{service-name}

Service	service.name	service.namespace
Payment Gateway Backend	mpac-pgw.backend	mpac-pgw
Portal API	mpac.svc-portal	mpac
SmartTab API	mpac.svc-smarttab	mpac
Portal Web (browser)	mpac.portal-web	mpac
Terminal WebView	mpac.webview	mpac
Terminal Business App	mpac.terminal.business-app	mpac

Environment Variables

Set these environment variables in your service:

# Required
OTEL_ENABLED=true
OTEL_SERVICE_NAME=mpac.svc-portal                  # Your service name
OTEL_EXPORTER_OTLP_ENDPOINT=localhost:4317           # Alloy endpoint (gRPC)
OTEL_EXPORTER_OTLP_HEADERS="Authorization=Bearer dev-otlp-key-2025"

# Recommended
OTEL_RESOURCE_ATTRIBUTES="service.namespace=mpac,deployment.environment=development,service.version=1.0.0"

# Optional
OTEL_TRACES_SAMPLER=parentbased_traceidratio
OTEL_TRACES_SAMPLER_ARG=1.0                          # 1.0 = 100% sampling (dev), 0.1 = 10% (prod)
OTEL_LOG_LEVEL=info

Go Service Integration

Install Dependencies

go get go.opentelemetry.io/otel
go get go.opentelemetry.io/otel/sdk
go get go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracegrpc
go get go.opentelemetry.io/contrib/instrumentation/github.com/gin-gonic/gin/otelgin

Initialize OpenTelemetry

package telemetry

import (
    "context"
    "os"

    "go.opentelemetry.io/otel"
    "go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracegrpc"
    "go.opentelemetry.io/otel/sdk/resource"
    sdktrace "go.opentelemetry.io/otel/sdk/trace"
    semconv "go.opentelemetry.io/otel/semconv/v1.24.0"
    "google.golang.org/grpc"
    "google.golang.org/grpc/credentials/insecure"
)

func InitTracer(ctx context.Context) (func(), error) {
    endpoint := os.Getenv("OTEL_EXPORTER_OTLP_ENDPOINT")
    if endpoint == "" {
        endpoint = "localhost:4317"
    }

    conn, err := grpc.NewClient(endpoint,
        grpc.WithTransportCredentials(insecure.NewCredentials()),
    )
    if err != nil {
        return nil, err
    }

    exporter, err := otlptracegrpc.New(ctx, otlptracegrpc.WithGRPCConn(conn))
    if err != nil {
        return nil, err
    }

    res, _ := resource.Merge(
        resource.Default(),
        resource.NewWithAttributes(
            semconv.SchemaURL,
            semconv.ServiceName(os.Getenv("OTEL_SERVICE_NAME")),
            semconv.ServiceNamespace(os.Getenv("OTEL_SERVICE_NAMESPACE")),
        ),
    )

    tp := sdktrace.NewTracerProvider(
        sdktrace.WithBatcher(exporter),
        sdktrace.WithResource(res),
    )
    otel.SetTracerProvider(tp)

    return func() { tp.Shutdown(ctx) }, nil
}

Instrument Gin Router

import "go.opentelemetry.io/contrib/instrumentation/github.com/gin-gonic/gin/otelgin"

router := gin.Default()
router.Use(otelgin.Middleware("mpac.svc-smarttab"))

Python Service Integration

Install Dependencies

uv pip install opentelemetry-api opentelemetry-sdk \
    opentelemetry-exporter-otlp-proto-grpc \
    opentelemetry-instrumentation-fastapi \
    opentelemetry-instrumentation-sqlalchemy \
    opentelemetry-instrumentation-httpx

Initialize OpenTelemetry

# app/telemetry.py
import os
from opentelemetry import trace
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor
from opentelemetry.exporter.otlp.proto.grpc.trace_exporter import OTLPSpanExporter
from opentelemetry.sdk.resources import Resource, SERVICE_NAME, SERVICE_NAMESPACE


def init_telemetry():
    if not os.getenv("OTEL_ENABLED", "").lower() == "true":
        return

    resource = Resource.create({
        SERVICE_NAME: os.getenv("OTEL_SERVICE_NAME", "mpac.svc-portal"),
        SERVICE_NAMESPACE: "mpac",
    })

    exporter = OTLPSpanExporter(
        endpoint=os.getenv("OTEL_EXPORTER_OTLP_ENDPOINT", "localhost:4317"),
        insecure=True,
    )

    provider = TracerProvider(resource=resource)
    provider.add_span_processor(BatchSpanProcessor(exporter))
    trace.set_tracer_provider(provider)

Instrument FastAPI

from opentelemetry.instrumentation.fastapi import FastAPIInstrumentor

app = FastAPI()
FastAPIInstrumentor.instrument_app(app)

Structured Logging

For logs to be properly parsed and correlated with traces, use JSON structured logging with these fields:

{
    "level": "info",
    "msg": "Payment processed",
    "trace_id": "abc123...",
    "span_id": "def456...",
    "request_id": "req-789",
    "corr_id": "corr-012",
    "merchant_id": "M001",
    "store_id": "S001",
    "method": "POST",
    "path": "/api/v1/payments",
    "status": 200,
    "latency": 45.2
}

Go (zerolog / slog)

logger.Info().
    Str("trace_id", span.SpanContext().TraceID().String()).
    Str("span_id", span.SpanContext().SpanID().String()).
    Str("merchant_id", tenant.MerchantID).
    Msg("Payment processed")

Python (structlog)

import structlog

logger = structlog.get_logger()
logger.info("payment_processed",
    trace_id=span.get_span_context().trace_id,
    merchant_id=tenant.merchant_id,
)

Verification

After instrumenting your service, verify data is flowing:

1. Start the observability stack: make up (from mpac-obs root)
2. Start your service with OTEL environment variables set
3. Generate some traffic (make API calls)
4. Check in Grafana (http://localhost:3000):
   • Traces: Explore -> Tempo -> Search for your service name
   • Logs: Explore -> Loki -> {service="your-service-name"}
   • Metrics: Explore -> Prometheus -> http_server_request_duration_seconds_count

Correlation

All telemetry is automatically correlated via:

• trace_id / span_id: Links logs to specific traces
• service label: Groups all telemetry by service name
• request_id / corr_id: Application-level correlation

In Grafana:

• Click a trace -> "Logs for this trace" to see related logs
• Click a log line -> "View trace" to jump to the trace
• Dashboard panels filter by the same service variable