Intercom Observability

Overview

Comprehensive observability for Intercom integrations covering Prometheus metrics, OpenTelemetry traces, structured logging, and alert rules for error rates, latency, and rate-limit usage. Read this page for the workflow and shape of each layer, then drill into references/implementation.md for the full, copy-pasteable code and references/examples.md for end-to-end worked scenarios.

Prerequisites

• Prometheus or compatible metrics backend
• OpenTelemetry SDK (optional, for tracing)
• Pino or similar structured logger
• Grafana or alerting system

Instructions

Build the six observability layers in order. Each step below is the summary and the essential skeleton — the complete implementation for every step lives in references/implementation.md.

Step 1: Prometheus metrics

Define five instruments on a shared Registry: a request counter, a duration histogram, an error counter, a rate-limit gauge, and a webhook counter. Label by endpoint/method/status (never by unbounded IDs — see Error Handling).

import { Registry, Counter, Histogram, Gauge } from "prom-client";
const registry = new Registry();
const intercomRequests = new Counter({
  name: "intercom_api_requests_total",
  help: "Total Intercom API requests",
  labelNames: ["endpoint", "method", "status"] as const,
  registers: [registry],
});
// + duration Histogram, error Counter, rate-limit Gauge, webhook Counter

Full metric set → references/implementation.md, Step 1.

Step 2: Instrumented client wrapper

Wrap IntercomClient in a Proxy that times every service method, increments the success/error counters, records error/status codes on IntercomError, and zeros the rate-limit gauge on a 429 — so instrumentation is automatic for all endpoints. Full proxy → references/implementation.md, Step 2.

Step 3: Structured logging

Configure Pino with a contact serializer that emits only id/role and never logs email, name, or phone. Add logIntercomOp and logWebhook helpers for consistent operation/webhook log lines. Full logger → references/implementation.md, Step 3.

Step 4: OpenTelemetry tracing

Wrap calls in tracedIntercomCall, which opens a per-operation intercom.* span, sets OK/ERROR status, records exceptions, and attaches status_code/error_code/request_id attributes on Intercom errors. Full tracer → references/implementation.md, Step 4.

Step 5: Alert rules

Ship the Prometheus rule group with five alerts: high error rate (>5%), high P95 latency (>3s), low rate limit (<1000), auth failures (401s), and webhook failures. Full YAML → references/implementation.md, Step 5.

Step 6: Metrics endpoint

Expose the registry on GET /metrics for Prometheus to scrape. Full route → references/implementation.md, Step 6.

Output

Applying this skill produces:

• Instrumented client — an IntercomClient proxy that emits metrics on every call, with zero per-call changes to existing code.
• Metrics — intercom_api_requests_total, intercom_api_request_duration_seconds, intercom_api_errors_total, intercom_rate_limit_remaining, intercom_webhooks_processed_total, scraped at GET /metrics.
• Traces — one per-operation intercom.* span per call, with Intercom error attributes on failures.
• Structured logs — PII-redacted JSON operation and webhook log lines.
• Alerts — a Prometheus rule group covering error rate, latency, rate limit, auth, and webhooks.

Key metrics summary

Metric	Type	Alert Threshold
intercom_api_requests_total	Counter	N/A (baseline)
intercom_api_request_duration_seconds	Histogram	P95 > 3s
intercom_api_errors_total	Counter	> 5% error rate
intercom_rate_limit_remaining	Gauge	< 1000
intercom_webhooks_processed_total	Counter	Failed > 10%

Error Handling

Issue	Cause	Solution
High cardinality	Too many unique labels	Use endpoint groups, not IDs
Missing metrics	Uninstrumented calls	Wrap client with proxy
Alert storms	Wrong thresholds	Tune based on baseline data
Log volume too high	Debug logging in prod	Set LOG_LEVEL=info

Examples

The following scenarios are covered in full in references/examples.md:

• Contact lookup end-to-end — one contacts.find call producing a counter increment, a histogram sample, a span, and a PII-redacted log line.
• Rate-limit (429) event — how the proxy zeros the rate-limit gauge and which alerts fire.
• Webhook success/failure accounting — counting processed vs. failed webhooks per topic.
• Scraping /metrics — the raw Prometheus exposition and how it feeds Grafana and the alert rules.

Minimal end-to-end skeleton:

const client = instrumentedClient(new IntercomClient({ token: process.env.INTERCOM_ACCESS_TOKEN! }));
const contact = await tracedIntercomCall(
  "contacts.find",
  { "intercom.contact_id": contactId },
  () => client.contacts.find({ contactId })
);

Resources

• full implementation walkthrough — every step's complete code
• worked examples — end-to-end scenarios and observed output
• Prometheus Best Practices
• OpenTelemetry Node.js
• Pino Logger

Next Steps

For incident response once these signals are firing, see the intercom-incident-runbook skill, which turns these alerts into a triage-and-mitigation procedure.