Streaming AI Telemetry: Processing 5M+ Daily LLM Inferences with Apache Kafka and Edge Workers

1. The Decoupling Mandate: Zero-Latency Telemetry

When deploying enterprise generative AI applications, security teams demand full telemetry: token counts, input prompt embeddings, sentiment shifts, and automated PII redaction checks. However, executing these evaluations synchronously inside the request-response lifecycle creates catastrophic latency spikes.

To solve this at XecureAI, we decoupled telemetry ingestion entirely. Incoming user requests stream through Next.js Edge Runtime workers, which forward fire-and-forget telemetry payloads to an Apache Kafka cluster partitioned across distributed regions.

2. Producer Partitioning and Consumer Worker Pools

By keying Kafka messages on tenant IDs and session hashes, we guarantee message ordering for conversational histories without creating broker hotspots. Consumer worker pools written in Go and Node.js process streaming chunks in parallel, running automated regex compliance scans and model risk telemetry.

import { Kafka, Partitioners } from "kafkajs";

const kafka = new Kafka({
  clientId: "telemetry-edge-stream",
  brokers: [process.env.KAFKA_BROKER_URL!],
  ssl: true,
});

export const producer = kafka.producer({
  createPartitioner: Partitioners.DefaultPartitioner,
});

export async function emitInferenceTelemetry(event: {
  tenantId: string;
  sessionId: string;
  model: string;
  promptTokens: number;
  completionTokens: number;
  durationMs: number;
}) {
  await producer.send({
    topic: "llm.inference.telemetry",
    messages: [
      {
        key: event.tenantId,
        value: JSON.stringify({ ...event, timestamp: Date.now() }),
      },
    ],
  });
}