The Missing Contracts in Event-Driven Systems
REST APIs have OpenAPI. gRPC services have .proto files. Event-driven systems built on Kafka or AMQP had nothing. No standard way to document what events a service published, what schema each event followed, or what topics consumers should subscribe to. Teams learned about events by reading source code or asking in Slack.
AsyncAPI: OpenAPI for Event-Driven
Fran Mendez created AsyncAPI in 2016 as the event-driven equivalent of OpenAPI. An AsyncAPI document describes channels (Kafka topics or AMQP queues), operations (publish or subscribe), message schemas (using JSON Schema or Avro), and server connection details. The same tooling philosophy as OpenAPI: one spec, generate documentation, generate client code, validate messages against schema.
AsyncAPI adoption has grown significantly as event-driven architectures became mainstream. Major cloud providers (AWS EventBridge, Azure Event Grid) support it. The Kafka ecosystem has tooling built around it. If your team is designing event contracts, AsyncAPI is the standard way to document them.
CloudEvents: A Standard Event Envelope
CloudEvents (CNCF, 2018) standardized the envelope around event data - the metadata that describes what an event is, not the event payload itself. A CloudEvents-compliant event has standard required attributes: id (unique event ID), source (URI identifying the event origin), specversion (CloudEvents version), and type (event type identifier). Optional attributes add time, datacontenttype, subject, and more.
The value is interoperability. An event published by a Go service using CloudEvents can be consumed by a Python service, routed by AWS EventBridge, and stored in a schema registry - all without custom event parsing. The cloud providers all support CloudEvents natively, which means your events become portable between cloud platforms.
The Transactional Outbox Pattern
The transactional outbox solves the dual-write problem in event-driven systems. When a service updates its database and publishes an event, what happens if the database write succeeds but the event publish fails? Or the other way around? You have inconsistent state distributed across two systems with no recovery path.
The outbox pattern solves this with a single atomic operation: write to your application database AND write the event to an 'outbox' table in the same transaction. A separate process reads the outbox table and publishes to your message broker. If the publish fails, the outbox entry is retried. If the process crashes, it restarts from the last unpublished entry. Consistency is guaranteed because the source of truth is the database transaction.
Debezium and Change Data Capture
Debezium (Red Hat, 2016) automates the outbox publishing step using Change Data Capture (CDC). It reads the database binary log (Postgres WAL, MySQL binlog) and publishes every row change as a Kafka event. Your service writes only to the database. Debezium handles the event publishing. No dual-write risk, no outbox table management, and your existing database becomes the event source.
Event-Driven Contract Testing
Contract testing for event-driven systems is harder than for REST APIs because events are asynchronous and the producer and consumer may deploy independently. Pact supports asynchronous message contract testing: the consumer writes a test that says 'I expect to receive a message that looks like this', and the producer writes a test that verifies it publishes messages that match all consumer contracts. This provider-consumer contract testing catches schema mismatches before production without requiring running brokers in the test environment.
Schema registries (Confluent Schema Registry, AWS Glue Schema Registry) enforce contracts at runtime. When a producer publishes an event, the schema registry validates it against the registered schema. Forward compatibility means new schemas can read data written with old schemas. Backward compatibility means old schemas can read data written with new schemas. Full compatibility means both. Choosing the right compatibility mode depends on whether your consumers or producers update first.
The Outbox Pattern's Implementation Variants
The basic outbox pattern (write to database + write to outbox table in one transaction, poll outbox table to publish) works but polling introduces latency. The CDC variant (Debezium reads the database WAL) eliminates polling latency: events are published within milliseconds of the database write. The trade-off is operational complexity: running Debezium requires a Kafka cluster, careful connector configuration, and monitoring. For most systems, polling the outbox table every second or two is acceptable and dramatically simpler to operate.
The relay pattern is a third option: the application writes to the outbox table, and a relay process within the same service reads the outbox and publishes to the broker. The relay runs in a separate thread or goroutine. This is simpler than Debezium (no external infrastructure) and lower latency than external polling. The relay can be implemented in 50 lines in most languages. It should be in your toolkit before reaching for CDC.
- CloudEvents attributes should be present on every event in your system. The id, source, type, and time attributes answer 'what happened, where, and when' for any event in your pipeline.
- Event type naming convention: use reverse-DNS format for source + past-tense verb for type. Example: com.stripe.payment.succeeded.
- AsyncAPI supports OpenAPI-compatible JSON Schema for message payload schemas. If your REST API already has OpenAPI schemas, reuse them in your AsyncAPI spec.
- Dead letter queues (DLQ) should have monitoring and alerts. A growing DLQ means a consumer is failing consistently - find out why before it becomes a data loss incident.
- Message TTL (time-to-live) should match the business semantics of the event. An order.placed event that is not processed within 24 hours may need different handling than a telemetry.metric event that is not processed within 60 seconds.
Designing Event Schemas That Age Well
Event schemas have a property that REST API schemas do not: they are often stored and replayed months or years later. A REST response schema can be changed with a version bump and a migration period. An event schema must remain readable by consumers that may not update immediately and by data pipelines that replay historical events. This means you must design for forward compatibility from the start: use optional fields, never remove fields (mark them deprecated instead), never change field types, and version your event schemas explicitly.
Schema registries (Confluent Schema Registry, AWS Glue Schema Registry) solve the schema evolution problem systematically. Producers register schemas and receive a schema ID. The schema ID is embedded in every message. Consumers look up the schema by ID to deserialize. Avro's schema evolution rules (backward, forward, full compatibility modes) are enforced by the registry at publish time. This infrastructure investment pays for itself the first time you need to replay three months of events through a new consumer that was not running when the events were published.
Event-Driven Contract Testing
Contract testing for event-driven systems is harder than for REST APIs because events are asynchronous and the producer and consumer may deploy independently. Pact supports asynchronous message contract testing: the consumer writes a test that says 'I expect to receive a message that looks like this', and the producer writes a test that verifies it publishes messages that match all consumer contracts. This provider-consumer contract testing catches schema mismatches before production without requiring running brokers in the test environment.
Schema registries (Confluent Schema Registry, AWS Glue Schema Registry) enforce contracts at runtime. When a producer publishes an event, the schema registry validates it against the registered schema. Forward compatibility means new schemas can read data written with old schemas. Backward compatibility means old schemas can read data written with new schemas. Full compatibility means both. Choosing the right compatibility mode depends on whether your consumers or producers update first.
The Outbox Pattern's Implementation Variants
The basic outbox pattern (write to database + write to outbox table in one transaction, poll outbox table to publish) works but polling introduces latency. The CDC variant (Debezium reads the database WAL) eliminates polling latency: events are published within milliseconds of the database write. The trade-off is operational complexity: running Debezium requires a Kafka cluster, careful connector configuration, and monitoring. For most systems, polling the outbox table every second or two is acceptable and dramatically simpler to operate.
The relay pattern is a third option: the application writes to the outbox table, and a relay process within the same service reads the outbox and publishes to the broker. The relay runs in a separate thread or goroutine. This is simpler than Debezium (no external infrastructure) and lower latency than external polling. The relay can be implemented in 50 lines in most languages. It should be in your toolkit before reaching for CDC.
- CloudEvents attributes should be present on every event in your system. The id, source, type, and time attributes answer 'what happened, where, and when' for any event in your pipeline.
- Event type naming convention: use reverse-DNS format for source + past-tense verb for type. Example: com.stripe.payment.succeeded.
- AsyncAPI supports OpenAPI-compatible JSON Schema for message payload schemas. If your REST API already has OpenAPI schemas, reuse them in your AsyncAPI spec.
- Dead letter queues (DLQ) should have monitoring and alerts. A growing DLQ means a consumer is failing consistently - find out why before it becomes a data loss incident.
- Message TTL (time-to-live) should match the business semantics of the event. An order.placed event that is not processed within 24 hours may need different handling than a telemetry.metric event that is not processed within 60 seconds.
Designing Event Schemas That Age Well
Event schemas have a property that REST API schemas do not: they are often stored and replayed months or years later. A REST response schema can be changed with a version bump and a migration period. An event schema must remain readable by consumers that may not update immediately and by data pipelines that replay historical events. This means you must design for forward compatibility from the start: use optional fields, never remove fields (mark them deprecated instead), never change field types, and version your event schemas explicitly.
Schema registries (Confluent Schema Registry, AWS Glue Schema Registry) solve the schema evolution problem systematically. Producers register schemas and receive a schema ID. The schema ID is embedded in every message. Consumers look up the schema by ID to deserialize. Avro's schema evolution rules (backward, forward, full compatibility modes) are enforced by the registry at publish time. This infrastructure investment pays for itself the first time you need to replay three months of events through a new consumer that was not running when the events were published.
Event-Driven Contract Testing
Contract testing for event-driven systems is harder than for REST APIs because events are asynchronous and the producer and consumer may deploy independently. Pact supports asynchronous message contract testing: the consumer writes a test that says 'I expect to receive a message that looks like this', and the producer writes a test that verifies it publishes messages that match all consumer contracts. This provider-consumer contract testing catches schema mismatches before production without requiring running brokers in the test environment.
Schema registries (Confluent Schema Registry, AWS Glue Schema Registry) enforce contracts at runtime. When a producer publishes an event, the schema registry validates it against the registered schema. Forward compatibility means new schemas can read data written with old schemas. Backward compatibility means old schemas can read data written with new schemas. Full compatibility means both. Choosing the right compatibility mode depends on whether your consumers or producers update first.
The Outbox Pattern's Implementation Variants
The basic outbox pattern (write to database + write to outbox table in one transaction, poll outbox table to publish) works but polling introduces latency. The CDC variant (Debezium reads the database WAL) eliminates polling latency: events are published within milliseconds of the database write. The trade-off is operational complexity: running Debezium requires a Kafka cluster, careful connector configuration, and monitoring. For most systems, polling the outbox table every second or two is acceptable and dramatically simpler to operate.
The relay pattern is a third option: the application writes to the outbox table, and a relay process within the same service reads the outbox and publishes to the broker. The relay runs in a separate thread or goroutine. This is simpler than Debezium (no external infrastructure) and lower latency than external polling. The relay can be implemented in 50 lines in most languages. It should be in your toolkit before reaching for CDC.
- CloudEvents attributes should be present on every event in your system. The id, source, type, and time attributes answer 'what happened, where, and when' for any event in your pipeline.
- Event type naming convention: use reverse-DNS format for source + past-tense verb for type. Example: com.stripe.payment.succeeded.
- AsyncAPI supports OpenAPI-compatible JSON Schema for message payload schemas. If your REST API already has OpenAPI schemas, reuse them in your AsyncAPI spec.
- Dead letter queues (DLQ) should have monitoring and alerts. A growing DLQ means a consumer is failing consistently - find out why before it becomes a data loss incident.
- Message TTL (time-to-live) should match the business semantics of the event. An order.placed event that is not processed within 24 hours may need different handling than a telemetry.metric event that is not processed within 60 seconds.
Designing Event Schemas That Age Well
Event schemas have a property that REST API schemas do not: they are often stored and replayed months or years later. A REST response schema can be changed with a version bump and a migration period. An event schema must remain readable by consumers that may not update immediately and by data pipelines that replay historical events. This means you must design for forward compatibility from the start: use optional fields, never remove fields (mark them deprecated instead), never change field types, and version your event schemas explicitly.
Schema registries (Confluent Schema Registry, AWS Glue Schema Registry) solve the schema evolution problem systematically. Producers register schemas and receive a schema ID. The schema ID is embedded in every message. Consumers look up the schema by ID to deserialize. Avro's schema evolution rules (backward, forward, full compatibility modes) are enforced by the registry at publish time. This infrastructure investment pays for itself the first time you need to replay three months of events through a new consumer that was not running when the events were published.