An Open Changelog Format: The Missing Layer in the Open Data Stack

Why an open change log matters

The modern data stack opened from the bottom up. Open file formats — Parquet, ORC, Avro — standardized how data is encoded at rest, so any engine can read the bytes. Open table formats — Iceberg, Delta Lake, Hudi — standardized how those files behave as tables, so transactions, schema, and time travel stopped depending on any one vendor. Both wins came the same way: standardize the format, compete on the engine.

One layer never got the treatment: the changelog.

Change data capture (CDC) has become the connective tissue of the modern data estate. Every insert, update, delete, and schema change streams out of operational databases as it happens, feeding lakehouses, real-time analytics, replication, and event-driven systems. For the most part, the capture part works fine — consumers have lots of choice between open-source and commercial tool vendors. The problem is that every CDC engine, database, and ingest layer speaks its own dialect (formats, types, protocols, semantics, and so on).

Event formats differ. Type encodings differ. Ordering guarantees, transaction semantics, and schema-change handling all differ. So every producer-consumer pairing becomes a bespoke integration project, and the industry keeps rebuilding the same N×M connector matrix, but in their own boutique, proprietary way that creates a closed ecosystem around their product. Each engine, wired to each downstream platform, with fidelity quietly lost at every seam. Each platform seam is opaque and hard to reason about. Truncated decimals. Ambiguous timestamps. Mysterious timezones. Dropped transaction boundaries. DDL handled as an afterthought. Customers pay for this as lock-in and fragile pipelines. Vendors pay for it as duplicated engineering that differentiates no one.

An open changelog format replaces the fragmented ecosystem of bespoke N×M matrices with a wire contract, the same way open table formats replaced per-engine table management. A working group has a draft on the table — a specification currently called OpenCDC — that specifies the structure, semantics, ordering, type fidelity, and lifecycle of the events a CDC producer emits, precisely enough that a consumer can interpret any conformant stream without knowing which engine produced it. It standardizes the events, not the engines. Your capture tool stays; its output becomes portable.

What a changelog event looks like

Before the principles, here's the thing itself. As defined in the specification today, every event is a fully conformant CloudEvent — the CNCF envelope standard already used across the event-driven ecosystem — extended with CDC-specific attributes, carrying a change payload. Three layers, each with a clear purpose.

Everything here is readable with a JSON parser and the specification document. JSON is a minimal conformance requirement, but it is not the only syntax the spec can and will support.

The design principles that make it work

The spec document opens with its design principles, and every normative decision (e.g., in the Architecture Decision Record) traces back to them. Four major themes matter most if you're evaluating an open changelog as a consumer of change data.

Nothing to install, nothing to ask

A conformant stream must be fully consumable with only a parser and the spec. No broker, no schema registry, no SDK, no vendor tooling. Infrastructure like a registry is allowed as an optimization, but it can never be required.

That works because the stream describes itself. Schemas are first-class events: an OBJECT_METADATA event declares a table's complete structure before any data for that table flows, and any change to the emitted schema produces a new declaration before data flows under it. In this way, every data event points back to its governing schema through the envelope's dataschema field.

The practical consequence: your consumer can be built by a different team, on a different stack, years later, and it still decodes the stream from the stream alone.

Streams that state their guarantees

This is the part most CDC formats never attempted, mainly because they were always tied to specific (known ahead of time) tooling. Instead of letting consumers assume what a stream guarantees, the producer declares it in the stream, using a STREAM_METADATA event, along independent capability axes. Two of those axes govern transaction ordering, which are important to understand in more detail.

Does the producer emit one ordered sequence, or several? That's ordering_scope. A value of "stream" means one totally ordered sequence — every event has a definite position relative to every other event: [e1, e2, e3, e4, …]. A value of "channel" means several ordered sequences, say one per table: A:[a1, a2, a3]  B:[b1, b2]. Order holds within each channel; between channels it isn't stated, and consumers build no logic on it. That is the entire claim: one sequence or several.

When transactions overlap, do their events stay grouped? That's transaction_interleaving. A value of "none" means transactions are emitted whole, one after another: [T1, T1, T1] [T2, T2]. A value of "possible" means events from different transactions may mix: [T1, T2, T1, T3, T2, …]. Each transaction stays internally ordered, but the transactions overlap. That is the entire claim: contiguous or not.

When interleaving is possible, ordering alone can no longer tell a consumer that a transaction is complete — so the spec makes explicit TRX_COMMIT markers mandatory for every transaction. Completion becomes deterministic. No timeouts, no guessing.

One scoping decision is important to fully understand: the specification is focused on what the producer can guarantee, not on what a particular message transport has implemented. The specification itself is designed to be transport-neutral. Therefore, these axes describe what the producer emits, not what arrives after a broker or transport has handled the stream. That's what lets a connector-style producer make an honest, testable claim: it can declare exactly what it emits even when it doesn't control delivery. End-to-end guarantees across specific transports are the job of binding profiles, which are on the roadmap — more on that below.

No translation layer, no surprise fields

Two principles form the interoperability backbone. First: any event that requires consumer-side knowledge of the originating vendor to decode is non-conformant. Full stop. Independently built producers and consumers from different vendors must be able to exchange events with no translation layer between them.

Second: every schema ships closed-world, for example in the specification with additionalProperties: false. If a field shows up in a payload that wasn't declared in the schema, that's a validation failure, not a warning. It sounds strict because it is. The payoff is a contract precise enough to build against: if a field appears, it was declared; if it wasn't declared, the producer is broken, and your validator says so.

Source-neutral, encoding-neutral

The core standard contains no Oracle-specific, DB2-specific, MySQL-specific, or PostgreSQL-specific constructs. Schema evolution follows the same self-sufficiency rule as everything else: detectable by reading the stream, with no polling or database introspection. OBJECT_METADATA carries that guarantee unconditionally on every conformant stream. Producers that can see source DDL can additionally declare a ddl_capture capability and emit DDL events with provenance — the DDL operation's kind, the verbatim statement where available, its exact position in transaction order.

And JSON, despite appearing in every example, is the normative example format rather than the mandatory wire encoding. Avro, Protobuf, Arrow IPC — all conformant, provided the logical structure, field semantics, and type rules are preserved. High-throughput deployments get binary efficiency; the spec stays verifiable with a text editor.

Types that don't lie

Type handling is where CDC pipelines quietly lose data today — though not the way you might guess. Every CDC connector already normalizes types; that's the job. The problem is that each one normalizes into its own private vocabulary, shaped by its own configuration, and usually throws the source declaration away. The same timestamp column reaches a consumer as epoch microseconds from one tool, epoch milliseconds from the same tool under a different setting, and an ISO-8601 string from a third. Telling them apart means recognizing the producer's private schema names — exactly the vendor knowledge a portable format must not require. And when that consumer needs to create the table downstream, the original declaration is gone: NUMBER(10,2) arrived as "some kind of decimal," VARCHAR2(20 BYTE) as "a string," and the target schema is a guess.

The spec's answer is to make every column carry two layers of type information, because the two failures above are different failures. logical_type holds a name from one canonical vocabulary, shared by every producer, and it drives interpretation: consumers route, validate, and map without parsing engine-specific DDL — that's what heterogeneous pipelines need, an Oracle change landing in Iceberg or Postgres with nothing lost in translation. source_type holds the verbatim DDL declaration from the originating engine — NUMBER(10,2), VARCHAR2(20 BYTE) — never interpreted, never normalized, so a consumer that knows the source engine can reconstruct exact semantics, and a like-for-like replica can be built from the declaration instead of a guess. Fidelity and interoperability, carried together instead of traded off.

Real engines make this harder than it sounds. The same type name can mean different things on different engines. Some types exist on one engine and nowhere else. Precision and ranges rarely line up. The spec's stance on all of it is consistent: carry the difference explicitly instead of papering over it, and never change a value silently to make it fit a cleaner model. Exactly how each case is represented — which types earn a shared canonical name, how engine-only types travel — is active working-group territory, being refined against real engines rather than settled in the abstract. The commitment underneath doesn't move: a consumer can read a conformant stream correctly without knowing which engine produced it, and what the source declared survives the trip.

What conformance actually means

"OpenCDC conformant" needs to be a testable claim, so the spec defines a Minimum Viable Interoperability Profile: the floor every conformant implementation must clear, covering DML semantics, mandatory schema delivery, the canonical type system, stable event identity for replay, transaction ordering, and stream liveness. Behind the spec's prose sits a machine-readable requirements register from which the conformance matrix is generated and CI-verified against the spec body on every change. If the document and the register drift, the build fails.

Where the working group is headed

The specification is still a draft and the working group has a path forward. Transport binding profiles are one of the biggest items: the current spec already carries informative Kafka consumption guidance as a deliberate precursor to a normative Kafka binding, and bindings are where delivered-stream guarantees — what survives a specific transport, end to end — get their formal home. A bidirectional interoperability profile is in active design, targeting cross-vendor two-way replication with loop suppression, with multiple candidate approaches under working-group review. Snapshot and initial-load semantics are slated for additional work to flesh out the details required to include them in the specification. And as the spec approaches v1.0, the group is preparing it for a standards-track home, with its CloudEvents lineage as the natural foundation.

Draft status is a feature right now. There are no shipped implementations to migrate, which means feedback today is the cheapest feedback the project will ever get. If you build databases, build pipelines, run a lakehouse, operate replication, or ship a CDC tool, the working group wants your input — especially on the capability axes and the minimum profile, where real-world deployment shapes matter most.