Delta Lake feature compatibility and protocols

Delta Lake table protocols specify which features a client must support to read or write a table. This page covers protocol versions, table features, compatibility requirements, and how Azure Databricks manages protocol upgrades. See also Review table details with describe detail.

Table protocol and compatibility

Every Delta Lake table has a protocol specification which indicates the set of capabilities required to read from and write to the table. Applications use the protocol specification to determine if they can support all the features that the table uses. If an application can't support a feature in the current protocol of a table, then that application can't read or write that table.

Most new capabilities Delta Lake require you to upgrade the table protocol.

The following table lists key terms that describe Delta Lake protocols:

Write protocols and writer features only affect compatibility with writer clients, allowing read-only access to the table from legacy workloads.

Not all Delta Lake features are compatible with each other.

Some table features can't be dropped once enabled. See Drop a Delta Lake table feature and downgrade table protocol.

Protocol versions and table features

All Delta Lake tables include an integer-based protocol version represented by minReaderVersion and minWriterVersion. Each version bundles multiple features together, and features are cumulative across versions. To comply with the Delta Lake protocol, clients must implement support for all features in a given version, including all previously released features.

In Databricks Runtime 12.2 LTS and above, table features replace the integer-based protocol with granular flags that indicate which features a table uses. This allows finer-grained compatibility checks between clients and tables.

Table writer features affect the way data is written. They require minWriterVersion=7 but don't block reader clients.

Table reader features affect the way data is read. All reader features are also writer features, and require minReaderVersion=3 and minWriterVersion=7. A client can't write to a table that it can't read.

When table features are enabled, they appear in the protocol as readerFeatures or writerFeatures. Delta Lake resolves the table protocol to the lowest version that supports all enabled features. See Lowest possible protocol.

Protocol changes

The protocol for a table changes under the following conditions:

• If a new feature is enabled, the protocol is upgraded.
• If a table feature is dropped, the protocol is downgraded.

Turning off a table feature does not result in a protocol downgrade. You must drop the feature to fully remove it from the protocol. Not all table features can be dropped. See Drop a Delta Lake table feature and downgrade table protocol.

All protocol change operations conflict with concurrent writes. Streaming reads fail when they encounter a commit that changes table metadata. To continue, restart the affected streams. For recommended methods, see Production considerations for Structured Streaming.

What triggers a protocol upgrade

When you enable a feature, the table protocol is automatically upgraded.

Protocol upgrades are triggered in the following ways:

• Enable automatically based on the syntax used in CREATE or ALTER statements. For example, CLUSTER BY in a CREATE TABLE statement automatically enables liquid clustering, and GENERATED ALWAYS AS enables generated columns.
• Enable explicitly through table properties. For example, setting 'delta.enableDeletionVectors' = true enables deletion vectors.
• Enabling a feature might automatically enable required features. For example, enabling UniForm automatically enables column mapping, and enabling liquid clustering automatically enables checkpoint V2. Review the relevant Azure Databricks documentation to determine which table features a given feature requires.

Reader features upgrade both the read and write protocols. For example, column mapping is a reader feature and requires upgrading both protocols because data is stored differently in storage.

Writer features, such as CHECK constraints, only upgrade the write protocol.

Lowest possible protocol

Delta Lake resolves the table protocol to the lowest version that supports all enabled features. This can only lower minReaderVersion or minWriterVersion, never raise them. Table features are never removed automatically. Use DROP FEATURE to remove a table feature from the protocol.

If all enabled features are fully supported at a lower integer-based protocol version, the table might revert to that version and drop readerFeatures or writerFeatures from the protocol. This doesn't disable any features. Lower protocol versions increase compatibility because all clients must respect them.

Azure Databricks and Databricks Runtime compatibility

Azure Databricks introduces support for new Delta Lake features in Databricks Runtime releases:

• Tables written by a lower Databricks Runtime version have full read and write support in higher Databricks Runtime versions.
• Tables written by a higher Databricks Runtime version might use table features not supported in lower Databricks Runtime versions.
• Some features allow writes from lower Databricks Runtime versions without fully applying all optimizations for the enabled feature.

If you only use Delta Lake tables through Azure Databricks, you only need to track feature support using minimum Databricks Runtime requirements. If you read or write tables from external systems, you must verify that those clients support the table features enabled on your tables.

Backported table feature support

In Databricks Runtime 12.2 LTS and above, table features replaced the integer-based protocol with granular flags that indicate which features a table uses.

Azure Databricks has backported table feature support to Databricks Runtime 11.3 LTS and below, instead of only supporting integer protocol versions, but only for features already supported in that release.

For example, you can read from and write to a table with generated columns enabled using table features in Databricks Runtime 9.1 LTS. However, you can't use Databricks Runtime 9.1 LTS to read from and write to a table with identity columns enabled using table features because identity columns requires Databricks Runtime 10.4 LTS and above.

When using table features with backported support, some operations available in a given Databricks Runtime version might not be available in the corresponding OSS Delta Lake version. If your architecture includes OSS Delta Lake clients, test compatibility before enabling table features on production tables.

Delta Lake features and required Databricks Runtime versions

The following table lists the lowest Databricks Runtime version with full support for each feature, such that all generally available capabilities for both reads and writes are supported.

See Databricks Runtime release notes versions and compatibility.

Features by protocol version

Delta Lake uses separate minReaderVersion and minWriterVersion values to specify protocol capabilities. The open source Delta Lake protocol has standardized on table features, but some clients still use legacy protocol versioning. Because some clients might not support all features, Databricks recommends that you verify with your client documentation and test compatibility before enabling new features on production tables.

Apache Iceberg uses a single format-version instead of separate reader and writer versions. An Iceberg format version indicates which features are available but doesn't mandate their use. Features are opt-in, except for row tracking which is mandatory in format version 3. When a feature shows N/A in the Iceberg column, it's a Delta-specific feature without a direct Iceberg equivalent.

The following table lists protocol version requirements for Delta Lake and Apache Iceberg table features. The feature type indicates whether a feature must be respected for writes only or for both reads and writes.

(1): Liquid clustering implements hidden partitioning.

(2): Requires column mapping to be enabled.