Standard compute requirements and limitations

This page includes a list of requirements and limitations for standard compute. If you are using classic compute, Databricks recommends using standard access mode unless your workload is dependent on one of the limitations listed below.

Current standard compute limitations

The following sections list limitations for standard compute based on the most recent Databricks Runtime version. For limitations that apply to older Databricks Runtime versions, see Runtime-dependent limitations.

If these features are required for your workload, use dedicated compute instead.

General standard compute limitations

• Databricks Runtime for ML is not supported. Instead, install any ML library not bundled with the Databricks Runtime as a compute-scoped library.
• GPU-enabled compute is not supported.
• Spark-submit job tasks are not supported. Use a JAR task instead.
• DBUtils and other clients can only read from cloud storage using an external location.
• DBFS root and mounts do not support FUSE.

Language limitations

• R is not supported.

Spark API limitations

• Spark Context (sc), spark.sparkContext, and sqlContext are not supported for Scala:
  • Azure Databricks recommends using the spark variable to interact with the SparkSession instance.
  • The following sc functions are also not supported: emptyRDD, range, init_batched_serializer, parallelize, pickleFile, textFile, wholeTextFiles, binaryFiles, binaryRecords, sequenceFile, newAPIHadoopFile, newAPIHadoopRDD, hadoopFile, hadoopRDD, union, runJob, setSystemProperty, uiWebUrl, stop, setJobGroup, setLocalProperty, getConf.
• Setting certain Spark configuration properties is not supported. Creating or editing a cluster that sets a restricted property fails with an error. For the complete list, see Spark configuration limitations.
• When creating a DataFrame from local data using spark.createDataFrame, row sizes cannot exceed 128MB.
• RDD APIs are not supported.
• Spark Connect, which is used in more recent versions of Databricks Runtime, defers analysis and name resolution to execution time, which may change the behavior of your code. See Compare Spark Connect to Spark Classic.

UDF limitations

• Hive UDFs are not supported. Instead, use UDFs in Unity Catalog.
• Scala UDFs cannot be used inside higher-order functions.

Streaming limitations

• Working with socket sources is not supported.
• The sourceArchiveDir must be in the same external location as the source when you use option("cleanSource", "archive") with a data source managed by Unity Catalog.
• For Kafka sources and sinks, the following options are not supported:
  • kafka.sasl.client.callback.handler.class
  • kafka.sasl.login.callback.handler.class
  • kafka.sasl.login.class
  • kafka.partition.assignment.strategy
• Python foreachBatch does not support ThreadPoolExecutor or multi-threaded execution. Multi-threaded execution may not throw errors but can result in data corruption or inconsistent results.

Network and file system limitations

• Standard compute runs commands as a low-privilege user forbidden from accessing sensitive parts of the filesystem.
• POSIX-style paths (/) for DBFS are not supported.
• Only workspace admins and users with ANY FILE permissions can directly interact with files using DBFS.

• You cannot connect to the instance metadata service or Azure WireServer.

Environment variable limitations

Only a predefined set of environment variables is available to the Spark engine and init scripts on standard compute. Environment variables that you set on a cluster but that aren't in this set remain available to your user code, including UDFs, but aren't available to the Spark engine or init scripts.

This set includes common configuration, credential, and runtime variables. The following examples are not exhaustive:

• Networking and proxies: HTTP_PROXY, HTTPS_PROXY, NO_PROXY
• TLS certificates: REQUESTS_CA_BUNDLE, SSL_CERT_FILE
• AWS credentials and region: AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY, AWS_REGION, AWS_DEFAULT_REGION
• Azure credentials: AZURE_CLIENT_ID, AZURE_CLIENT_SECRET, AZURE_TENANT_ID
• Google Cloud credentials: GOOGLE_APPLICATION_CREDENTIALS
• Databricks connection: DATABRICKS_HOST, DATABRICKS_TOKEN
• Locale and time zone: TZ, LANG, LC_ALL
• Parallelism and threading: OMP_NUM_THREADS, OPENBLAS_NUM_THREADS, MKL_NUM_THREADS, NUMEXPR_NUM_THREADS
• Observability: DD_API_KEY, DD_SITE, DD_ENV
• Unity Catalog defaults: CATALOG, CATALOG_NAME

Scala kernel limitations

The following limitations apply when using the scala kernel on standard compute:

• Certain classes cannot be used in your code if they conflict with the internal almond kernel library, most notably Input. For a list of almond's defined imports, see almond imports.
• Logging directly to log4j is not supported.
• In the UI, the dataframe schema dropdown is not supported.
• If your driver hits OOM, the Scala REPL will not terminate.
• //connector/sql-aws-connectors:sql-aws-connectors is not in the Scala REPL's bazel target, use results in ClassNotFoundException.
• The Scala kernel is incompatible with SQLImplicits.

Runtime-dependent limitations

The following limitations have been resolved through runtime updates, but might still apply to your workload if you use an older runtime.

Language support

Spark API support

UDF support

Streaming support

Additionally, for Python, foreachBatch has the following behavior changes on Databricks Runtime 14.0 and above:

• print() commands write output to the driver logs.
• You cannot access the dbutils.widgets submodule inside the function.
• Any files, modules, or objects referenced in the function must be serializable and available on Spark.

Network and file system support

Spark configuration limitations

On Databricks Runtime 19 and above, you can't set the following Spark configuration properties in standard access mode. Creating or editing a cluster that sets any of these properties, or a property that begins with one of the listed prefixes (shown with a trailing .*), fails with an error. Remove these properties from your cluster configuration, compute policies, and job definitions before upgrading.