How to get or set Databricks spark configuration

In this post, I summarize how to get or set a Databricks spark configuration/property.

get the value for a specific setting/configuration

#

To get all configurations in Python:

from pyspark.sql import SparkSession

spark = SparkSession.builder.getOrCreate()

all_conf = spark.sparkContext.getConf().getAll()

This will show all the configurations available. To get the value for a specific conf, e.g., for ‘spark.databricks.clusterUsageTags.region’, use the following code instead:

spark.conf.get("spark.databricks.clusterUsageTags.region")

refs:

• get all spark conf: https://stackoverflow.com/a/45029640/6064933
• get or set specific conf, https://kb.databricks.com/data/get-and-set-spark-config

Check Databricks version

#

To check the Databricks runtime version, use the following code:

from pyspark.sql import SparkSession
spark = SparkSession.builder.getOrCreate()

spark.conf.get("spark.databricks.clusterUsageTags.sparkVersion")

ref:

• check databricks spark version: https://stackoverflow.com/q/53740967/6064933

Set the value for a configuration

#

To set a specific configuration for spark, we can use:

spark.conf.set("spark.sql.session.timeZone", "Asia/Shanghai")