How to get or set Databricks spark configuration
Contents
In this post, I summarize how to get or set a Databricks spark configuration/property.
get the value for a specific setting/configuration
To get all configurations in Python:
from pyspark.sql import SparkSession
spark = SparkSession.builder.getOrCreate()
all_conf = spark.sparkContext.getConf().getAll()
This will show all the configurations available. To get the value for a specific conf, e.g., for ‘spark.databricks.clusterUsageTags.region’, use the following code instead:
spark.conf.get("spark.databricks.clusterUsageTags.region")
refs:
- get all spark conf: https://stackoverflow.com/a/45029640/6064933
- get or set specific conf, https://kb.databricks.com/data/get-and-set-spark-config
Check Databricks version
To check the Databricks runtime version, use the following code:
from pyspark.sql import SparkSession
spark = SparkSession.builder.getOrCreate()
spark.conf.get("spark.databricks.clusterUsageTags.sparkVersion")
ref: