In this post, I summarize how to get or set a Databricks spark configuration/property.
get the value for a specific setting/configuration #
To get all configurations in Python:
from pyspark.sql import SparkSession
spark = SparkSession.builder.getOrCreate()
all_conf = spark.sparkContext.getConf().getAll()
This will show all the configurations available. To get the value for a specific conf, e.g., for ‘spark.databricks.clusterUsageTags.region’, use the following code instead:
spark.conf.get("spark.databricks.clusterUsageTags.region")
refs:
- get all spark conf: https://stackoverflow.com/a/45029640/6064933
- get or set specific conf, https://kb.databricks.com/data/get-and-set-spark-config
Check Databricks version #
To check the Databricks runtime version, use the following code:
from pyspark.sql import SparkSession
spark = SparkSession.builder.getOrCreate()
spark.conf.get("spark.databricks.clusterUsageTags.sparkVersion")
ref:
- check databricks spark version: https://stackoverflow.com/q/53740967/6064933
Set the value for a configuration #
To set a specific configuration for spark, we can use:
spark.conf.set("spark.sql.session.timeZone", "Asia/Shanghai")