Skip to main content
  1. Posts/

How to get or set Databricks spark configuration

··121 words·1 min·
Note Spark Databricks
Table of Contents

In this post, I summarize how to get or set a Databricks spark configuration/property.

get the value for a specific setting/configuration
#

To get all configurations in Python:

from pyspark.sql import SparkSession

spark = SparkSession.builder.getOrCreate()

all_conf = spark.sparkContext.getConf().getAll()

This will show all the configurations available. To get the value for a specific conf, e.g., for ‘spark.databricks.clusterUsageTags.region’, use the following code instead:

spark.conf.get("spark.databricks.clusterUsageTags.region")

refs:

Check Databricks version
#

To check the Databricks runtime version, use the following code:

from pyspark.sql import SparkSession
spark = SparkSession.builder.getOrCreate()

spark.conf.get("spark.databricks.clusterUsageTags.sparkVersion")

ref:

Set the value for a configuration
#

To set a specific configuration for spark, we can use:

spark.conf.set("spark.sql.session.timeZone", "Asia/Shanghai")

Related

Change Timezone in Databricks Spark
··382 words·2 mins
Note Databricks Spark
How to Download Files from Google Cloud Storage in the Databricks Workspace Notebook
··551 words·3 mins
Note Databricks GCP Ubuntu
Databricks Cli Usage
·141 words·1 min
Note Databricks