Spark Session Builder

The SparkSession.builder.getOrCreate() method is used to create a SparkSession object, which is the entry point to programming with Spark. A SparkSession provides a unified interface to work with structured data (DataFrames and Datasets) and allows you to configure various Spark properties. If a SparkSession already exists, getOrCreate() returns the existing one; otherwise, it creates a new one.

1. Syntax

PySpark:

spark = SparkSession.builder \
                    .appName("AppName") \
                    .config("key", "value") \
                    .getOrCreate()

2. Key Features

Entry Point: The SparkSession is the entry point to Spark functionality.
Singleton: Ensures that only one SparkSession exists per JVM (or Python process).
Configuration: Allows you to configure Spark properties (e.g., app name, master URL, memory settings).
Unified API: Provides access to DataFrames, Datasets, SQL, and streaming.

3. Parameters

appName: Sets the name of the Spark application.
config: Allows you to set Spark configuration properties (e.g., spark.executor.memory, spark.sql.shuffle.partitions).
master: Specifies the master URL (e.g., local, yarn, k8s).

4. Examples

Example 1: Creating a Basic SparkSession

PySpark:

from pyspark.sql import SparkSession

# Create a SparkSession
spark = SparkSession.builder \
                    .appName("MyApp") \
                    .getOrCreate()

# Print the SparkSession object
print(spark)

Output:

<pyspark.sql.session.SparkSession object at 0x7f8b1c2b3d90>

Example 2: Configuring Spark Properties

PySpark:

# Create a SparkSession with custom configurations
spark = SparkSession.builder \
                    .appName("MyApp") \
                    .config("spark.executor.memory", "2g") \
                    .config("spark.sql.shuffle.partitions", "100") \
                    .getOrCreate()

# Print the SparkSession object
print(spark)

Output:

<pyspark.sql.session.SparkSession object at 0x7f8b1c2b3d90>

Example 3: Using an Existing SparkSession

PySpark:

# Create a SparkSession
spark1 = SparkSession.builder \
                     .appName("MyApp") \
                     .getOrCreate()

# Try to create another SparkSession
spark2 = SparkSession.builder \
                     .appName("AnotherApp") \
                     .getOrCreate()

# Check if both SparkSession objects are the same
print(spark1 == spark2)  # Output: True

Output:

True

Example 4: Setting the Master URL

PySpark:

# Create a SparkSession with a local master URL
spark = SparkSession.builder \
                    .appName("MyApp") \
                    .master("local[*]") \
                    .getOrCreate()

# Print the SparkSession object
print(spark)

Output:

<pyspark.sql.session.SparkSession object at 0x7f8b1c2b3d90>

Example 5: Creating a SparkSession with Hive Support

PySpark:

# Create a SparkSession with Hive support
spark = SparkSession.builder \
                    .appName("MyApp") \
                    .enableHiveSupport() \
                    .getOrCreate()

# Print the SparkSession object
print(spark)

Output:

<pyspark.sql.session.SparkSession object at 0x7f8b1c2b3d90>

Example 6: Creating a SparkSession with a Specific Master URL

PySpark:

# Create a SparkSession with a specific master URL
spark = SparkSession.builder \
                    .appName("MyApp") \
                    .master("yarn") \
                    .getOrCreate()

# Print the SparkSession object
print(spark)

Output:

<pyspark.sql.session.SparkSession object at 0x7f8b1c2b3d90>

Example 7: Creating a SparkSession with Custom Logging

PySpark:

# Create a SparkSession with custom logging
spark = SparkSession.builder \
                    .appName("MyApp") \
                    .config("spark.logConf", "true") \
                    .getOrCreate()

# Print the SparkSession object
print(spark)

Output:

<pyspark.sql.session.SparkSession object at 0x7f8b1c2b3d90>

5. Common Use Cases

Creating a SparkSession to work with structured data (DataFrames and Datasets).
Configuring Spark properties for specific applications (e.g., memory, parallelism).
Ensuring a single SparkSession instance in an application.

6. Performance Considerations

Configuration: Properly configuring Spark properties (e.g., memory, parallelism) is essential for performance.
Singleton: Reusing an existing SparkSession avoids the overhead of creating a new one.

7. Key Takeaways

Purpose: The SparkSession.builder.getOrCreate() method is used to create or retrieve a SparkSession object.
Singleton: Ensures that only one SparkSession exists per JVM (or Python process).
Configuration: Allows you to configure Spark properties (e.g., app name, master URL, memory settings).
Performance: Creating a SparkSession is a lightweight operation, but configuring it properly is essential for performance.

​1. Syntax

​2. Key Features

​3. Parameters

​4. Examples

​Example 1: Creating a Basic SparkSession

​Example 2: Configuring Spark Properties

​Example 3: Using an Existing SparkSession

​Example 4: Setting the Master URL

​Example 5: Creating a SparkSession with Hive Support

​Example 6: Creating a SparkSession with a Specific Master URL

​Example 7: Creating a SparkSession with Custom Logging

​5. Common Use Cases

​6. Performance Considerations

​7. Key Takeaways

1. Syntax

2. Key Features

3. Parameters

4. Examples

Example 1: Creating a Basic SparkSession

Example 2: Configuring Spark Properties

Example 3: Using an Existing SparkSession

Example 4: Setting the Master URL

Example 5: Creating a SparkSession with Hive Support

Example 6: Creating a SparkSession with a Specific Master URL

Example 7: Creating a SparkSession with Custom Logging

5. Common Use Cases

6. Performance Considerations

7. Key Takeaways