The withColumnRenamed() command in Spark is used to rename an existing column in a DataFrame. This is particularly useful when you need to standardize column names, make them more descriptive, or align them with a specific naming convention.

​
1. Syntax

PySpark:

df.withColumnRenamed(existing_col_name, new_col_name)

Spark SQL:

SELECT existing_col_name AS new_col_name FROM table_name;

​
2. Parameters

  • existing_col_name: The current name of the column you want to rename.
  • new_col_name: The new name you want to assign to the column.

​
3. Return Type

  • Returns a new DataFrame with the renamed column.

​
4. Examples

​
Example 1: Renaming a Single Column

PySpark:

from pyspark.sql import SparkSession

spark = SparkSession.builder.appName("WithColumnRenamedExample").getOrCreate()

data = [("Anand", 25), ("Bala", 30), ("Kavitha", 28), ("Raj", 35)]
columns = ["Name", "Age"]

df = spark.createDataFrame(data, columns)

# Renaming the 'Age' column to 'Years'
df_renamed = df.withColumnRenamed("Age", "Years")
df_renamed.show()

Spark SQL:

SELECT Name, Age AS Years FROM people;

Output:

+-------+-----+
|   Name|Years|
+-------+-----+
|  Anand|   25|
|   Bala|   30|
| Kavitha|   28|
|    Raj|   35|
+-------+-----+

​
Example 2: Renaming Multiple Columns

PySpark:

# Renaming multiple columns
df_renamed = df.withColumnRenamed("Name", "FullName") \
               .withColumnRenamed("Age", "Years")
df_renamed.show()

Spark SQL:

SELECT Name AS FullName, Age AS Years FROM people;

Output:

+--------+-----+
|FullName|Years|
+--------+-----+
|   Anand|   25|
|    Bala|   30|
|  Kavitha|   28|
|     Raj|   35|
+--------+-----+

​
Example 3: Renaming Columns with Special Characters

PySpark:

# Renaming a column with special characters
df_with_special_chars = df.withColumnRenamed("Name", "Full Name")
df_with_special_chars.show()

Spark SQL:

SELECT Name AS `Full Name`, Age FROM people;

Output:

+--------+---+
|Full Name|Age|
+--------+---+
|   Anand| 25|
|    Bala| 30|
|  Kavitha| 28|
|     Raj| 35|
+--------+---+

​
Example 4: Renaming Columns in a DataFrame with Multiple Columns

PySpark:

data = [("Anand", 25, "M"), ("Bala", 30, "F"), ("Kavitha", 28, "F"), ("Raj", 35, "M")]
columns = ["Name", "Age", "Gender"]

df = spark.createDataFrame(data, columns)

# Renaming multiple columns
df_renamed = df.withColumnRenamed("Name", "FullName") \
               .withColumnRenamed("Age", "Years") \
               .withColumnRenamed("Gender", "Sex")
df_renamed.show()

Spark SQL:

SELECT Name AS FullName, Age AS Years, Gender AS Sex FROM people;

Output:

+--------+-----+---+
|FullName|Years|Sex|
+--------+-----+---+
|   Anand|   25|  M|
|    Bala|   30|  F|
|  Kavitha|   28|  F|
|     Raj|   35|  M|
+--------+-----+---+

​
Example 5: Renaming Columns in a DataFrame with Nested Structures

PySpark:

from pyspark.sql.types import StructType, StructField, StringType, IntegerType

schema = StructType([
    StructField("Name", StringType(), True),
    StructField("Age", IntegerType(), True),
    StructField("Address", StructType([
        StructField("City", StringType(), True),
        StructField("State", StringType(), True)
    ]), True)
])

data = [("Anand", 25, ("Chennai", "Tamil Nadu")), 
        ("Bala", 30, ("Bangalore", "Karnataka")), 
        ("Kavitha", 28, ("Hyderabad", "Telangana"))]

df = spark.createDataFrame(data, schema)

# Renaming nested columns
df_renamed = df.withColumnRenamed("Address.City", "City") \
               .withColumnRenamed("Address.State", "State")
df_renamed.show(truncate=False)

Output:

+-------+---+------------+
|Name   |Age|Address     |
+-------+---+------------+
|Anand  |25 |{Chennai, Tamil Nadu}|
|Bala   |30 |{Bangalore, Karnataka}|
|Kavitha|28 |{Hyderabad, Telangana}|
+-------+---+------------+

​
Example 6: Renaming Columns with Dynamic Names

PySpark:

# Renaming columns dynamically
new_column_names = {"Name": "FullName", "Age": "Years"}

for old_name, new_name in new_column_names.items():
    df = df.withColumnRenamed(old_name, new_name)

df.show()

Output:

+--------+-----+
|FullName|Years|
+--------+-----+
|   Anand|   25|
|    Bala|   30|
|  Kavitha|   28|
|     Raj|   35|
+--------+-----+

​
5. Common Use Cases

  • Renaming columns to match a specific schema or data model.
  • Preparing data for joins or merges by ensuring consistent column names.
  • Improving the readability of column names for reporting or analysis.

​
6. Performance Considerations

  • Renaming columns is a lightweight operation as it only changes metadata and does not involve data transformation.
  • Use this command to avoid data duplication or unnecessary transformations.

​
7. Key Takeaways

  1. The withColumnRenamed() command is used to rename one or more columns in a DataFrame.
  2. It allows you to rename columns to make them more descriptive or align them with a specific naming convention.
  3. Renaming columns is a metadata operation and does not involve data movement, making it very efficient.
  4. In Spark SQL, similar renaming can be achieved using AS in SELECT statements.
  5. Works efficiently on large datasets as it does not involve data transformation.