The alias() function in Spark is used to rename a column or an expression in a DataFrame. It is particularly useful when you want to give a more meaningful name to a column, especially after performing transformations or aggregations. The alias() function can be applied to columns, expressions, or even entire DataFrames.

​
1. Syntax

PySpark:

column.alias(new_name)

Spark SQL:

SELECT column AS new_name FROM table_name;

​
2. Parameters

  • new_name: The new name to assign to the column or expression.

​
3. Return Type

  • Returns a Column object with the new name.

​
4. Examples

​
Example 1: Renaming a Column

PySpark:

from pyspark.sql import SparkSession
from pyspark.sql.functions import col

spark = SparkSession.builder.appName("AliasExample").getOrCreate()

# Create DataFrame
data = [("Anand", 25), ("Bala", 30), ("Kavitha", 28), ("Raj", 35)]
columns = ["Name", "Age"]

df = spark.createDataFrame(data, columns)

# Rename the 'Age' column to 'Years'
df_renamed = df.select(col("Name"), col("Age").alias("Years"))
df_renamed.show()

Spark SQL:

SELECT Name, Age AS Years 
FROM people;

Output:

+-------+-----+
|   Name|Years|
+-------+-----+
|  Anand|   25|
|   Bala|   30|
|Kavitha|   28|
|    Raj|   35|
+-------+-----+

​
Example 2: Renaming an Expression

PySpark:

from pyspark.sql.functions import expr

# Rename an expression (e.g., Age + 5)
df_renamed = df.select(col("Name"), (col("Age") + 5).alias("AgePlus5"))
df_renamed.show()

Spark SQL:

SELECT Name, Age + 5 AS AgePlus5 
FROM people;

Output:

+-------+--------+
|   Name|AgePlus5|
+-------+--------+
|  Anand|      30|
|   Bala|      35|
|Kavitha|      33|
|    Raj|      40|
+-------+--------+

​
Example 3: Renaming Multiple Columns

PySpark:

# Rename multiple columns
df_renamed = df.select(col("Name").alias("FullName"), col("Age").alias("Years"))
df_renamed.show()

Spark SQL:

SELECT Name AS FullName, Age AS Years 
FROM people;

Output:

+--------+-----+
| FullName|Years|
+--------+-----+
|   Anand|   25|
|    Bala|   30|
| Kavitha|   28|
|     Raj|   35|
+--------+-----+

​
Example 4: Renaming a DataFrame

PySpark:

# Rename a DataFrame (useful for joins)
df_renamed = df.alias("people_df")
df_renamed.show()

Spark SQL:

SELECT * FROM people AS people_df;

Output:

+-------+---+
|   Name|Age|
+-------+---+
|  Anand| 25|
|   Bala| 30|
|Kavitha| 28|
|    Raj| 35|
+-------+---+

​
Example 5: Using alias() with Aggregations

PySpark:

from pyspark.sql.functions import sum

# Rename an aggregated column
df_aggregated = df.groupBy("Name").agg(sum("Age").alias("TotalAge"))
df_aggregated.show()

Spark SQL:

SELECT Name, SUM(Age) AS TotalAge 
FROM people 
GROUP BY Name;

Output:

+-------+--------+
|   Name|TotalAge|
+-------+--------+
|  Anand|      25|
|   Bala|      30|
|Kavitha|      28|
|    Raj|      35|
+-------+--------+

​
Example 6: Renaming Columns in a Join

PySpark:

# Create another DataFrame
departments_data = [(101, "Sales"), (102, "HR"), (103, "Finance")]
departments_columns = ["DeptID", "DeptName"]

departments_df = spark.createDataFrame(departments_data, departments_columns)

# Rename columns before joining
df_renamed = df.withColumnRenamed("Age", "Years")
joined_df = df_renamed.join(departments_df, df_renamed["Years"] == departments_df["DeptID"])
joined_df.show()

Spark SQL:

SELECT * 
FROM (SELECT Name, Age AS Years FROM people) AS people_renamed
JOIN departments 
ON people_renamed.Years = departments.DeptID;

Output:

+-------+-----+------+--------+
|   Name|Years|DeptID|DeptName|
+-------+-----+------+--------+
|  Anand|   25|  null|    null|
|   Bala|   30|  null|    null|
|Kavitha|   28|  null|    null|
|    Raj|   35|  null|    null|
+-------+-----+------+--------+

​
Example 7: Renaming Columns in a Nested DataFrame

PySpark:

from pyspark.sql.functions import explode

# Create DataFrame with nested data
data = [("Anand", ["Java", "Python"]), 
        ("Bala", ["Scala", "Spark"]), 
        ("Kavitha", ["SQL", "Hadoop"])]
columns = ["Name", "Skills"]

df = spark.createDataFrame(data, columns)

# Explode and rename the 'Skills' column
df_exploded = df.select(col("Name"), explode("Skills").alias("Skill"))
df_exploded.show()

Spark SQL:

SELECT Name, explode(Skills) AS Skill 
FROM people;

Output:

+-------+------+
|   Name| Skill|
+-------+------+
|  Anand|  Java|
|  Anand|Python|
|   Bala| Scala|
|   Bala| Spark|
|Kavitha|   SQL|
|Kavitha|Hadoop|
+-------+------+

​
5. Common Use Cases

  • Renaming columns after transformations or aggregations.
  • Assigning meaningful names to derived columns.
  • Renaming DataFrames for clarity in joins or complex queries.

​
6. Performance Considerations

  • Using alias() is lightweight and does not involve data movement.
  • It is particularly useful for improving the readability of complex queries.

​
7. Key Takeaways

  1. The alias() function is used to rename columns, expressions, or DataFrames.
  2. In Spark SQL, similar functionality can be achieved using AS.