> ## Documentation Index
> Fetch the complete documentation index at: https://rajanand.org/llms.txt
> Use this file to discover all available pages before exploring further.

# Spark: drop function

The `drop()` command in Spark is used to remove one or more columns from a DataFrame. This is particularly useful when you need to clean up your dataset by removing unnecessary or redundant columns.

***

## 1. **Syntax**

**PySpark:**

```python theme={"system"}
df.drop(*cols)
```

**Spark SQL:**

```sql theme={"system"}
SELECT col1, col2, ..., colN FROM table_name;
```

(Exclude the columns you want to drop from the `SELECT` statement.)

## 2. **Parameters**

* **cols**: A list of column names (as strings) or column objects to be dropped from the DataFrame.

## 3. **Return Type**

* Returns a new DataFrame with the specified columns removed.

## 4. **Examples**

### **Example 1: Dropping a Single Column**

**PySpark:**

```python theme={"system"}
from pyspark.sql import SparkSession

spark = SparkSession.builder.appName("DropExample").getOrCreate()

data = [("Anand", 25, "M"), ("Bala", 30, "F"), ("Kavitha", 28, "F"), ("Raj", 35, "M")]
columns = ["Name", "Age", "Gender"]

df = spark.createDataFrame(data, columns)

# Dropping the 'Gender' column
df_dropped = df.drop("Gender")
df_dropped.show()
```

**Spark SQL:**

```sql theme={"system"}
SELECT Name, Age FROM people;
```

**Output:**

```
+-------+---+
|   Name|Age|
+-------+---+
|  Anand| 25|
|   Bala| 30|
| Kavitha| 28|
|    Raj| 35|
+-------+---+
```

### **Example 2: Dropping Multiple Columns**

**PySpark:**

```python theme={"system"}
# Dropping multiple columns
df_dropped = df.drop("Age", "Gender")
df_dropped.show()
```

**Spark SQL:**

```sql theme={"system"}
SELECT Name FROM people;
```

**Output:**

```
+-------+
|   Name|
+-------+
|  Anand|
|   Bala|
| Kavitha|
|    Raj|
+-------+
```

### **Example 3: Dropping Columns Using a List**

**PySpark:**

```python theme={"system"}
# Dropping columns using a list
columns_to_drop = ["Age", "Gender"]
df_dropped = df.drop(*columns_to_drop)
df_dropped.show()
```

**Spark SQL:**

```sql theme={"system"}
SELECT Name FROM people;
```

**Output:**

```
+-------+
|   Name|
+-------+
|  Anand|
|   Bala|
| Kavitha|
|    Raj|
+-------+
```

### **Example 4: Dropping Columns with Special Characters**

**PySpark:**

```python theme={"system"}
# Dropping a column with special characters
df_with_special_chars = df.withColumnRenamed("Gender", "Gender/Sex")
df_dropped = df_with_special_chars.drop("Gender/Sex")
df_dropped.show()
```

**Spark SQL:**

```sql theme={"system"}
SELECT Name, Age FROM people;
```

**Output:**

```
+-------+---+
|   Name|Age|
+-------+---+
|  Anand| 25|
|   Bala| 30|
| Kavitha| 28|
|    Raj| 35|
+-------+---+
```

### **Example 5: Dropping Columns in a DataFrame with Nested Structures**

**PySpark:**

```python theme={"system"}
from pyspark.sql.types import StructType, StructField, StringType, IntegerType

schema = StructType([
    StructField("Name", StringType(), True),
    StructField("Age", IntegerType(), True),
    StructField("Address", StructType([
        StructField("City", StringType(), True),
        StructField("State", StringType(), True)
    ]), True)
])

data = [("Anand", 25, ("Chennai", "Tamil Nadu")), 
        ("Bala", 30, ("Bangalore", "Karnataka")), 
        ("Kavitha", 28, ("Hyderabad", "Telangana"))]

df = spark.createDataFrame(data, schema)

# Dropping nested columns
df_dropped = df.drop("Address.City")
df_dropped.show(truncate=False)
```

**Output:**

```
+-------+---+------------+
|Name   |Age|Address     |
+-------+---+------------+
|Anand  |25 |{Tamil Nadu}|
|Bala   |30 |{Karnataka} |
|Kavitha|28 |{Telangana} |
+-------+---+------------+
```

### **Example 6: Dropping Columns Dynamically**

**PySpark:**

```python theme={"system"}
# Dropping columns dynamically
columns_to_drop = ["Age", "Gender"]

for column in columns_to_drop:
    df = df.drop(column)

df.show()
```

**Output:**

```
+-------+
|   Name|
+-------+
|  Anand|
|   Bala|
| Kavitha|
|    Raj|
+-------+
```

## 5. **Common Use Cases**

* Removing sensitive or irrelevant data before sharing or analysis.
* Preparing data for machine learning by removing irrelevant features.
* Cleaning up data after joins or transformations to remove redundant columns.

## 6. **Performance Considerations**

* Dropping columns is a lightweight operation as it only changes metadata and does not involve data transformation.
* Use this command to avoid data duplication or unnecessary transformations.

## 7. **Key Takeaways**

1. The `drop()` command is used to remove one or more columns from a DataFrame.
2. It allows you to drop columns by specifying their names or using a list of column names.
3. Dropping columns is a metadata operation and does not involve data movement, making it very efficient.
4. In Spark SQL, similar functionality can be achieved by excluding columns from the `SELECT` statement.
5. Works efficiently on large datasets as it does not involve data transformation.