> ## Documentation Index
> Fetch the complete documentation index at: https://rajanand.org/llms.txt
> Use this file to discover all available pages before exploring further.

# Data Governance

## **1. The Four Areas of Data Governance**

### **What is Data Governance?**

A framework for managing data availability, usability, integrity, and security in an organization.

### **The Four Key Areas**

1. **Data Discovery**
   * Metadata management (e.g., tagging, searchable catalogs).
   * *Example*: Unity Catalog's searchable metadata.

2. **Data Quality**
   * Ensures accuracy via validation rules (e.g., `NOT NULL` constraints).
   * *Example*: Delta Lake schema enforcement.

3. **Data Security**
   * Access control (RBAC), encryption, masking.
   * *Example*: Unity Catalog’s fine-grained permissions.

4. **Data Lineage**
   * Tracks data flow from source to consumption.
   * *Example*: Databricks Lineage Tracking in notebooks.

***

## **2. Metastores vs. Catalogs**

### **Hive Metastore (Legacy)**

* **What**: Stores table metadata (schema, location) for Spark SQL.
* **Limitations**:
  * No centralized governance.
  * Limited to workspace-level access.

### **Unity Catalog (Modern)**

* **What**: A unified governance layer for data across workspaces/clouds.
* **Benefits**:
  * Centralized permissions (RBAC).
  * Three-level namespace (`catalog.schema.table`).

| **Feature**        | **Hive Metastore** | **Unity Catalog**            |
| ------------------ | ------------------ | ---------------------------- |
| **Scope**          | Workspace-local    | Multi-workspace, multi-cloud |
| **Access Control** | Basic (POSIX-like) | Fine-grained (RBAC)          |
| **Lineage**        | Limited            | Built-in                     |

***

## **3. Unity Catalog Securables**

### **What are Securables?**

Objects that can have permissions applied in Unity Catalog.

### **Key Securables**

1. **Catalog**: Top-level container (e.g., `sales_catalog`).
2. **Schema (Database)**: Logical grouping of tables (e.g., `sales_catalog.europe`).
3. **Table/View**: Data objects (e.g., `sales_catalog.europe.orders`).
4. **Function**: UDFs (e.g., `sales_catalog.utils.discount`).

### **Permission Examples**

```sql theme={"system"}
GRANT SELECT ON TABLE sales_catalog.europe.orders TO analyst_group;
GRANT USAGE ON SCHEMA sales_catalog.europe TO marketing_team;
```

***

## **4. Service Principals**

### **What is a Service Principal?**

A non-human identity (e.g., for automation) with controlled permissions.

### **Use Cases**

* CI/CD pipelines.
* Automated jobs (e.g., nightly ETL).

### **How to Create**

1. **Azure**: Via Azure Active Directory (AAD).
2. **AWS**: IAM roles.
3. **GCP**: Service accounts.

### **Best Practice**

* Use instead of personal accounts for production workflows.

***

## **5. Cluster Security Modes for Unity Catalog**

### **Compatible Modes**

1. **Single User**
   * Runs as a single identity (user or service principal).
   * *Use Case*: Jobs clusters.

2. **No Isolation Shared**
   * Multiple users, but no data isolation.
   * *Use Case*: All-purpose clusters with UC.

### **Incompatible Modes**

* **Shared with Table Access Control**: Conflicts with UC’s RBAC.

***

## **6. Creating a UC-Enabled All-Purpose Cluster**

### **Steps**

1. Navigate to **Compute** → **Create Cluster**.
2. Select:
   * **Access Mode**: "Single User" or "No Isolation Shared".
   * **Unity Catalog**: Attach a metastore.
3. Set permissions:
   ```sql theme={"system"}
   GRANT ALL PRIVILEGES ON CATALOG sales_catalog TO cluster_user;
   ```

### **Example Configuration**

| **Setting**   | **Value**                     |
| ------------- | ----------------------------- |
| Cluster Mode  | No Isolation Shared           |
| Unity Catalog | Enabled                       |
| Access Mode   | Assigned to service principal |

***

## **7. Creating a Databricks SQL (DBSQL) Warehouse**

### **What is a DBSQL Warehouse?**

A serverless compute endpoint for SQL queries.

### **Steps**

1. Navigate to **SQL** → **Warehouses** → **Create**.
2. Configure:
   * **Size**: 2X-Small to 4X-Large.
   * **Auto-stop**: 10 mins to 24 hours.
3. Attach to Unity Catalog:
   ```sql theme={"system"}
   GRANT USE_WAREHOUSE ON WAREHOUSE analytics_wh TO bi_team;
   ```

***

## **8. Querying a Three-Layer Namespace**

### **Syntax**

```sql theme={"system"}
SELECT * FROM catalog.schema.table;
```

### **Example**

```sql theme={"system"}
-- Query a table in 'prod' catalog, 'finance' schema
SELECT * FROM prod.finance.transactions;
```

### **Benefits**

* Avoids naming conflicts (e.g., `sales.transactions` vs. `hr.transactions`).

***

## **9. Implementing Data Object Access Control**

### **Steps**

1. **Grant Permissions**:
   ```sql theme={"system"}
   GRANT SELECT ON TABLE prod.finance.transactions TO finance_team;
   ```
2. **Revoke Permissions**:
   ```sql theme={"system"}
   REVOKE SELECT ON TABLE prod.finance.transactions FROM contractor;
   ```

### **Supported Privileges**

* `SELECT`, `MODIFY`, `CREATE`, `USAGE`, `ALL PRIVILEGES`.

***

## **10. Best Practices**

### **Colocate Metastores with Workspaces**

* **Why**: Reduces latency and simplifies management.
* **How**: Deploy metastore in the same region as workspaces.

### **Use Service Principals for Connections**

* **Why**: Avoid personal credentials in automation.
* **Example**:
  ```python theme={"system"}
  spark.read.table("prod.sales.orders")  # Authenticates via service principal
  ```

### **Segregate Business Units by Catalog**

* **Why**: Isolate data and permissions by department.
* **Example**:
  * `marketing_catalog`
  * `finance_catalog`

***

# **Summary Table: Key Concepts**

| Topic                        | Key Takeaway                                                             |
| ---------------------------- | ------------------------------------------------------------------------ |
| **Four Governance Areas**    | Discovery, Quality, Security, Lineage.                                   |
| **Unity Catalog Securables** | Catalogs, schemas, tables, functions.                                    |
| **Service Principals**       | Non-human identities for automation.                                     |
| **Three-Layer Namespace**    | `catalog.schema.table` for unified queries.                              |
| **Best Practices**           | Colocate metastores, use service principals, segregate catalogs by team. |