​
Unity Catalog

  • It is a new governance solution for the databricks platform

What is Unity catalog?

  • Centralized governance solution across all your workspaces on any cloud.

  • Unify governance for all data and AI assets

    • files, talbes, ML models and dashboards
    • based on SQL

​
Architecture

Before unity catalog:

  • Workspace 1

    • user/group management
    • Hive Metastore
    • Access controls
    • Compute resources

  • Workspace 2

    • user/group management
    • Hive Metastore
    • Access controls
    • Compute resources

After unity catalog:

  • Unity Catalog

    • User/group management
    • UC metastores
    • Access controls

  • Workspace 1

    • Compute resources

  • Workspace 2

    • Compute resources

Workspaces are connected to the unity catalog.

​
3 level name space

SELECT * FROM schema.table ==> SELECT * FROM catalog.schema.table

Unity catalog hierarchy:

  • UC metastore
    • Storage credentials
    • External locations
    • Share
    • Recipient
    • Catalog
      • Schema (database)
        • Table
        • View
        • Function

Unity catalog metastore != Hive metastore

  • The hive metastore is the default metastore linked to each databricks workspace.
  • UC metastore offers improved security and advanced features.
  • UC metastore can have as many catalogs as desired.
  • UC supports authn to the underlying storage directories through storage credentials.

Three types of identities/principles:

  • Users: identified by e-mail addresses. Ex. account administrator
  • Groups: grouping of users and service principles. Ex. data engineers, data scientists, data analysts
    • Nested groups is possible
  • Service principals
    • Service principals are used to authenticate applications and services to access data in UC.
    • It is identified by Application id
    • It is recommended to use service principals instead of users for applications and services.

Identity Federation:

  • Identities are created at Account console level. Then identities can be assigned to different workspaces.

Privileges:

  • Privileges are assigned to users/groups/service principals at the catalog/schema/table level.
  • Create - Create a new object
  • Usage - Read the object
  • Select - Read the data in the object
  • Modify - Update the object
  • Read Files
  • Write Files
  • Execute

​
Security model (Unity catalog)

GRANT <Privilege> ON <Securable object> TO <Principle>

  1. Privileges
  2. Securable objects
  3. Principles

Unity catalog uses different security mode than hive metastore for granting the Privileges.

  • Once UC is enabled, the hive metastore is still accessible.
  • UC is not a replacement for the hive metastore. It is an additional layer of security and governance on top of the hive metastore.
  • Regardless of the UC metastore assigned to the workspace, the catalog named hive_metastore provides access to the hive metastore local to that workspace.

UC features:

  • Centralized governance
  • Automated lineage
  • Built-in data search and discovery
  • No hard migration required

Account console: https://accounts.cloud.databricks.com

  • Account admin can create UC metastores and assign them to workspaces.
  • Account admin can create catalogs and assign them to UC metastores.
  • Account admin can create external locations and assign them to UC metastores.
  • Account admin can create storage credentials and assign them to UC metastores.
  • Account admin can create shares and assign them to UC metastores.
  • Account admin can create recipients and assign them to UC metastores.