​
Overview

  • Undercurrents:
    • Practices that apply across the entire data engineering lifecycle:
      1. Security
      2. Data Management
      3. Data Architecture
      4. DataOps
      5. Orchestration
      6. Software Engineering

​
Security

  • Core Principle: Protect sensitive data (e.g., personal, proprietary).
  • Key Practices:
    • Principle of Least Privilege: Grant users/applications only the access they need.
    • Data Sensitivity: Avoid ingesting sensitive data unless absolutely necessary.
    • Cloud Security: Understand IAM (Identity and Access Management), encryption, and networking protocols.
  • Cultural Aspect:
    • Security is a shared responsibility across the organization.
    • Avoid security theater (superficial compliance without a true security culture).
  • Common Mistakes:
    • Exposing S3 buckets or databases to the public internet.
    • Ignoring basic precautions like secure password sharing.
  • Key Takeaway: Security is about principles, protocols, and people.

​
Data Management

  • Definition: The development, execution, and supervision of plans to deliver, control, protect, and enhance the value of data.
  • DAMA (Data Management Association):
    • Provides the Data Management Body of Knowledge (DMBOK).
    • Covers 11 knowledge areas, including data governance, data modeling, and data integration.
  • Data Governance:
    • Ensures data quality, integrity, security, and usability.
    • Central to all other data management areas.
  • Data Quality:
    • High-quality data is accurate, complete, discoverable, and timely.
    • Poor data quality leads to wasted time, poor decisions, and loss of trust.
  • Key Takeaway: Data management ensures data is a valuable business asset.

​
Data Architecture

  • Definition: The design of systems to support the evolving data needs of an enterprise through flexible and reversible decisions.
  • Key Principles:
    1. Choose Common Components Wisely: Use components that facilitate collaboration.
    2. Plan for Failure: Design for both success and failure scenarios.
    3. Architect for Scalability: Build systems that scale up and down with demand.
    4. Architecture is Leadership: Think like an architect to lead and mentor others.
    5. Always Be Architecting: Continuously evolve systems to meet changing needs.
    6. Build Loosely Coupled Systems: Use interchangeable components for flexibility.
    7. Make Reversible Decisions: Ensure design choices can be easily changed.
    8. Prioritize Security: Apply security principles like least privilege and zero trust.
    9. Embrace FinOps: Optimize costs while maximizing revenue potential.
  • Key Takeaway: Good data architecture is flexible, scalable, and secure.

​
DataOps

  • Definition: A set of cultural habits and practices borrowed from DevOps to improve the development and quality of data products.
  • Key Pillars:
    1. Automation:
      • Use CI/CD (Continuous Integration/Continuous Delivery) for data pipelines.
      • Automate tasks like ingestion, transformation, and serving.
    2. Observability and Monitoring:
      • Monitor pipelines to detect failures early.
      • Avoid bad data lingering in reports or dashboards.
    3. Incident Response:
      • Rapidly identify and resolve issues.
      • Foster open and blameless communication.
  • Key Takeaway: DataOps improves efficiency, quality, and reliability of data systems.

​
Orchestration

  • Definition: Coordinating and managing tasks in data pipelines.
  • Approaches:
    1. Manual Execution: Useful for prototyping but not sustainable.
    2. Pure Scheduling: Automate tasks at specific times but lacks dependency management.
    3. Orchestration Frameworks:
      • Tools like Apache Airflow, Dagster, Prefect, and Mage.
      • Automate tasks with dependencies and monitoring.
  • Directed Acyclic Graphs (DAGs):
    • Represent data pipelines as flowcharts with nodes (tasks) and edges (dependencies).
    • Ensure data flows in one direction without loops.
  • Key Takeaway: Orchestration frameworks automate and optimize data pipelines.

​
Software Engineering

  • Core Skill: Write clean, readable, testable, and deployable code.
  • Languages and Frameworks:
    • SQL, Python, Bash, Spark, Kafka, Java, Scala, Rust, Go.
  • Key Areas:
    • Data Processing: Write code for ingestion, transformation, and serving.
    • Open Source Contributions: Contribute to frameworks like Apache Airflow.
    • Infrastructure as Code: Automate infrastructure setup using code.
  • Key Takeaway: Strong software engineering skills are essential for adding value as a data engineer.

​
Key Takeaways

  1. Undercurrents:
    • Security, Data Management, DataOps, Data Architecture, Orchestration, and Software Engineering are foundational to data engineering.
  2. Security:
    • Protect data through least privilege, encryption, and a security-first culture.
  3. Data Management:
    • Ensure data is high-quality, secure, and usable through governance and best practices.
  4. Data Architecture:
    • Design flexible, scalable, and secure systems that evolve with business needs.
  5. DataOps:
    • Automate, monitor, and respond to incidents to improve efficiency and reliability.
  6. Orchestration:
    • Use frameworks like Apache Airflow to automate and manage complex data pipelines.
  7. Software Engineering:
    • Write production-grade code to build and maintain robust data systems.

Source: DeepLearning.ai data engineering course.