Data Engineering Undercurrents

​

Overview

• Undercurrents:
  • Practices that apply across the entire data engineering lifecycle:
    1. Security
    2. Data Management
    3. Data Architecture
    4. DataOps
    5. Orchestration
    6. Software Engineering

​

Security

• Core Principle: Protect sensitive data (e.g., personal, proprietary).
• Key Practices:
  • Principle of Least Privilege: Grant users/applications only the access they need.
  • Data Sensitivity: Avoid ingesting sensitive data unless absolutely necessary.
  • Cloud Security: Understand IAM (Identity and Access Management), encryption, and networking protocols.
• Cultural Aspect:
  • Security is a shared responsibility across the organization.
  • Avoid security theater (superficial compliance without a true security culture).
• Common Mistakes:
  • Exposing S3 buckets or databases to the public internet.
  • Ignoring basic precautions like secure password sharing.
• Key Takeaway: Security is about principles, protocols, and people.

​

Data Management

• Definition: The development, execution, and supervision of plans to deliver, control, protect, and enhance the value of data.
• DAMA (Data Management Association):
  • Provides the Data Management Body of Knowledge (DMBOK).
  • Covers 11 knowledge areas, including data governance, data modeling, and data integration.
• Data Governance:
  • Ensures data quality, integrity, security, and usability.
  • Central to all other data management areas.
• Data Quality:
  • High-quality data is accurate, complete, discoverable, and timely.
  • Poor data quality leads to wasted time, poor decisions, and loss of trust.
• Key Takeaway: Data management ensures data is a valuable business asset.

​

Data Architecture

• Definition: The design of systems to support the evolving data needs of an enterprise through flexible and reversible decisions.
• Key Principles:
  1. Choose Common Components Wisely: Use components that facilitate collaboration.
  2. Plan for Failure: Design for both success and failure scenarios.
  3. Architect for Scalability: Build systems that scale up and down with demand.
  4. Architecture is Leadership: Think like an architect to lead and mentor others.
  5. Always Be Architecting: Continuously evolve systems to meet changing needs.
  6. Build Loosely Coupled Systems: Use interchangeable components for flexibility.
  7. Make Reversible Decisions: Ensure design choices can be easily changed.
  8. Prioritize Security: Apply security principles like least privilege and zero trust.
  9. Embrace FinOps: Optimize costs while maximizing revenue potential.
• Key Takeaway: Good data architecture is flexible, scalable, and secure.

​

DataOps

• Definition: A set of cultural habits and practices borrowed from DevOps to improve the development and quality of data products.
• Key Pillars:
  1. Automation:
     • Use CI/CD (Continuous Integration/Continuous Delivery) for data pipelines.
     • Automate tasks like ingestion, transformation, and serving.
  2. Observability and Monitoring:
     • Monitor pipelines to detect failures early.
     • Avoid bad data lingering in reports or dashboards.
  3. Incident Response:
     • Rapidly identify and resolve issues.
     • Foster open and blameless communication.
• Key Takeaway: DataOps improves efficiency, quality, and reliability of data systems.

​

Orchestration

• Definition: Coordinating and managing tasks in data pipelines.
• Approaches:
  1. Manual Execution: Useful for prototyping but not sustainable.
  2. Pure Scheduling: Automate tasks at specific times but lacks dependency management.
  3. Orchestration Frameworks:
     • Tools like Apache Airflow, Dagster, Prefect, and Mage.
     • Automate tasks with dependencies and monitoring.
• Directed Acyclic Graphs (DAGs):
  • Represent data pipelines as flowcharts with nodes (tasks) and edges (dependencies).
  • Ensure data flows in one direction without loops.
• Key Takeaway: Orchestration frameworks automate and optimize data pipelines.

​

Software Engineering

• Core Skill: Write clean, readable, testable, and deployable code.
• Languages and Frameworks:
  • SQL, Python, Bash, Spark, Kafka, Java, Scala, Rust, Go.
• Key Areas:
  • Data Processing: Write code for ingestion, transformation, and serving.
  • Open Source Contributions: Contribute to frameworks like Apache Airflow.
  • Infrastructure as Code: Automate infrastructure setup using code.
• Key Takeaway: Strong software engineering skills are essential for adding value as a data engineer.

​

Key Takeaways

1. Undercurrents:
   • Security, Data Management, DataOps, Data Architecture, Orchestration, and Software Engineering are foundational to data engineering.
2. Security:
   • Protect data through least privilege, encryption, and a security-first culture.
3. Data Management:
   • Ensure data is high-quality, secure, and usable through governance and best practices.
4. Data Architecture:
   • Design flexible, scalable, and secure systems that evolve with business needs.
5. DataOps:
   • Automate, monitor, and respond to incidents to improve efficiency and reliability.
6. Orchestration:
   • Use frameworks like Apache Airflow to automate and manage complex data pipelines.
7. Software Engineering:
   • Write production-grade code to build and maintain robust data systems.

Source: DeepLearning.ai data engineering course.