Source Systems

​

Introduction to Source Systems

​

Key Concepts

1. Data Engineering Lifecycle:

• Stages: Data generation, ingestion, transformation, storage, serving, and monitoring.
• Importance of Ingestion: 80% of AI/ML work involves data engineering, yet often overlooked.

2. Source Systems:

• Types:
  • Databases: Relational (SQL) and non-relational (NoSQL).
  • Files: Structured (CSV), semi-structured (JSON), unstructured (text, images, videos).
  • Streaming Systems: Continuous data flow (e.g., IoT devices, logs).

3. Data Types:

• Structured: Tabular data (rows and columns).
• Semi-structured: JSON, XML (key-value pairs, nested structures).
• Unstructured: Text, images, audio, video.

​

Databases

1. Relational Databases:

• Structure: Tables with rows and columns, linked by keys.
• CRUD Operations: Create, Read, Update, Delete.
• Normalization: Minimizes redundancy by splitting data into related tables.
• Primary Key: Uniquely identifies each row.
• Foreign Key: Links tables by referencing a primary key.
• SQL: Standard language for querying relational databases (e.g., SELECT, JOIN, WHERE).

2. NoSQL Databases:

• Types: Key-value, document, wide-column, graph.
• Flexibility: No fixed schema, supports unstructured/semi-structured data.
• Scalability: Horizontal scaling across multiple servers.
• Eventual Consistency: Data may not be immediately consistent across nodes.
• Examples: MongoDB (document store), DynamoDB (key-value store).

3. ACID Principles:

• Atomicity: Transactions are all-or-nothing.
• Consistency: Data remains valid after transactions.
• Isolation: Concurrent transactions don’t interfere.
• Durability: Completed transactions are permanent.

​

Files and Object Storage

1. Files:

• Types: CSV, JSON, text, images, videos.
• Source Systems: File systems (Google Drive), object storage (Amazon S3, ADLS, GCS).

2. Object Storage:

• Structure: Flat, no hierarchy (despite folder-like UIs).
• Key Features:
  • UUID: Unique identifier for each object.
  • Metadata: Additional info (e.g., creation date, file type).
  • Immutable: Objects can’t be updated; new versions are created.
• Use Cases: Data lakes, machine learning datasets.

​

Streaming Systems

1. Logs:

• Definition: Append-only records of events (e.g., user activity, system errors).
• Use Cases: Monitoring, debugging, anomaly detection.
• Log Levels: Debug, Info, Warn, Error, Fatal.

2. Streaming Data:

• Components:
  • Event Producer: Generates messages (e.g., IoT devices, APIs).
  • Event Consumer: Processes messages (e.g., payment systems, inventory updates).
  • Event Router: Distributes messages (e.g., Apache Kafka, Amazon Kinesis).
• Types:
  • Message Queues: Temporary storage (e.g., Amazon SQS).
  • Streaming Platforms: Persistent logs (e.g., Apache Kafka).

​

DataOps and Orchestration

1. DataOps:

• Focus: Automation, monitoring, and quality assurance in data pipelines.
• Tools: Infrastructure as code, monitoring tools.

2. Orchestration:

• DAGs: Directed Acyclic Graphs for workflow management (e.g., Apache Airflow).
• Tasks: Automating pipeline tasks, ensuring data quality.

---

​

Connecting to Source Systems

1. Methods of Connection:

• AWS Management Console: Manual, not repeatable or traceable.
• Command Line Interface (CLI): More programmatic but still manual.
• SDKs (e.g., Boto3): Programmatic and repeatable, ideal for automation.
• API Connectors (e.g., JDBC, ODBC): Used to connect applications to databases.

2. Key Components for Connection:

• Endpoint: The address of the resource (e.g., database endpoint).
• Port: The communication endpoint.
• Credentials: Username, password, or access keys for authentication.

3. Best Practices:

• Avoid manual processes for repeatability and traceability.
• Use SDKs or APIs for automation and scalability.

​

Identity and Access Management

1. Identity and Access Management (IAM) Basics:

• Purpose: Manage permissions for accessing cloud resources.
• Principle of Least Privilege: Grant only the necessary permissions for a limited time.

2. IAM Components:

• Root User: The account creator with full access.
• IAM Users: Individuals with specific permissions.
• IAM Groups: Collections of users with shared permissions.
• IAM Roles: Temporary permissions for users, applications, or services.
• Policies: JSON documents defining permissions for resources.

3. Common IAM Issues:

• Misconfigured permissions.
• Expired temporary credentials.
• Storing credentials insecurely (e.g., public GitHub repositories).

4. Best Practices:

• Use roles for temporary access instead of long-term credentials.
• Regularly review and update IAM policies.
• Avoid granting excessive permissions.

​

Networking in the Cloud

1. Key Concepts:

• Regions and Availability Zones (AZs):
  • Regions: Geographic areas with multiple AZs.
  • AZs: Isolated data centers within a region.
• Virtual Private Cloud (VPC): A private network within AWS.
• Subnets: Divisions within a VPC for organizing resources.
• Internet Gateway: Allows public subnets to connect to the internet.
• NAT Gateway: Allows private subnets to access the internet without exposing them to inbound traffic.

2. Route Tables: Define how traffic is routed within a VPC.

• Public Subnets: Route traffic to the internet via an internet gateway.
• Private Subnets: Route traffic to the internet via a NAT gateway.

3. Security Groups and Network ACLs:

• Security Groups:
  • Act as virtual firewalls at the instance level.
  • Stateful: Allow return traffic automatically.
  • Commonly used for EC2 instances, RDS databases, and load balancers.
• Network ACLs:
  • Provide subnet-level security.
  • Stateless: Require explicit inbound and outbound rules.
  • Useful for granular control over traffic.

4. Common Networking Issues:

• Misconfigured route tables.
• Missing or incorrect security group rules.
• Incorrect subnet associations.

5. Best Practices:

• Use public subnets for internet-facing resources (e.g., load balancers).
• Use private subnets for internal resources (e.g., databases).
• Regularly review and update security group and network ACL rules.

​

Troubleshooting Connectivity Issues

1. Steps to Debug:

• Verify the VPC has an internet gateway attached.
• Check route tables for correct routing rules.
• Ensure subnets are associated with the correct route tables.
• Review security groups for necessary inbound/outbound rules.
• Check network ACLs for traffic restrictions.
• Confirm instances are in the correct subnets and associated with the right security groups.

2. Common Scenarios:

• Access Denied: Check IAM permissions and credentials.
• No Internet Access: Verify route tables and NAT gateway configurations.
• Connection Timeout: Check security groups and network ACLs.

---

​

Key Takeaways

• Data Ingestion: Critical for building robust data pipelines.
• Source Systems: Databases, files, and streaming systems are the primary sources.
• Data Types: Structured, semi-structured, and unstructured data require different handling.
• Tools: SQL, NoSQL, object storage (S3), streaming platforms (Kafka), and orchestration tools (Airflow) are essential for data engineers.
• ACID Compliance: Ensures data integrity in transactional systems.
• IAM: Central to managing access and permissions in cloud-based architectures.
• Networking: Understanding VPCs, subnets, route tables, and security groups is critical for building and troubleshooting data pipelines.

---

Source: DeepLearning.ai source systems course.