> ## Documentation Index
> Fetch the complete documentation index at: https://rajanand.org/llms.txt
> Use this file to discover all available pages before exploring further.

# Requirements Gathering

## **Introduction**

* **Goal**: Understand how to gather requirements and design data systems that meet stakeholder needs and business goals.
* **Key Concepts**:
  * **Hierarchy of Needs**: Business goals → Stakeholder needs → System requirements (functional and non-functional).
  * **Requirements Gathering**: Conversations with stakeholders to understand their needs and current systems.
  * **Documentation**: Clearly document functional and non-functional requirements.

## **Hierarchy of Needs**

1. **Business Goals**🚩:
   * High-level objectives (e.g., increase revenue, improve customer retention, expand to new markets).
   * Example: "The company aims to grow by launching new products and improving customer retention."
2. **Stakeholder Needs**:
   * What stakeholders (e.g., marketing, data scientists) need to achieve business goals.
   * Example: "Marketing needs real-time dashboards to monitor product sales and a recommender system for personalized product recommendations."
3. **System Requirements**:
   * **Functional Requirements**: What the system must do (e.g., serve data no more than one hour old).
   * **Non-Functional Requirements**: Characteristics of the system (e.g., scalability, reliability, latency).

## **Requirements Gathering Process**

1. **Identify Stakeholders**:
   * Talk to leadership (e.g., CTO, CEO) to understand business goals.
   * Engage with end users (e.g., marketing, data scientists) to understand their needs.
2. **Understand Current Systems**:
   * Learn about existing systems and their limitations.
   * Example: Marketing currently gets daily data files, but they need real-time data.
3. **Ask Key Questions**:
   * What actions will stakeholders take with the data?
   * What problems exist with the current system?
   * Who else should you talk to for more information?
4. **Document Requirements**:
   * Use a hierarchical format to connect business goals, stakeholder needs, and system requirements.

## **Functional Requirements**

* **Analytics Dashboards**:
  * Serve data no more than one hour old.
  * Example: "The system must provide real-time product sales data for marketing dashboards."
* **Recommender System**:
  * Provide training data for the recommender model.
  * Ingest, transform, and serve user data to the model.
  * Return product recommendations to the sales platform.
  * Example: "The system must serve personalized product recommendations based on user behavior."

## **Non-Functional Requirements**

* **Analytics Dashboards**:
  * **Scalability**: Handle peak user activity without slowing down.
  * **Reliability**: Perform data quality checks to ensure data conforms to the expected format.
  * **Maintainability**: Easily adapt to changes in data schema.
* **Recommender System**:
  * **Latency**: Serve recommendations in less than one second.
  * **Scalability**: Handle maximum concurrent users.
  * **Reliability**: Default to popular products if the recommender fails.

## **Conversations with Stakeholders**

1. **Marketing Team**:
   * Needs real-time dashboards to monitor product sales and react to demand spikes.
   * Wants a personalized recommender system for customers.
2. **Data Scientists**:
   * Currently work with daily data files but need real-time data for dashboards and recommender models.
3. **Software Engineers**:
   * Plan to set up a read replica database and API for continuous data access.
   * Will notify data engineers of schema changes and system outages.

## **Trade-Offs in Requirements Gathering**

* **Iron Triangle** 🔼:
  * **Scope**: Features and functionality of the system.
  * **Timeline**: How quickly the system needs to be built.
  * **Cost**: Budget constraints for the project.
* **Key Insight**: You can't optimize all three simultaneously (e.g., fast and cheap may compromise quality).
* **Solution**:
  * Build **loosely coupled systems** for flexibility.
  * Make **reversible decisions** (two-way doors).
  * Deeply understand **stakeholder needs** to prioritize effectively.

## Sample Project: Recommender System

### **Key Components of the Project**

* **Recommender System**: A content-based recommender system is being developed to recommend products to users based on:
  * **User features**: Customer number, credit limit, city, postal code, country.
  * **Product features**: Product code, quantity in stock, buy price, MSRP, product line, product scale.
  * **User interactions**: Products browsed or added to the cart.
* **Two Data Pipelines**:
  1. **Batch Data Pipeline**: Delivers training data to the data scientist for model retraining.
  2. **Streaming Data Pipeline**: Provides real-time product recommendations to users based on their activity.

### **Functional Requirements**

* **Batch Pipeline**:
  * Deliver training data in tabular format.
  * Include user features, product features, and user ratings (1-5).
  * Support retraining the model periodically (weekly, monthly, or quarterly).
  * Handle modifications in data format (e.g., new user or product features).
* **Streaming Pipeline**:
  * Provide real-time recommendations with subsecond latency (1-2 seconds).
  * Handle up to 10,000 concurrent users, with potential for growth.
  * Use pre-trained recommender system to generate recommendations.
  * Save model outputs for later analysis.

### **Non-Functional Requirements**

* **Latency**: Recommendations must be generated in under 1 second to match page rendering times.
* **Scalability**: The system must handle spikes of up to 10,000 concurrent users and scale as the company grows.
* **Flexibility**: The system should accommodate changes in data format (e.g., new features).
* **Operational Overhead**: Minimize the effort required to deliver new batches of training data.

### **Recommender System Details**

* **Content-Based Recommender**:
  * Uses vector embeddings for users and products to find similarities.
  * Predicts user ratings for products based on embeddings.
  * Combines recommendations from:
    * User features (e.g., “Based on your profile, you may like…”).
    * Product interactions (e.g., “Based on your browsing history, you may like…”).
* **Vector Database**:
  * Stores precomputed product embeddings for faster similarity searches.
  * Organizes embeddings so similar products are close together, speeding up retrieval.

### **Implementation Steps**

1. **Extract Requirements**: Identify functional and non-functional requirements from the conversation.
2. **Select Tools**: Choose AWS tools and services that meet the requirements.
3. **Lab Exercise**:
   * Set up batch pipeline to deliver training data.
   * Use pre-trained recommender system for streaming pipeline.
   * Implement vector database for fast similarity searches.

## **Key Takeaways**

1. **Stakeholder Engagement**:
   * Talk to leadership, end users, and source system owners to understand needs and constraints.
2. **Documentation**:
   * Clearly document functional and non-functional requirements using a hierarchical format.
3. **Trade-Offs**:
   * Balance scope, timeline, and cost by applying principles like loose coupling and reversible decisions.
4. **System Design**:
   * Design systems that are scalable, reliable, and maintainable to meet stakeholder needs and business goals.

***

[Source](https://link.rajanand.org/introduction-to-data-engineering-coursera): DeepLearning.ai data engineering course.