Overview

This repository is a comprehensive demonstration of advanced Django development techniques, designed to provide a deep understanding of optimizing and scaling Django applications. It integrates a variety of tools and technologies, including Redis, Django REST Framework, Celery, Silk, Django Debug Toolbar, Celery Beat, and Celery Flower, to showcase best practices in modern web development. The project is structured around a series of practical labs, each focusing on a specific aspect of Django development, such as query optimization, background task processing, caching, and profiling.

Through these labs, I gained hands-on experience in:

• Designing efficient database schemas and optimizing ORM queries.
• Profiling and monitoring application performance to identify bottlenecks.
• Implementing background task processing for heavy workloads.
• Scheduling periodic tasks and monitoring their execution.
• Leveraging caching mechanisms to improve application performance.

---

Project Structure

Root Directory

• requirements.txt: Contains the list of Python dependencies required for the project.
• assets/: Contains images that illustrate various aspects of the project.
• Project/: The main Django project folder, including settings, URLs, and WSGI configurations.
• Movies/: The Django app folder, containing models, views, serializers, and management commands.
• data/: Contains CSV files used for importing data into the database.

---

Labs and Features

LAB 1: ORM Basics

• Selected the Movie Lens Small Latest Dataset from Kaggle.
• Designed a proper database schema to achieve modeling, relationships, and constraints.
• Built Django models for the schema and loaded data into the database.
• Ran queries to retrieve all data and printed related key fields for each record.
• Observed the N+1 query problem using monitoring tools.
• Enhanced the query using select_related and analyzed the performance improvement.

LAB 2: Advanced ORM Queries

• Built dynamic filter queries using the Q() expression.
• Updated field values directly in SQL using the F() expression.
• Selected specific fields using the only() and defer() methods.
• Retrieved data as dictionaries using appropriate ORM methods.
• Applied indexing on proper fields in the models.
• Compared the performance of queries on indexed vs non-indexed columns.
• Adjusted CONN_MAX_AGE in the database settings and observed the changes.

LAB 3: Monitoring and Profiling

• Configured Django Debug Toolbar, Silk, and cProfile for the project.
• Monitored and profiled views containing ORM queries and functions.
• Analyzed profiling data such as function timings, query counts, request info, render times, and the heaviest requests.

LAB 4: Background Immediate Tasks

• Configured Celery with Redis as the message broker and Flower for monitoring.
• Simulated two heavy tasks in tasks.py.
• Offloaded these tasks from the request-response cycle by running them in the background.
• Monitored the tasks using Flower.

LAB 5: Background Scheduled Tasks

• Configured Celery Beat for task scheduling.
• Scheduled one task directly in the code to run every 3 minutes.
• Scheduled two tasks via the Django admin panel (one to run at a specific time and another at regular intervals).
• Verified task scheduling in Celery Beat and monitored execution in Celery and Flower.

LAB 6: Caching

• Configured Redis as the caching backend for Django.
• Applied caching to specific heavy functions, ORM-heavy queries, and full views.
• Analyzed the caching section in the Django Debug Toolbar for each request.

---

Assets

The assets folder contains images that provide insights into the project's functionality and performance. Below is a detailed description of each image:

1. celery logs: Displays the logs generated by Celery workers, showcasing the execution of background tasks.
2. configured DDT: Demonstrates the configuration of Django Debug Toolbar (DDT) for debugging and monitoring queries.
3. Django administration periodictask: Shows the Django admin interface for managing periodic tasks configured via Celery Beat.
4. dynamic filtering: Highlights an example of dynamic filtering implemented in the application using Django ORM.
5. flower - scheduled tasks: Captures the Flower dashboard displaying scheduled tasks managed by Celery Beat.
6. flower summary: Provides a summary view of tasks monitored via Flower, including their statuses and execution times.
7. flower tasks: Displays the list of tasks executed and monitored through Flower.
8. indexing benchmark: Highlights the performance benchmarks achieved through database indexing.
9. no optimization: Illustrates the performance of queries without any optimizations applied.
10. optimization (defer): Demonstrates the use of defer() for optimizing queries by deferring the loading of specific fields.
11. optimization (only): Shows the use of only() to fetch specific fields in queries, reducing the amount of data retrieved.
12. optimization (select_related): Highlights the use of select_related() for optimizing related object queries and reducing database hits.
13. return as tuple: Displays an example of returning query results as tuples for efficient data handling.
14. Silk Some Request: Captures a snapshot of Silk profiling for specific requests, showing detailed performance metrics.
15. Silk Summary: Provides a summary of Silk profiling results, including request timings and query counts.
16. update(F): Demonstrates the use of Django's F() expressions for efficient field updates directly in the database.

---

Setup Instructions

1. Clone the Repository:

   git clone https://github.com/zaher-m/Django-ORM-Opt.git

2. Install Dependencies:

   pip install -r requirements.txt

3. Run Migrations:

   python manage.py migrate

4. Import Data: Use the custom management command to import data from CSV files:

   python manage.py import_csv_data

5. Run the Development Server:

   python manage.py runserver

---

Contributing

1. Fork the repository.
2. Create a new branch for your feature or bug fix.
3. Submit a pull request with a detailed description of your changes.