Skip to content

⚡ Bolt: optimize manual lending N+1 queries #22

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
Woschj wants to merge 1 commit into
base: main
Choose a base branch
from
Open

⚡ Bolt: optimize manual lending N+1 queries #22

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 3 additions & 0 deletions .jules/bolt.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -8,3 +8,6 @@
## 2025-07-14 - [Aggregation for Activity Feeds]
**Learning:** Replacing loop-based lookups (N+1 queries) with single aggregation pipelines in `get_recent_activity` reduces database roundtrips by over 90% (from 42 queries to 2). Using `$unwind` without `preserveNullAndEmptyArrays` effectively replicates `if tool and worker:` filtering logic in the database layer.
**Action:** Apply similar aggregation patterns to other feed-like features (e.g., `manual_lending` in `app/routes/admin/system.py`).
## 2025-01-24 - [N+1 Query Resolution in Manual Lending]
**Learning:** The `manual_lending` route had a severe N+1 query problem where each active lending and recent consumable usage triggered two additional `find_one` calls. Replacing these with separate aggregation pipelines using `$lookup` and `$project` reduces the query count from O(N) to O(1).
**Action:** Always check loop-based lookups in routes that list multiple items and prioritize aggregation even if local `mongomock` tests don't show immediate performance gains, as the real benefit is in reducing database roundtrips.
111 changes: 75 additions & 36 deletions app/routes/admin/system.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -124,44 +124,83 @@ def manual_lending():
# Hole aktuelle Ausleihen
current_lendings = []

# Aktuelle Werkzeug-Ausleihen
active_tool_lendings = mongodb.find('lendings', {'returned_at': None})
for lending in active_tool_lendings:
tool = mongodb.find_one('tools', {'barcode': lending['tool_barcode']})
worker = mongodb.find_one('workers', {'barcode': lending['worker_barcode']})

if tool and worker:
current_lendings.append({
'item_name': tool['name'],
'item_barcode': tool['barcode'],
'worker_name': f"{worker['firstname']} {worker['lastname']}",
'worker_barcode': worker['barcode'],
'action_date': lending['lent_at'],
'category': 'Werkzeug',
'amount': None
})

# Aktuelle Verbrauchsmaterial-Ausgaben (letzte 30 Tage)
# Aktuelle Werkzeug-Ausleihen (Optimiert mit Aggregation zur Vermeidung von N+1 Problemen)
active_tool_lendings_pipeline = [
{'$match': {'returned_at': None}},
{
'$lookup': {
'from': 'tools',
'localField': 'tool_barcode',
'foreignField': 'barcode',
'as': 'tool_info'
}
},
{'$unwind': '$tool_info'},
{
'$lookup': {
'from': 'workers',
'localField': 'worker_barcode',
'foreignField': 'barcode',
'as': 'worker_info'
}
},
{'$unwind': '$worker_info'},
{
'$project': {
'item_name': '$tool_info.name',
'item_barcode': '$tool_info.barcode',
'worker_name': {'$concat': ['$worker_info.firstname', ' ', '$worker_info.lastname']},
'worker_barcode': '$worker_info.barcode',
'action_date': '$lent_at',
'category': {'$literal': 'Werkzeug'},
'amount': {'$literal': None}
}
}
]

current_lendings.extend(list(mongodb.aggregate('lendings', active_tool_lendings_pipeline)))

# Aktuelle Verbrauchsmaterial-Ausgaben der letzten 30 Tage (Optimiert mit Aggregation)
thirty_days_ago = datetime.now() - timedelta(days=30)
recent_consumable_usages = mongodb.find('consumable_usages', {
'used_at': {'$gte': thirty_days_ago},
'quantity': {'$lt': 0} # Nur Ausgaben (negative Werte), nicht Entnahmen
})
recent_consumable_usages_pipeline = [
{
'$match': {
'used_at': {'$gte': thirty_days_ago},
'quantity': {'$lt': 0} # Nur Ausgaben (negative Werte)
}
},
{
'$lookup': {
'from': 'consumables',
'localField': 'consumable_barcode',
'foreignField': 'barcode',
'as': 'consumable_info'
}
},
{'$unwind': '$consumable_info'},
{
'$lookup': {
'from': 'workers',
'localField': 'worker_barcode',
'foreignField': 'barcode',
'as': 'worker_info'
}
},
{'$unwind': '$worker_info'},
{
'$project': {
'item_name': '$consumable_info.name',
'item_barcode': '$consumable_info.barcode',
'worker_name': {'$concat': ['$worker_info.firstname', ' ', '$worker_info.lastname']},
'worker_barcode': '$worker_info.barcode',
'action_date': '$used_at',
'category': {'$literal': 'Verbrauchsmaterial'},
'amount': '$quantity'
}
}
]

for usage in recent_consumable_usages:
consumable = mongodb.find_one('consumables', {'barcode': usage['consumable_barcode']})
worker = mongodb.find_one('workers', {'barcode': usage['worker_barcode']})

if consumable and worker:
current_lendings.append({
'item_name': consumable['name'],
'item_barcode': consumable['barcode'],
'worker_name': f"{worker['firstname']} {worker['lastname']}",
'worker_barcode': worker['barcode'],
'action_date': usage['used_at'],
'category': 'Verbrauchsmaterial',
'amount': usage['quantity']
})
current_lendings.extend(list(mongodb.aggregate('consumable_usages', recent_consumable_usages_pipeline)))

# Sortiere nach Datum (neueste zuerst)
def safe_date_key(lending):
Expand Down