Skip to content

Ost 408 feat global & perso recommendations #18

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
spideystreet wants to merge 202 commits into
base: staging
Choose a base branch
from
Open

Ost 408 feat global & perso recommendations #18

spideystreet wants to merge 202 commits into from

Conversation

@spideystreet
Copy link
Collaborator

@spideystreet spideystreet commented Jan 28, 2026 •
edited
Loading

Feature : Embeddings, Classification & Recommandation

Grosse PR qui met en place : calcul d'embeddings vectoriels, classification automatique par SLM et moteur de recommandation (User <-> Project).

Docker & DevOps

  • Startup (scripts/init.sh) : Nouvel entrypoint qui gère l'install des deps dbt et lance dbt build au démarrage. Fini les crashs relation does not exist !
  • Networking : Fix de la com inter-conteneurs

Go Services (Refactor)

  • Architecture : Split propre entre fetcher (récupération brute) et scraper (logique métier)
  • Concurrence : Optimisation des routines de fetch (Readme, Languages, Topics) pour tenir la charge
  • Cleanup : Suppression du code mort et unification des structures de données via common.go

Data Engineering & ML (Dagster)

  • Stratégie : Filtrage renforcé sur la qualité (good-first-issues > 5, activité < 24h, stars 500..1K) pour garantir des projets vivants et qualité
  • Classification Hybride :
    • FastText : Détection de langue (core_github__detect_languages)
    • LLM (Qwen2.5) : Classification sémantique (Domaine/Catégorie) via LLMClassifierResource
  • Embeddings : Vectorisation (384d) des projets et des profils utilisateurs via SentenceTransformer
  • Orchestration : Job run_all_job planifié toutes les 6h pour commencer

Transformation (dbt & SQL)

  • Data Quality : Macros de nettoyage (clean_text, jsonb_to_list) pour préparer le terrain au ML
  • Contexte LLM : Les macros build_project_context build_user_context pour maximiser la pertinence des embeddings
  • Matching Tables:
    • public.match_global_recommendation : Reco globale, Top 5 (Trending/Récents)
    • public.match_user_recommendation : Recos persos (Cosine Similarity sur pgvector)

DB (Prisma & Postgres)

  • Structure : Alignement strict Multi-Schemas (public, github, match, ml)
  • Vector Search : Activation de l'extension vector et support Prisma via Unsupported("vector(384)")
  • Seeding : Fix des imports ESM/CJS pour que npm run seed fonctionne enfin en local sans pleurer

Important

@Olyxz :

  • Vérifier que l'extension pgvector est bien activée sur staging
  • S'assurer que les workers ont assez de RAM pour charger le modèle Qwen ( sinon los problemos )
  • Vérifie qu'on a bien des users avec catégories & domaines dans la db pour tester
  • Lance un run_all_job pour voir si tout passe ok

@spideystreet
docs: up env examples
7469947
@spideystreet
fix: using right env for postgres
ecd5662
@spideystreet
fix: ost-linker new name
84427c6
@spideystreet
fix: system dependancies
cc3fd7a
@spideystreet
fix: del custom source for pytorch
c509dee
@spideystreet
refactor: del mapping categories assets before refacto
53b3a95
@spideystreet
refactor: del repo meta nomalization
869a8e9
@spideystreet
fix: logs cleanup job activated by default
eded63b
@spideystreet
build: using pgvector image for future embeddings
cd8ed12
@spideystreet
feat: structure for futur embedding jobs
69c0d34
@spideystreet
docs: add prisma conf
8b2d92d
@spideystreet
feat: up seed for categories
f5cc565
@spideystreet
feat: makefile for setup
471c5b2
@spideystreet
feat(db): add embedding tables on migration & relations on schema
eb93fdb
@spideystreet
feat: add embedding model as dagster resource, for cpu usage
c2981d0
@spideystreet
refactor: del gitlab sources for now
1022233
@spideystreet
feat: add embedding model as ConfigurableResource (dagster)
6db08f4
@spideystreet
refactor: split assets in python modules
f7b7185
@spideystreet
refactor: modularisation of assets, del assets checks (outdated)
e8c116c
@spideystreet
docs: github api rate limit
04197f1
@spideystreet
refactor: del gitlab conf (outdated)
0402220
@spideystreet
docs: up conf with Linker
c40afc1
@spideystreet
docs: del gitlab
bc7a2b6
@spideystreet
feat: add Makefile for easy setup
ea92bba
@spideystreet
refactor: del gitlab references
ce8d85c
@spideystreet
refactor: moving raw_github__to_df on raw/
44b9be0
@spideystreet
refactor: using in process executor to avoid SIGBUS errors with python
ef2e338
@spideystreet
fix: improving golang binary with vaiables & logs
dd7cacf
@spideystreet
refactor: modularisation of assets
cf86dfa
@spideystreet
build: shm size to avoid sigbus error
5c64f98
@spideystreet
docs(dbt): reco precision
057690c
@spideystreet
fix(pipeline): wire embedding asset to int_project_embedding_candidate
e956d44 
- Fix incorrect upstream dependency (was pvt_public_project)
- Update column accessors (project_id, rich_context_string)
- Refactor SQL query to constant
@spideystreet
docs: improve dbt model and dagster asset descriptions
8655385 
- Update Dagster job descriptions to focus on orchestration flow
- Clarify classification asset docstrings
- Enhance DBT ML model descriptions (stg/pvt) to explain business logic over implementation details
@spideystreet
chore(dbt): remove stale config for non-existent model int_github_emb…
d2f0d9b 
…edding
@spideystreet
config: update excluded terms list for scraper
170211c
@spideystreet
chore(infra): dockerize application
a0d5fae 
- Add multi-stage Dockerfile (Go builder + Python Runtime)
- Add docker-compose.yml with pgvector support
- Add .dockerignore
@spideystreet
config: 10 ops max for github query
4b85c5e
@spideystreet
chore: add logs for classified projects evolution
58b8ee1
@spideystreet
config: up to date config with needed vars & parameters
5b25f33
@spideystreet
config: up lineage with llm classifier as resource + good parameters …
dd78aa0 
…for cpu usage in docker
@spideystreet
feat: optimised query parameters to find acurate projects
db6102a
@spideystreet
config: group name ml
e1df582
@spideystreet
build: up dockerignore
3b69861
@spideystreet
fix: seed import syntax
7de373c
@spideystreet
docs: up env example
0ab4690
@spideystreet
docs: add embedding & raw tables not managed by dbt, used by linker t…
3ed6b16 
…o fetch datas
@spideystreet
fix: correct lineage of groups, to ensure they launch together
b37c14e
@spideystreet
build: correct env var usage
b279790
@spideystreet
docs: up README to date
5249257
@spideystreet
feat(prisma): allign with backend & add extensions for linker
210e0ec
@spideystreet
build: entrypoint script to dbt build & deps
bf71e03
@spideystreet
chore: up gitignore
7bfdf01
@spideystreet
chore(docker): configure entrypoint script and dependencies
e08e0d2
@spideystreet
fix: pg client no need
a005caf
@spideystreet
chore: entrypoint pg is ready step outdated
332d490
@spideystreet
feat(schedule): add run_all_schedule 5x daily (Europe/Paris)
381b5f6
@spideystreet spideystreet requested a review from Olyxz16 January 28, 2026 14:31
@spideystreet spideystreet self-assigned this Jan 28, 2026
@spideystreet spideystreet added the enhancement New feature or request label Jan 28, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@Olyxz16 Olyxz16 Awaiting requested review from Olyxz16

Assignees

@spideystreet spideystreet

Labels

enhancement New feature or request

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@spideystreet