ZeroToML

Yet another AI from scratch repo.

• The files are appropriately named.
  • notes on my notes. They're still very rough.
  • test for it in action on limited resources.
  • examples long training on real datasets. This doesn't train well due to the small networks.
• May contain AI slop.

Roadmap

Research

• Decision Trees
  • LightGBM: A Highly Efficient Gradient Boosting Decision Tree
• Transformers
  • Encoder
  • Decoder
  • Encoder + Decoder
  • Impact on Different tokenizers (i.e. BPE tokenizer vs Character level)
  • Impact on number of heads
• Diffusion
  • DDPM — Denoising Diffusion Probabilistic Models
  • DDIM — Denoising Diffusion Implicit Models
  • SDE — Score-Based Generative Modeling through SDEs
  • Flow Matching
  • Rectified Flow — Flow Straight and Fast: Learning to Generate and Transfer Data with Rectified Flow
• RL
  • PPO - Proximal Policy Optimization Algorithms
  • GRPO - Group Relative Policy Optimization
  • SAC - Soft Actor-Critic Algorithms and Applications
  • Framework: STOMP/OAK - Reward-Respecting Subtasks for Model-Based Reinforcement Learning
• Neural Network primitives
  • CNN - A guide to convolution arithmetic for deep learning
    • Discrete convolutions
    • Stride
    • Padding
    • Pooling
    • Transposed
    • Dilated convolutions
    • Works with Gradients
• Errors and residuals

Architecture

• Abstract common functions? Might reduce readability :(
  • Use Multiple Dispatch and Structs/Types
  • Filename, Model name is Paper name
• Stateless? Memory allocations concerns if truely end-to-end stateless. We get history for free though. Maybe Stateless functions only?

Features

• Remove Flux / NNlib dependancies in core code.
• Remove Zygote / AutoDiff? We lose flexibility in changing the model.
• Test from-scratch versions against established libraries for correctness.
• Einstien notation-like that better exposes the math instead of hiding behind API abstractions. Maybe start with Tullio?

TODO

• Improve training in examples/