Deep Learning Optimization Research

Experiments with deep learning models, optimization algorithms, and combinatorial optimization. See report/ for a whitepaper of my current research interests.

A fundamental principle in machine learning and neuroscience is chaos dynamics, which hypothesizes an "edge of chaos" that enables optimal information flow and capacity. It balances chaos (exploding gradients) with order (vanishing gradients) to support rich yet controllable systems. This can be naturally studied through the lens of Lyapunov exponents.

This repository hosts code exploring these dynamics alongside various optimization techniques.

Create virtual environment:

python3 -m venv .venv
source .venv/bin/activate

Install dependencies:

pip install --upgrade pip
pip install -r requirements.txt

co/

Combinatorial optimization experiments:

• Langevin dynamics for sampling and optimization
• Probabilistic bit simulations for Ising models
• Simulated annealing and simulated bifurcation algorithms

gd/

Gradient descent implementations and experiments:

• Vanilla SGD, debiased SGD with momentum, Muon, error-feedback SGD example
• comparison of optimizers for GAN training

models/

Basic neural network models:

• Boltzmann machines (based on probabilistic-bit framework)
• ResNet and Transformer

Misc.

Various experiments from the statistical physics and machine learning literature:

• lypa_rnn.py explores Lyapunov analysis for RNNs with heavy-tailed weight distributions
• lypa_transformer.py computes the "edge-of-chaos" for randomly initialized transformers.
• train_boltzman.py is a hello-world example for training a Boltzmann machine
• pbits_demo.py shows convergence to the standard Boltzmann distribution