Expressivity-Efficiency Tradeoffs for Hybrid Sequence Models

This repository includes all of the experiments relevant to the associated paper.

• micro: A collection of old experiments with pytorch transformers and mamba. Not currently used, but might be useful as a reference.
• micro_hf: A collection of experiments for micro-scale experiments, using huggingface models. run.ipynb will execute the experiments, and process.ipynb can be used to analyze the results. Each individual run does not take long to run on an RTX 4090.
• mini: A collection of larger experiments, mostly ported from the ones in the Repeat After Me paper. Added to this are hybrid models implemented in huggingface.