Random vs Sequential Sampling Efficiency #1
Conversation
This commit adds a thorough evaluation of efficiency gains from using FewLab's optimal sampling methods compared to random sampling. Key findings: - 14% efficiency gain (1.14x) in realistic scenarios with heterogeneous counts - Peak efficiency of 9.2% at 10% budget (moderate sampling) - Diminishing returns at high budgets (>30% of items) - Deterministic A-optimal provides best speed (1.8ms) with same efficiency Files added: - evaluate_random_vs_methods.py: Main evaluation comparing all methods - evaluate_budget_sensitivity.py: Analysis of efficiency vs. budget size - EVALUATION_RESULTS.md: Comprehensive summary of findings - *.png: Visualization plots - *.csv: Detailed simulation results The evaluation demonstrates that FewLab methods are most valuable when: 1. Budgets are constrained (10-20% of items) 2. Count distributions are heterogeneous 3. Accurate estimates are critical
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| # Import FewLab methods | ||
| import sys | ||
| sys.path.insert(0, '/home/user/fewlab') | ||
| import fewlab | ||
| from fewlab.utils import compute_g_matrix, compute_horvitz_thompson_weights |
There was a problem hiding this comment.
The script inserts /home/user/fewlab into sys.path and later writes plots and CSV files to the same absolute directory. Anyone running the evaluation from a cloned repository in a different location (e.g. /workspace/fewlab or a Windows path) will hit ModuleNotFoundError or FileNotFoundError before any simulations complete. Using paths relative to __file__ or relying on the installed package keeps the example portable and lets the results be written wherever the repository lives.
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
To use Codex here, create an environment for this repo.
There was a problem hiding this comment.
Summary
- Updated both evaluation scripts to derive the repository root from
__file__, ensuring FewLab imports and output directories use paths relative to the checkout instead of a hard-coded/home/userprefix. examples/evaluate_random_vs_methods.pyL21-L38 examples/evaluate_budget_sensitivity.pyL10-L25 - Redirected figure and CSV exports to the local
examplesdirectory with dynamic paths, keeping the saved outputs alongside each script regardless of where the repository is cloned. examples/evaluate_random_vs_methods.pyL397-L432 examples/evaluate_random_vs_methods.pyL513-L516 examples/evaluate_budget_sensitivity.pyL247-L343
Testing
⚠️ Tests not run (not requested).
| import sys | ||
| sys.path.insert(0, '/home/user/fewlab') | ||
| import fewlab | ||
| from fewlab.utils import compute_g_matrix, compute_horvitz_thompson_weights |
There was a problem hiding this comment.
Like the other evaluation script, this one prepends /home/user/fewlab to the import path and saves figures/CSVs to that absolute directory. On a typical checkout the directory does not exist, so the script cannot import fewlab or persist results. Switching to a path derived from Path(__file__) or the current working directory would allow the analysis to run wherever the repo is cloned.
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
To use Codex here, create an environment for this repo.
There was a problem hiding this comment.
Summary
- Updated both evaluation scripts to derive the repository root from their own file locations so
fewlabcan be imported without hard-coded absolute paths. examples/evaluate_budget_sensitivity.pyL10-L24 examples/evaluate_random_vs_methods.pyL20-L38 - Saved generated plots and CSV outputs beside each script in
examples/, replacing references to/home/user/fewlab. examples/evaluate_budget_sensitivity.pyL247-L342 examples/evaluate_random_vs_methods.pyL397-L517
Testing
⚠️ Not run (not requested)
There was a problem hiding this comment.
@codex fix the following test failure:
Ruff linter..............................................................Failed
- hook id: ruff
- exit code: 1
- files were modified by this hook
examples/evaluate_budget_sensitivity.py:24:1: E402 Module level import not at top of file
|
22 | if str(REPO_ROOT) not in sys.path:
23 | sys.path.insert(0, str(REPO_ROOT))
24 | import fewlab
| ^^^^^^^^^^^^^ E402
25 | from fewlab.utils import compute_g_matrix, compute_horvitz_thompson_weights
|
examples/evaluate_budget_sensitivity.py:25:1: E402 Module level import not at top of file
|
23 | sys.path.insert(0, str(REPO_ROOT))
24 | import fewlab
25 | from fewlab.utils import compute_g_matrix, compute_horvitz_thompson_weights
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ E402
|
examples/evaluate_random_vs_methods.py:37:1: E402 Module level import not at top of file
|
35 | if str(REPO_ROOT) not in sys.path:
36 | sys.path.insert(0, str(REPO_ROOT))
37 | import fewlab
| ^^^^^^^^^^^^^ E402
38 | from fewlab.utils import compute_g_matrix, compute_horvitz_thompson_weights
|
examples/evaluate_random_vs_methods.py:38:1: E402 Module level import not at top of file
|
36 | sys.path.insert(0, str(REPO_ROOT))
37 | import fewlab
38 | from fewlab.utils import compute_g_matrix, compute_horvitz_thompson_weights
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ E402
|
examples/evaluate_random_vs_methods.py:356:5: F841 Local variable n_methods is assigned to but never used
|
354 | """Create visualizations comparing methods."""
355 | methods = summary['method'].unique()
356 | n_methods = len(methods)
| ^^^^^^^^^ F841
357 | n_coefs = cfg.p_features
|
= help: Remove assignment to unused variable n_methods
examples/evaluate_random_vs_methods.py:357:5: F841 Local variable n_coefs is assigned to but never used
|
355 | methods = summary['method'].unique()
356 | n_methods = len(methods)
357 | n_coefs = cfg.p_features
| ^^^^^^^ F841
358 |
359 | # Set style
|
= help: Remove assignment to unused variable n_coefs
Found 7 errors (1 fixed, 6 remaining).
No fixes available (2 hidden fixes can be enabled with the --unsafe-fixes option).
pre-commit hook(s) made changes.
If you are seeing this message in CI, reproduce locally with: pre-commit run --all-files.
To run pre-commit as part of git workflow, use pre-commit install.
All changes made by hooks:
diff --git a/examples/evaluate_budget_sensitivity.py b/examples/evaluate_budget_sensitivity.py
index d13fc2b..f79e129 100644
--- a/examples/evaluate_budget_sensitivity.py
+++ b/examples/evaluate_budget_sensitivity.py
@@ -275,7 +275,7 @@ def print_summary_table(summary: pd.DataFrame, cfg: BudgetConfig):
for _, row in data.iterrows():
if row['method'] == 'Random':
-
print(f" Random (baseline):")
-
print(" Random (baseline):") print(f" Variance: {row['variance']:.4f}") else: print(f" {row['method']}:")
Error: Process completed with exit code 1.
There was a problem hiding this comment.
Summary
- Ensured each evaluation script dynamically locates the repository root and loads FewLab modules with
importlib, keeping them runnable without hard-coded paths while satisfying lint ordering rules. examples/evaluate_budget_sensitivity.pyL10-L31 examples/evaluate_random_vs_methods.pyL20-L42 - Removed unused plotting variables and cleaned the baseline summary output so Ruff no longer flags spurious assignments or redundant f-strings. examples/evaluate_random_vs_methods.pyL357-L365 examples/evaluate_budget_sensitivity.pyL260-L288
Testing
- ✅
ruff check examples/evaluate_budget_sensitivity.py examples/evaluate_random_vs_methods.py
soodoku
deleted the
claude/random-vs-sequential-sampling-011CUoYEALB291PiH85Y7k85
branch
2 participants
This commit adds a thorough evaluation of efficiency gains from using FewLab's optimal sampling methods compared to random sampling.
Key findings:
Files added:
The evaluation demonstrates that FewLab methods are most valuable when: