Refactor: Extract database layer into separate crate#10
Merged
Conversation
Initial workspace setup to extract database layer into a separate crate. This is the first step in a multi-ticket refactoring to improve modularity. Changes: - Convert root Cargo.toml to workspace manifest - Create db/ crate directory with minimal lib.rs stub - Create cli/ crate directory with Cargo.toml and main.rs stub - Move CLI dependencies to cli/Cargo.toml - Move db dependencies (cozo, thiserror) to db/Cargo.toml - Add db as path dependency in cli crate The original src/ directory is preserved for incremental migration in subsequent tickets. Relates to ticket: scratch/tickets/01-workspace-foundation.md
Implements ticket 2 of 8 - Build database crate core foundation. Changes: - Update db/Cargo.toml with all dependencies (cozo, serde, thiserror, regex, include_dir) - Copy types module to db/src/types/ (call.rs, results.rs, trace.rs, mod.rs) - Copy db.rs to db/src/db.rs (505 lines with DB utilities) - Create db/src/lib.rs with public module declarations and re-exports - Add test-utils feature flag for conditional test exports The db crate now has: - Database connection management (open_db, open_mem_db) - Query execution utilities (run_query, run_query_no_params) - Type-safe extraction helpers (extract_string, extract_i64, etc.) - CallRowLayout for structured call data extraction - All core types (Call, FunctionRef, ModuleGroup, TraceResult, etc.) Verification: - cargo build -p db: ✅ compiles successfully - cargo test -p db: ✅ all 24 tests pass Original src/ directory preserved for continued migration. Relates to ticket: scratch/tickets/02-db-crate-core.md
Implements ticket 3 of 8 - Extract SQL-specific query builders. Changes: - Create db/src/query_builders.rs with ConditionBuilder and OptionalConditionBuilder - Extract SQL builders from src/utils.rs (lines 8-154) - Extract builder tests from src/utils.rs (lines 640-684) - Update db/src/lib.rs with query_builders module and re-exports The query builders provide: - ConditionBuilder: Builds SQL conditions for exact or regex matching - OptionalConditionBuilder: Builds optional SQL conditions with defaults All 7 builder tests passing: - test_condition_builder_exact_match - test_condition_builder_regex_match - test_condition_builder_with_leading_comma - test_optional_condition_builder_with_value - test_optional_condition_builder_without_value - test_optional_condition_builder_with_default - test_optional_condition_builder_with_leading_comma Verification: - cargo build -p db: ✅ compiles successfully - cargo test -p db query_builders: ✅ all 7 tests pass Next ticket will update 6 query files to import from this new module. Relates to ticket: scratch/tickets/03-query-builders.md
Implements ticket 4 of 8 - Move queries module. Changes: - Copy all 31 query files from src/queries/ to db/src/queries/ - Update 6 query files to import from query_builders instead of utils: - calls.rs, trace.rs, reverse_trace.rs, path.rs, function.rs, dependencies.rs - Update db/src/lib.rs to export queries module - Add clap dependency (required by hotspots.rs) - Add serde_json dev-dependency (required by import.rs tests) Query modules now in db crate: - Data import: import.rs, import_models.rs, schema.rs - Basic lookups: location.rs, function.rs, search.rs, file.rs - Call graph: calls.rs, calls_from.rs, calls_to.rs, trace.rs, reverse_trace.rs, path.rs - Dependencies: depends_on.rs, depended_by.rs, dependencies.rs - Code quality: unused.rs, hotspots.rs, duplicates.rs, complexity.rs, large_functions.rs, many_clauses.rs, cycles.rs, clusters.rs - Type system: specs.rs, types.rs, structs.rs, struct_usage.rs, accepts.rs, returns.rs Verification: - cargo build -p db: ✅ compiles successfully - cargo test -p db --lib: ✅ 37 tests pass - No import warnings The db crate is now fully functional with all query logic. Relates to ticket: scratch/tickets/04-move-queries.md
Implements ticket 5 of 8 - Move test infrastructure.
Changes:
- Copy all fixtures to db/src/fixtures/
- call_graph.json, type_signatures.json, structs.json
- output/ directory with all command output fixtures
- extracted_trace.json for large trace tests
- Copy and modify test_utils.rs to db/src/test_utils.rs
- Remove Execute-dependent functions (execute_cmd, execute_on_empty_db)
- Keep db-specific helpers (setup_test_db, create_temp_json_file, etc.)
- Add #![cfg(feature = "test-utils")] feature gate
- Update db/src/lib.rs with feature-gated test modules
- Update db/Cargo.toml:
- Add tempfile and serde_json as optional dependencies
- Define test-utils feature
- Update db.rs and import.rs to conditionally export test functions
Functions available in test-utils feature:
- create_temp_json_file() - Create temporary JSON for testing
- setup_test_db() - Set up test database with fixture data
- setup_empty_test_db() - Create empty database for error testing
- call_graph_db(), type_signatures_db(), structs_db() - Pre-configured DBs
- load_output_fixture() - Load expected output for validation
Verification:
- cargo test -p db: ✅ 37 tests pass
- cargo build -p db --features test-utils: ✅ compiles
The db crate is now complete with all core functionality and test infrastructure.
CLI crate can use test helpers via: db = { path = "../db", features = ["test-utils"] }
Relates to ticket: scratch/tickets/05-test-infrastructure.md
Implements ticket 6 of 8 - Migrate CLI source files. Changes: - Copy core CLI files to cli/src/: - main.rs (entry point, module declarations, CLI runner) - cli.rs (clap argument parser) - output.rs (OutputFormat trait, Outputable trait) - dedup.rs (deduplication utilities) - test_macros.rs (test infrastructure macros) - Copy entire commands/ directory to cli/src/commands/ - All 27 command subdirectories with execute.rs, output.rs, models.rs - Create cli/src/utils.rs with presentation logic only: - group_by_module, group_by_module_with_file, group_calls - convert_to_module_groups - format_type_definition and supporting functions - Excluded: ConditionBuilder, OptionalConditionBuilder (moved to db crate) Commands migrated: - accepts, boundaries, browse_module, calls_from, calls_to - clusters, complexity, cycles, depended_by, depends_on - describe, duplicates, function, god_modules, hotspots - import, large_functions, location, many_clauses, path - returns, reverse_trace, search, setup, struct_usage - trace, unused Note: Import paths still point to old crate:: locations and will be updated in ticket 7. The cli crate won't compile yet - this is expected. Original src/ directory preserved for reference. Relates to ticket: scratch/tickets/06-migrate-cli-files.md
Implements ticket 7 of 8 - Update CLI import paths.
This is the critical integration ticket that connects the cli crate to the db crate.
Changes (100+ files modified):
Core files:
- cli/src/main.rs: Remove pub mod types, import db::open_db
- cli/src/commands/mod.rs: Use db::DbInstance in Execute trait
- cli/src/output.rs: Import db::types::ModuleGroupResult
- cli/src/utils.rs: Import db::types::{ModuleGroup, Call}
All 27 commands updated:
- execute.rs files: crate::queries::* → db::queries::*
- execute.rs files: crate::types::* → db::types::*
- mod.rs files: cozo::DbInstance → db::DbInstance
- output.rs files: crate::types::* → db::types::*
- test files: Updated imports to use db:: prefix
Commands updated:
- accepts, boundaries, browse_module, calls_from, calls_to
- clusters, complexity, cycles, depended_by, depends_on
- describe, duplicates, function, god_modules, hotspots
- import, large_functions, location, many_clauses, path
- returns, reverse_trace, search, setup, struct_usage
- trace, unused
Orphan rule fixes (impl blocks on external types):
Converted 11 impl blocks to standalone builder functions:
- build_accepts_result() (accepts)
- build_calls_from_result() (calls_from)
- build_callee_result() (calls_to)
- build_trace_result() (trace)
- build_dependent_caller_result() (depended_by)
- build_dependency_result() (depends_on)
- build_function_signatures_result() (function)
- build_return_info_result() (returns)
- build_reverse_trace_result() (reverse_trace)
- build_usage_info_result() + build_struct_modules_result() (struct_usage)
- build_unused_functions_result() (unused)
Database crate updates:
- db/src/lib.rs: Re-export DbInstance for convenient access
- Added cozo to db crate public API
Import statistics:
- 0 remaining crate::queries or crate::types references
- 81+ new db:: imports added across all command files
Verification:
- cargo build: ✓ SUCCESS
- rg "use crate::(queries|types)" cli/src/: no matches
- All imports now properly reference db crate
The workspace is now fully integrated with proper crate boundaries.
The entire workspace compiles successfully.
Relates to ticket: scratch/tickets/07-update-imports.md
Fixed broken tests from workspace migration: - Update template paths in setup command to reference workspace root (Changed from $CARGO_MANIFEST_DIR/templates/ to $CARGO_MANIFEST_DIR/../templates/) - All 492 CLI tests + 40 DB tests now passing Enable previously ignored doctest: - Add proper imports to query_builders.rs doctest example - 1 doctest now passing (1 remains intentionally ignored) Apply clippy fixes (37 auto-fixes + 1 manual): - Use .as_deref() instead of .as_ref().map(|s| s.as_str()) - Use $crate instead of crate in test macro definitions - Fix PathBuf comparison in main.rs - Collapse nested if statements - Remove unused 'visited' parameter from dfs_find_cycles() (Algorithm already uses path vector for cycle detection) Clean up old src/ directory (231 files deleted): - Removed all files from previous single-crate structure - Workspace structure now complete with db/ and cli/ crates Results: - All 533 tests passing (492 CLI + 40 DB + 1 doctest) - Clippy warnings reduced from 41 to 15 (63% reduction) - Remaining warnings are acceptable design decisions
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Overview
This PR completes a comprehensive refactoring to extract the database layer into a separate crate within a Cargo workspace. The database layer can now be used independently of the CLI, improving modularity and enabling future extensibility (web API, TUI, etc.).
Architecture Changes
Before
After
Completed Work (8 Tickets)
✅ Ticket 1: Workspace Foundation
dbandclicrates✅ Ticket 2: DB Crate Core
types/(Call, FunctionRef, ModuleGroup, TraceResult, etc.)db.rs(database connection and query utilities)db/src/lib.rs✅ Ticket 3: Query Builders
ConditionBuilderandOptionalConditionBuilderfrom utils.rsdb/src/query_builders.rs✅ Ticket 4: Query Modules Migration
db/src/queries/crate::types→crate::types✅ Ticket 5: Test Infrastructure
test-utilsfeature flagdbwithtest-utilsfeature✅ Ticket 6: CLI Migration
cli/src/code_search✅ Ticket 7: Import Updates
crate::→db::✅ Ticket 8: Validation & Cleanup
src/directory (231 files)Key Improvements
Modularity:
Testability:
Extensibility:
Code Quality:
Test Results
All tests passing ✅
Build Commands
Breaking Changes
None for end users - the CLI binary works identically. Internal module paths have changed for future development.
Documentation
CLAUDE.mdwith workspace structureRelated Files
Cargo.tomldb/Cargo.toml,db/src/lib.rscli/Cargo.toml,cli/src/main.rsCLAUDE.md