Skip to content

Comments

Refactor: Extract database layer into separate crate#10

Merged
CamonZ merged 8 commits intomasterfrom
refactor-db-crate
Dec 23, 2025
Merged

Refactor: Extract database layer into separate crate#10
CamonZ merged 8 commits intomasterfrom
refactor-db-crate

Conversation

@CamonZ
Copy link
Owner

@CamonZ CamonZ commented Dec 22, 2025

Overview

This PR completes a comprehensive refactoring to extract the database layer into a separate crate within a Cargo workspace. The database layer can now be used independently of the CLI, improving modularity and enabling future extensibility (web API, TUI, etc.).

Architecture Changes

Before

code_search/
├── Cargo.toml
└── src/
    ├── main.rs
    ├── cli.rs
    ├── commands/    (27 commands)
    ├── queries/     (31 query modules)
    ├── types/
    ├── db.rs
    └── ...

After

code_search/
├── Cargo.toml (workspace)
├── db/          (library crate)
│   ├── Cargo.toml
│   └── src/
│       ├── lib.rs
│       ├── db.rs
│       ├── types/
│       ├── queries/        (31 modules)
│       ├── query_builders.rs
│       ├── fixtures/       (test-utils feature)
│       └── test_utils.rs   (test-utils feature)
└── cli/         (binary crate)
    ├── Cargo.toml
    └── src/
        ├── main.rs
        ├── cli.rs
        ├── commands/       (27 commands)
        ├── output.rs
        ├── utils.rs
        └── test_macros.rs

Completed Work (8 Tickets)

✅ Ticket 1: Workspace Foundation

  • Created Cargo workspace with db and cli crates
  • Set up workspace dependencies and package inheritance
  • Configured resolver version 2

✅ Ticket 2: DB Crate Core

  • Moved types/ (Call, FunctionRef, ModuleGroup, TraceResult, etc.)
  • Moved db.rs (database connection and query utilities)
  • Established public API surface in db/src/lib.rs

✅ Ticket 3: Query Builders

  • Extracted ConditionBuilder and OptionalConditionBuilder from utils.rs
  • Created db/src/query_builders.rs
  • Added comprehensive tests and documentation

✅ Ticket 4: Query Modules Migration

  • Moved all 31 query modules to db/src/queries/
  • Updated imports: crate::typescrate::types
  • Preserved all functionality and tests

✅ Ticket 5: Test Infrastructure

  • Created test-utils feature flag
  • Moved fixtures and test utilities to db crate
  • Updated CLI tests to use db with test-utils feature

✅ Ticket 6: CLI Migration

  • Migrated all CLI source files to cli/src/
  • Moved all 27 command modules
  • Updated package name to code_search

✅ Ticket 7: Import Updates

  • Updated all imports: crate::db::
  • Fixed 300+ import statements across 27+ commands
  • Verified compilation with no errors

✅ Ticket 8: Validation & Cleanup

  • Fixed broken tests (template paths)
  • Enabled ignored doctest
  • Applied clippy improvements (37 auto-fixes + 1 manual)
  • Removed unused DFS parameter
  • Deleted old src/ directory (231 files)
  • Reduced clippy warnings from 41 to 15 (63% reduction)

Key Improvements

Modularity:

  • Database layer is now a standalone library
  • Clear separation of concerns: data access vs. presentation
  • CLI depends on db, but db has no CLI dependencies

Testability:

  • Test fixtures shared via feature flag
  • Database tests run independently
  • 533 total tests (492 CLI + 40 DB + 1 doctest)

Extensibility:

  • Can build new interfaces (web API, TUI) using the db crate
  • Database logic reusable across multiple frontends
  • Foundation for future plugin system

Code Quality:

  • Clippy warnings reduced 63%
  • Proper dependency boundaries enforced
  • Documentation improved (workspace structure, doctest enabled)

Test Results

All tests passing ✅

CLI tests:     492 passed
DB tests:      40 passed
Doctests:      1 passed, 1 ignored (intentional)
Total:         533 tests

Build Commands

cargo build                      # Build entire workspace
cargo build -p cli               # Build CLI only
cargo build -p db                # Build DB library only
cargo test                       # Run all tests
cargo test -p db                 # Test DB layer only
cargo test -p code_search        # Test CLI layer only
cargo run -p code_search -- --help   # Run CLI

Breaking Changes

None for end users - the CLI binary works identically. Internal module paths have changed for future development.

Documentation

  • Updated CLAUDE.md with workspace structure
  • Added build commands and architecture overview
  • Documented dependency flow and testing approach

Related Files

  • Workspace config: Cargo.toml
  • DB crate: db/Cargo.toml, db/src/lib.rs
  • CLI crate: cli/Cargo.toml, cli/src/main.rs
  • Documentation: CLAUDE.md

Initial workspace setup to extract database layer into a separate crate.
This is the first step in a multi-ticket refactoring to improve modularity.

Changes:
- Convert root Cargo.toml to workspace manifest
- Create db/ crate directory with minimal lib.rs stub
- Create cli/ crate directory with Cargo.toml and main.rs stub
- Move CLI dependencies to cli/Cargo.toml
- Move db dependencies (cozo, thiserror) to db/Cargo.toml
- Add db as path dependency in cli crate

The original src/ directory is preserved for incremental migration
in subsequent tickets.

Relates to ticket: scratch/tickets/01-workspace-foundation.md
Implements ticket 2 of 8 - Build database crate core foundation.

Changes:
- Update db/Cargo.toml with all dependencies (cozo, serde, thiserror, regex, include_dir)
- Copy types module to db/src/types/ (call.rs, results.rs, trace.rs, mod.rs)
- Copy db.rs to db/src/db.rs (505 lines with DB utilities)
- Create db/src/lib.rs with public module declarations and re-exports
- Add test-utils feature flag for conditional test exports

The db crate now has:
- Database connection management (open_db, open_mem_db)
- Query execution utilities (run_query, run_query_no_params)
- Type-safe extraction helpers (extract_string, extract_i64, etc.)
- CallRowLayout for structured call data extraction
- All core types (Call, FunctionRef, ModuleGroup, TraceResult, etc.)

Verification:
- cargo build -p db: ✅ compiles successfully
- cargo test -p db: ✅ all 24 tests pass

Original src/ directory preserved for continued migration.

Relates to ticket: scratch/tickets/02-db-crate-core.md
Implements ticket 3 of 8 - Extract SQL-specific query builders.

Changes:
- Create db/src/query_builders.rs with ConditionBuilder and OptionalConditionBuilder
- Extract SQL builders from src/utils.rs (lines 8-154)
- Extract builder tests from src/utils.rs (lines 640-684)
- Update db/src/lib.rs with query_builders module and re-exports

The query builders provide:
- ConditionBuilder: Builds SQL conditions for exact or regex matching
- OptionalConditionBuilder: Builds optional SQL conditions with defaults

All 7 builder tests passing:
- test_condition_builder_exact_match
- test_condition_builder_regex_match
- test_condition_builder_with_leading_comma
- test_optional_condition_builder_with_value
- test_optional_condition_builder_without_value
- test_optional_condition_builder_with_default
- test_optional_condition_builder_with_leading_comma

Verification:
- cargo build -p db: ✅ compiles successfully
- cargo test -p db query_builders: ✅ all 7 tests pass

Next ticket will update 6 query files to import from this new module.

Relates to ticket: scratch/tickets/03-query-builders.md
Implements ticket 4 of 8 - Move queries module.

Changes:
- Copy all 31 query files from src/queries/ to db/src/queries/
- Update 6 query files to import from query_builders instead of utils:
  - calls.rs, trace.rs, reverse_trace.rs, path.rs, function.rs, dependencies.rs
- Update db/src/lib.rs to export queries module
- Add clap dependency (required by hotspots.rs)
- Add serde_json dev-dependency (required by import.rs tests)

Query modules now in db crate:
- Data import: import.rs, import_models.rs, schema.rs
- Basic lookups: location.rs, function.rs, search.rs, file.rs
- Call graph: calls.rs, calls_from.rs, calls_to.rs, trace.rs, reverse_trace.rs, path.rs
- Dependencies: depends_on.rs, depended_by.rs, dependencies.rs
- Code quality: unused.rs, hotspots.rs, duplicates.rs, complexity.rs,
  large_functions.rs, many_clauses.rs, cycles.rs, clusters.rs
- Type system: specs.rs, types.rs, structs.rs, struct_usage.rs,
  accepts.rs, returns.rs

Verification:
- cargo build -p db: ✅ compiles successfully
- cargo test -p db --lib: ✅ 37 tests pass
- No import warnings

The db crate is now fully functional with all query logic.

Relates to ticket: scratch/tickets/04-move-queries.md
Implements ticket 5 of 8 - Move test infrastructure.

Changes:
- Copy all fixtures to db/src/fixtures/
  - call_graph.json, type_signatures.json, structs.json
  - output/ directory with all command output fixtures
  - extracted_trace.json for large trace tests
- Copy and modify test_utils.rs to db/src/test_utils.rs
  - Remove Execute-dependent functions (execute_cmd, execute_on_empty_db)
  - Keep db-specific helpers (setup_test_db, create_temp_json_file, etc.)
  - Add #![cfg(feature = "test-utils")] feature gate
- Update db/src/lib.rs with feature-gated test modules
- Update db/Cargo.toml:
  - Add tempfile and serde_json as optional dependencies
  - Define test-utils feature
- Update db.rs and import.rs to conditionally export test functions

Functions available in test-utils feature:
- create_temp_json_file() - Create temporary JSON for testing
- setup_test_db() - Set up test database with fixture data
- setup_empty_test_db() - Create empty database for error testing
- call_graph_db(), type_signatures_db(), structs_db() - Pre-configured DBs
- load_output_fixture() - Load expected output for validation

Verification:
- cargo test -p db: ✅ 37 tests pass
- cargo build -p db --features test-utils: ✅ compiles

The db crate is now complete with all core functionality and test infrastructure.
CLI crate can use test helpers via: db = { path = "../db", features = ["test-utils"] }

Relates to ticket: scratch/tickets/05-test-infrastructure.md
Implements ticket 6 of 8 - Migrate CLI source files.

Changes:
- Copy core CLI files to cli/src/:
  - main.rs (entry point, module declarations, CLI runner)
  - cli.rs (clap argument parser)
  - output.rs (OutputFormat trait, Outputable trait)
  - dedup.rs (deduplication utilities)
  - test_macros.rs (test infrastructure macros)
- Copy entire commands/ directory to cli/src/commands/
  - All 27 command subdirectories with execute.rs, output.rs, models.rs
- Create cli/src/utils.rs with presentation logic only:
  - group_by_module, group_by_module_with_file, group_calls
  - convert_to_module_groups
  - format_type_definition and supporting functions
  - Excluded: ConditionBuilder, OptionalConditionBuilder (moved to db crate)

Commands migrated:
- accepts, boundaries, browse_module, calls_from, calls_to
- clusters, complexity, cycles, depended_by, depends_on
- describe, duplicates, function, god_modules, hotspots
- import, large_functions, location, many_clauses, path
- returns, reverse_trace, search, setup, struct_usage
- trace, unused

Note: Import paths still point to old crate:: locations and will be
updated in ticket 7. The cli crate won't compile yet - this is expected.

Original src/ directory preserved for reference.

Relates to ticket: scratch/tickets/06-migrate-cli-files.md
Implements ticket 7 of 8 - Update CLI import paths.

This is the critical integration ticket that connects the cli crate to the db crate.

Changes (100+ files modified):

Core files:
- cli/src/main.rs: Remove pub mod types, import db::open_db
- cli/src/commands/mod.rs: Use db::DbInstance in Execute trait
- cli/src/output.rs: Import db::types::ModuleGroupResult
- cli/src/utils.rs: Import db::types::{ModuleGroup, Call}

All 27 commands updated:
- execute.rs files: crate::queries::* → db::queries::*
- execute.rs files: crate::types::* → db::types::*
- mod.rs files: cozo::DbInstance → db::DbInstance
- output.rs files: crate::types::* → db::types::*
- test files: Updated imports to use db:: prefix

Commands updated:
- accepts, boundaries, browse_module, calls_from, calls_to
- clusters, complexity, cycles, depended_by, depends_on
- describe, duplicates, function, god_modules, hotspots
- import, large_functions, location, many_clauses, path
- returns, reverse_trace, search, setup, struct_usage
- trace, unused

Orphan rule fixes (impl blocks on external types):
Converted 11 impl blocks to standalone builder functions:
- build_accepts_result() (accepts)
- build_calls_from_result() (calls_from)
- build_callee_result() (calls_to)
- build_trace_result() (trace)
- build_dependent_caller_result() (depended_by)
- build_dependency_result() (depends_on)
- build_function_signatures_result() (function)
- build_return_info_result() (returns)
- build_reverse_trace_result() (reverse_trace)
- build_usage_info_result() + build_struct_modules_result() (struct_usage)
- build_unused_functions_result() (unused)

Database crate updates:
- db/src/lib.rs: Re-export DbInstance for convenient access
- Added cozo to db crate public API

Import statistics:
- 0 remaining crate::queries or crate::types references
- 81+ new db:: imports added across all command files

Verification:
- cargo build: ✓ SUCCESS
- rg "use crate::(queries|types)" cli/src/: no matches
- All imports now properly reference db crate

The workspace is now fully integrated with proper crate boundaries.
The entire workspace compiles successfully.

Relates to ticket: scratch/tickets/07-update-imports.md
Fixed broken tests from workspace migration:
- Update template paths in setup command to reference workspace root
  (Changed from $CARGO_MANIFEST_DIR/templates/ to $CARGO_MANIFEST_DIR/../templates/)
- All 492 CLI tests + 40 DB tests now passing

Enable previously ignored doctest:
- Add proper imports to query_builders.rs doctest example
- 1 doctest now passing (1 remains intentionally ignored)

Apply clippy fixes (37 auto-fixes + 1 manual):
- Use .as_deref() instead of .as_ref().map(|s| s.as_str())
- Use $crate instead of crate in test macro definitions
- Fix PathBuf comparison in main.rs
- Collapse nested if statements
- Remove unused 'visited' parameter from dfs_find_cycles()
  (Algorithm already uses path vector for cycle detection)

Clean up old src/ directory (231 files deleted):
- Removed all files from previous single-crate structure
- Workspace structure now complete with db/ and cli/ crates

Results:
- All 533 tests passing (492 CLI + 40 DB + 1 doctest)
- Clippy warnings reduced from 41 to 15 (63% reduction)
- Remaining warnings are acceptable design decisions
@CamonZ CamonZ changed the title Refactor: Extract database layer into separate crate (Ticket 1/8) Refactor: Extract database layer into separate crate Dec 23, 2025
@CamonZ CamonZ merged commit 7f71b6d into master Dec 23, 2025
1 check passed
@CamonZ CamonZ deleted the refactor-db-crate branch December 23, 2025 01:55
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant