Skip to content

Conversation

@NandaScott
Copy link
Owner

This PR adds an integration test that uses the actual Scryfall API to ensure that this library doesn't drift from their API, and makes sure that this library actually works. Another Github action was added to run this only when a PR is opened from develop -> main ensuring that we don't accidentally upset Scryfall.

@claude
Copy link

claude bot commented Jan 24, 2026

Pull Request Review: Integration Tests

Summary

This PR adds comprehensive integration tests that make real API calls to Scryfall, along with a GitHub Actions workflow to run these tests on PRs from develop to main. This is a valuable addition that helps detect API drift and ensures the library works correctly in production.

Strengths

  1. Well-documented purpose: The docstring clearly explains what integration tests do and when to run them
  2. Rate limiting respect: Proper 100ms delays between API calls to respect Scryfall rate limits
  3. Comprehensive coverage: Tests all major modules (cards, sets, bulk_data, rulings, catalogs, symbology)
  4. Smart workflow trigger: Only runs on PRs to main from develop or manual dispatch, avoiding excessive API calls
  5. Good separation: Integration tests are properly isolated from unit tests via pytest configuration

Critical Issues

1. Missing init.py in integration directory (tests/integration/)

  • The tests/integration/ directory needs an init.py file for pytest to discover it properly
  • Without this, pytest tests/integration/ -v will not work as documented

2. Integration tests excluded from pytest discovery (pytest.ini:5)

  • norecursedirs = tests/integration completely excludes integration tests from pytest
  • This conflicts with the docstring that says to run via pytest
  • Should use markers instead: run tests normally but skip them by default with -m not integration

3. Missing pytest dependency in workflow (.github/workflows/integration-tests.yml:28)

  • The workflow only installs pip install -e . but the script can also be run via pytest
  • Should install dev dependencies: pip install -e .[dev] for consistency

4. Global mutable state (tests/integration/integration_test.py:77)

  • results = IntegrationTestResults() is a module-level global that causes issues if run multiple times
  • Should be passed as parameter or recreated in main()

Code Quality Issues

  1. Inconsistent error handling: run_test() catches all exceptions with bare except Exception - should at least log the exception type
  2. Tests continue after critical failures: If test_import() fails, subsequent tests will fail with confusing errors
  3. Silent test failures in dependent tests: Many tests depend on previous test results (if card:) but do not record blocked/dependent failures
  4. Weak assertions: Tests only verify properties exist, not their types/values

Performance Considerations

  1. Sequential test execution: All tests run sequentially with 100ms delays (10+ seconds for 100+ tests). Consider making RATE_LIMIT_DELAY configurable via environment variable
  2. Large catalog downloads: CardNames() downloads 20,000+ cards but only checks total_values > 0

Recommendations

High Priority

  1. Add tests/integration/init.py
  2. Fix pytest.ini: remove norecursedirs, use marker filtering in workflow
  3. Fix global state: move results into main()
  4. Update workflow to install dev dependencies

Suggested Fix for pytest.ini

Remove the norecursedirs line and update .github/workflows/tests.yml to exclude integration tests:

pytest -m not integration --cov=scrython --cov-report=term-missing --cov-report=xml

Conclusion

This is a valuable addition to the project that will help catch API drift early. The implementation is thorough and well-structured. However, there are some integration issues with pytest that should be fixed before merging to ensure the tests can be run both ways as documented.

Recommendation: Request changes to fix critical issues 1-4, then approve.

Repository owner deleted a comment from claude bot Jan 25, 2026
Repository owner deleted a comment from claude bot Jan 25, 2026
Repository owner deleted a comment from claude bot Jan 25, 2026
Repository owner deleted a comment from claude bot Jan 25, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant