Skip to content

Conversation

@max-ostapenko
Copy link
Contributor

This pull request centralizes and standardizes the column definitions for the pages, requests, and parsed_css tables across the project. It introduces a single source of truth for column metadata in includes/descriptions.js and updates all relevant output and sample data definitions to reference these shared descriptions, ensuring consistency and easier maintenance. Additionally, new "latest" views are added for each table, selecting only the most recent data.

Centralization of column definitions:

  • Added a new columns object to includes/descriptions.js containing detailed column definitions for pages, requests, and parsed_css, including nested descriptions for complex fields.
  • Updated all output and sample data table definitions in definitions/output/crawl, definitions/output/sample_data, and new view definitions in definitions/output/latest to import and use the centralized columns definitions instead of duplicating column metadata inline.

Addition of "latest" views:

  • Introduced new view definitions for pages, requests, and parsed_css in the latest schema, each selecting data from the most recent crawl date and referencing the centralized column definitions.

These changes improve maintainability by ensuring all tables and views use consistent, up-to-date column descriptions and make it easier to update column metadata in the future.

@max-ostapenko max-ostapenko merged commit 04e7e86 into main Dec 21, 2025
31 checks passed
@max-ostapenko max-ostapenko deleted the easy-possum branch December 21, 2025 18:34
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants