Skip to content

Conversation

@eiffel777
Copy link
Contributor

This PR does two things.

  1. Changes the ingestion steps for creating the configuration history for an instance type. Now that we use a version of MariaDB that supports window functions we can use those to do create it instead of using PHP.
  2. Add a clause to the join statement in staging_event ingestor that joins to the instance_type table to get the correct instance type for the time the event happened. We are seeing slowness in this query because without the time clause because it joins to all rows that have the same instance_type, cpus, memory_mb, and disk_gb, which can be a lot in some cases.

A couple of notes.

  • instance_type_change_flag.json - uses LAG to set a 1 or 0 to denote if that row is where a change in configuation occurred
  • instance_type_config_group.json - uses the is_change column from modw_cloud.instance_type_change_flag and a window function to set a mark each different configuration for an instance type
  • instance_type_grouped.json - Groups all instance types using the config group to get a MIN start time for each configuration. Uses MAX on display and description columns to make it compliant with the ONLY_FULL_GROUP_BY mode. Since these column should have the same value within each group it should always have the correct value.
  • instance_type_staging.json - Sets the end time for each instance type configuration using LEAD.
  • The CloudInstanceTypeStateIngestor ingestor and it's associated test were deleted because they are no longer needed.

Tests performed

Tested in docker

Checklist:

  • The pull request description is suitable for a Changelog entry
  • The milestone is set correctly on the pull request
  • The appropriate labels have been added to the pull request

@eiffel777 eiffel777 added this to the 11.5.0 milestone Jan 29, 2026
@eiffel777 eiffel777 self-assigned this Jan 29, 2026
@eiffel777 eiffel777 added Category:ETL Extract Transform Load Category:Cloud Cloud Realm labels Jan 29, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Category:Cloud Cloud Realm Category:ETL Extract Transform Load

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant