From fcd0803efa6eaea839af705977f72308492a91ac Mon Sep 17 00:00:00 2001 From: Jamie Rodriguez <65564846+fivetran-jamie@users.noreply.github.com> Date: Mon, 23 Jun 2025 10:44:18 -0400 Subject: [PATCH 1/3] Update freshness config --- .../maintainer_pull_request_template.md | 52 +++++++++++-------- CHANGELOG.md | 28 +++++++++- LICENSE | 2 +- README.md | 8 +-- dbt_project.yml | 2 +- integration_tests/dbt_project.yml | 2 +- models/src_amazon_ads.yml | 10 ++-- 7 files changed, 69 insertions(+), 35 deletions(-) diff --git a/.github/PULL_REQUEST_TEMPLATE/maintainer_pull_request_template.md b/.github/PULL_REQUEST_TEMPLATE/maintainer_pull_request_template.md index 3220674..99c450c 100644 --- a/.github/PULL_REQUEST_TEMPLATE/maintainer_pull_request_template.md +++ b/.github/PULL_REQUEST_TEMPLATE/maintainer_pull_request_template.md @@ -1,29 +1,35 @@ -## PR Overview -**This PR will address the following Issue/Feature:** + +- `dbt run --full-refresh && dbt test` +- `dbt run` && `dbt test` (if incremental models are present) +- The related issue is linked, tagged, and appropriately assigned +- Documentation and version updates are included, if applicable +- `docs` have been regenerated (unless there are no code or YAML changes) +- BuildKite integration tests are passing +--> -**Please provide the finalized CHANGELOG entry which details the relevant changes included in this PR:** - +## PR Overview +**Package version introduced in this PR:** + +**This PR addresses the following Issue/Feature(s):** + -## PR Checklist -### Basic Validation -Please acknowledge that you have successfully performed the following commands locally: -- [ ] dbt run –full-refresh && dbt test -- [ ] dbt run (if incremental models are present) && dbt test +**Summary of changes:** + -Before marking this PR as "ready for review" the following have been applied: -- [ ] The appropriate issue has been linked, tagged, and properly assigned -- [ ] All necessary documentation and version upgrades have been applied -- [ ] docs were regenerated (unless this PR does not include any code or yml updates) -- [ ] BuildKite integration tests are passing -- [ ] Detailed validation steps have been provided below -### Detailed Validation -Please share any and all of your validation steps: - +### Submission Checklist +- [ ] Alignment meeting with the reviewer (if needed) + - [ ] Timeline and validation requirements discussed +- [ ] Provide validation details: + - [ ] **Validation Steps:** Check for unintentional effects (e.g., add/run consistency & integrity tests) + - [ ] **Testing Instructions:** Confirm the change addresses the issue(s) + - [ ] **Focus Areas:** Complex logic or queries that need extra attention -### If you had to summarize this PR in an emoji, which would it be? - -:dancer: \ No newline at end of file +### Changelog + +- [ ] Draft changelog for PR +- [ ] Final changelog for release review \ No newline at end of file diff --git a/CHANGELOG.md b/CHANGELOG.md index 4483314..a956eb3 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,7 +1,33 @@ -# dbt_amazon_ads_source version.version +# dbt_amazon_ads_source v0.5.0 + +[PR #24](https://github.com/fivetran/dbt_amazon_ads_source/pull/24) includes the following updates: + +## Breaking Change for dbt Core < 1.9.5 +> *Note: This is not relevant to Fivetran Quickstart users.* + +Migrated `freshness` from a top-level source property to a source `config` in alignment with [recent updates](https://github.com/dbt-labs/dbt-core/issues/11506) from dbt Core. This will resolve the following deprecation warning that users running dbt >= 1.9.5 may have received: + +``` +[WARNING]: Deprecated functionality +Found `freshness` as a top-level property of `amazon_ads` in file +`models/src_amazon_ads.yml`. The `freshness` top-level property should be moved +into the `config` of `amazon_ads`. +``` + +**IMPORTANT:** Users running dbt Core < 1.9.5 will not be able to utilize freshness tests in this release or any subsequent releases, as older versions of dbt will not recognize freshness as a source `config` and therefore not run the tests. + +If you are using dbt Core < 1.9.5 and want to continue running Amazon Ads freshness tests, please elect **one** of the following options: + 1. (Recommended) Upgrade to dbt Core >= 1.9.5 + 2. Do not upgrade your installed version of the `amazon_ads_source` package. Pin your dependency on v0.4.0 in your `packages.yml` file. + 3. Utilize a dbt [override](https://docs.getdbt.com/reference/resource-properties/overrides) to overwrite the package's `amazon_ads` source and apply freshness via the [old](https://github.com/fivetran/dbt_amazon_ads_source/blob/main/models/src_amazon_ads.yml#L11-L13) top-level property route. This will require you to copy and paste the entirety of the `src_amazon_ads.yml` [file](https://github.com/fivetran/dbt_amazon_ads_source/blob/main/models/src_amazon_ads.yml#L4-L369) and add an `overrides: amazon_ads_source` property. + +## Under the Hood +- Updated the package maintainer PR template. ## Documentation - Corrected references to connectors and connections in the README. ([#23](https://github.com/fivetran/dbt_amazon_ads_source/pull/23)) +- Updated LICENSE. +- Updated README headers. # dbt_amazon_ads_source v0.4.0 [PR #21](https://github.com/fivetran/dbt_amazon_ads_source/pull/21) includes the following updates: diff --git a/LICENSE b/LICENSE index 261eeb9..f9554a9 100644 --- a/LICENSE +++ b/LICENSE @@ -186,7 +186,7 @@ same "printed page" as the copyright notice for easier identification within third-party archives. - Copyright [yyyy] [name of copyright owner] + Copyright © 2025 Fivetran Inc. Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. diff --git a/README.md b/README.md index 5cb5542..bc8daf5 100644 --- a/README.md +++ b/README.md @@ -1,4 +1,6 @@ -

+# Amazon Ads Source dbt Package ([Docs](https://fivetran.github.io/dbt_amazon_ads_source/)) + +

@@ -10,11 +12,11 @@

-# Amazon Ads Source dbt Package ([Docs](https://fivetran.github.io/dbt_amazon_ads_source/)) ## What does this dbt package do? - Materializes [Amazon Ads staging tables](https://fivetran.github.io/dbt_amazon_ads_source/#!/overview/amazon_ads_source/models/?g_v=1&g_e=seeds), which leverage data in the format described by [this ERD](https://fivetran.com/docs/applications/amazon-ads#schemainformation). These staging tables clean, test, and prepare your Amazon Ads data from [Fivetran's connector](https://fivetran.com/docs/applications/amazon-ads) for analysis by doing the following: - Names columns for consistency across all packages and for easier analysis - Adds freshness tests to source data + > dbt Core >= 1.9.5 is required to run freshness tests out of the box. See other options [here](https://github.com/fivetran/dbt_amazon_ads_source/blob/main/CHANGELOG.md#breaking-change-for-dbt-core--195). - Adds column-level testing where applicable. For example, all primary keys are tested for uniqueness and non-null values. - Generates a comprehensive data dictionary of your Amazon Ads data through the [dbt docs site](https://fivetran.github.io/dbt_amazon_ads_source/). - These tables are designed to work simultaneously with our [Amazon Ads transformation package](https://github.com/fivetran/dbt_amazon_ads). @@ -32,7 +34,7 @@ If you are **not** using the [Amazon Ads transformation package](https://github ```yaml packages: - package: fivetran/amazon_ads_source - version: [">=0.4.0", "<0.5.0"] # we recommend using ranges to capture non-breaking changes automatically + version: [">=0.5.0", "<0.6.0"] # we recommend using ranges to capture non-breaking changes automatically ``` ### Step 3: Define database and schema variables diff --git a/dbt_project.yml b/dbt_project.yml index 954a9c2..4fcc397 100644 --- a/dbt_project.yml +++ b/dbt_project.yml @@ -1,5 +1,5 @@ name: 'amazon_ads_source' -version: '0.4.0' +version: '0.5.0' config-version: 2 require-dbt-version: [">=1.3.0", "<2.0.0"] diff --git a/integration_tests/dbt_project.yml b/integration_tests/dbt_project.yml index 0670361..d37ae4c 100644 --- a/integration_tests/dbt_project.yml +++ b/integration_tests/dbt_project.yml @@ -1,7 +1,7 @@ config-version: 2 name: 'amazon_ads_source_integration_tests' -version: '0.4.0' +version: '0.5.0' profile: 'integration_tests' diff --git a/models/src_amazon_ads.yml b/models/src_amazon_ads.yml index 854c8fa..5c589be 100644 --- a/models/src_amazon_ads.yml +++ b/models/src_amazon_ads.yml @@ -8,12 +8,12 @@ sources: loader: Fivetran loaded_at_field: _fivetran_synced - freshness: - warn_after: {count: 48, period: hour} - error_after: {count: 168, period: hour} config: - enabled: "{{ var('ad_reporting__amazon_ads_enabled', true) }}" - + enabled: "{{ var('ad_reporting__amazon_ads_enabled', true) }}" + freshness: + warn_after: {count: 48, period: hour} + error_after: {count: 168, period: hour} + tables: - name: ad_group_history description: Each record represents an ad group in Amazon Ads. From 2da22bbc974747f15e73576dd918896187576cf7 Mon Sep 17 00:00:00 2001 From: Jamie Rodriguez <65564846+fivetran-jamie@users.noreply.github.com> Date: Mon, 23 Jun 2025 11:04:04 -0400 Subject: [PATCH 2/3] snowflake --- integration_tests/requirements.txt | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/integration_tests/requirements.txt b/integration_tests/requirements.txt index 810bae1..8c55c23 100644 --- a/integration_tests/requirements.txt +++ b/integration_tests/requirements.txt @@ -4,4 +4,5 @@ dbt-redshift>=1.3.0,<2.0.0 dbt-postgres>=1.3.0,<2.0.0 dbt-spark>=1.3.0,<2.0.0 dbt-spark[PyHive]>=1.3.0,<2.0.0 -dbt-databricks>=1.6.0,<2.0.0 \ No newline at end of file +dbt-databricks>=1.6.0,<2.0.0 +certifi==2025.1.31 \ No newline at end of file From 1e827ba6a6fc291ad95c2caf9232c16a0facc0c2 Mon Sep 17 00:00:00 2001 From: Jamie Rodriguez <65564846+fivetran-jamie@users.noreply.github.com> Date: Tue, 24 Jun 2025 16:53:33 -0400 Subject: [PATCH 3/3] replace dbt 1.9.5 with 1.9.6 --- CHANGELOG.md | 10 +++++----- README.md | 2 +- 2 files changed, 6 insertions(+), 6 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index a956eb3..940f6c5 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,10 +2,10 @@ [PR #24](https://github.com/fivetran/dbt_amazon_ads_source/pull/24) includes the following updates: -## Breaking Change for dbt Core < 1.9.5 +## Breaking Change for dbt Core < 1.9.6 > *Note: This is not relevant to Fivetran Quickstart users.* -Migrated `freshness` from a top-level source property to a source `config` in alignment with [recent updates](https://github.com/dbt-labs/dbt-core/issues/11506) from dbt Core. This will resolve the following deprecation warning that users running dbt >= 1.9.5 may have received: +Migrated `freshness` from a top-level source property to a source `config` in alignment with [recent updates](https://github.com/dbt-labs/dbt-core/issues/11506) from dbt Core. This will resolve the following deprecation warning that users running dbt >= 1.9.6 may have received: ``` [WARNING]: Deprecated functionality @@ -14,10 +14,10 @@ Found `freshness` as a top-level property of `amazon_ads` in file into the `config` of `amazon_ads`. ``` -**IMPORTANT:** Users running dbt Core < 1.9.5 will not be able to utilize freshness tests in this release or any subsequent releases, as older versions of dbt will not recognize freshness as a source `config` and therefore not run the tests. +**IMPORTANT:** Users running dbt Core < 1.9.6 will not be able to utilize freshness tests in this release or any subsequent releases, as older versions of dbt will not recognize freshness as a source `config` and therefore not run the tests. -If you are using dbt Core < 1.9.5 and want to continue running Amazon Ads freshness tests, please elect **one** of the following options: - 1. (Recommended) Upgrade to dbt Core >= 1.9.5 +If you are using dbt Core < 1.9.6 and want to continue running Amazon Ads freshness tests, please elect **one** of the following options: + 1. (Recommended) Upgrade to dbt Core >= 1.9.6 2. Do not upgrade your installed version of the `amazon_ads_source` package. Pin your dependency on v0.4.0 in your `packages.yml` file. 3. Utilize a dbt [override](https://docs.getdbt.com/reference/resource-properties/overrides) to overwrite the package's `amazon_ads` source and apply freshness via the [old](https://github.com/fivetran/dbt_amazon_ads_source/blob/main/models/src_amazon_ads.yml#L11-L13) top-level property route. This will require you to copy and paste the entirety of the `src_amazon_ads.yml` [file](https://github.com/fivetran/dbt_amazon_ads_source/blob/main/models/src_amazon_ads.yml#L4-L369) and add an `overrides: amazon_ads_source` property. diff --git a/README.md b/README.md index bc8daf5..2620d56 100644 --- a/README.md +++ b/README.md @@ -16,7 +16,7 @@ - Materializes [Amazon Ads staging tables](https://fivetran.github.io/dbt_amazon_ads_source/#!/overview/amazon_ads_source/models/?g_v=1&g_e=seeds), which leverage data in the format described by [this ERD](https://fivetran.com/docs/applications/amazon-ads#schemainformation). These staging tables clean, test, and prepare your Amazon Ads data from [Fivetran's connector](https://fivetran.com/docs/applications/amazon-ads) for analysis by doing the following: - Names columns for consistency across all packages and for easier analysis - Adds freshness tests to source data - > dbt Core >= 1.9.5 is required to run freshness tests out of the box. See other options [here](https://github.com/fivetran/dbt_amazon_ads_source/blob/main/CHANGELOG.md#breaking-change-for-dbt-core--195). + > dbt Core >= 1.9.6 is required to run freshness tests out of the box. See other options [here](https://github.com/fivetran/dbt_amazon_ads_source/blob/main/CHANGELOG.md#breaking-change-for-dbt-core--196). - Adds column-level testing where applicable. For example, all primary keys are tested for uniqueness and non-null values. - Generates a comprehensive data dictionary of your Amazon Ads data through the [dbt docs site](https://fivetran.github.io/dbt_amazon_ads_source/). - These tables are designed to work simultaneously with our [Amazon Ads transformation package](https://github.com/fivetran/dbt_amazon_ads).