add fivetran_union_relations macro

.github/PULL_REQUEST_TEMPLATE/maintainer_pull_request_template.md

@@ -0,0 +1,51 @@
+## PR Overview
+**This PR will address the following Issue/Feature:**
+
+**This PR will result in the following new package version:**
+<!--- Please add details around your decision for breaking vs non-breaking version upgrade. If this is a breaking change, were backwards-compatible options explored? -->
+
+**Please detail what change(s) this PR introduces and any additional information that should be known during the review of this PR:**
+
+## PR Checklist
+### Basic Validation
+Please acknowledge that you have successfully performed the following commands locally:
+- [ ] dbt compile
+- [ ] dbt run –full-refresh
+- [ ] dbt run
+- [ ] dbt test
+- [ ] dbt run –vars (if applicable)
+
+Before marking this PR as "ready for review" the following have been applied:
+- [ ] The appropriate issue has been linked and tagged
+- [ ] You are assigned to the corresponding issue and this PR
+- [ ] BuildKite integration tests are passing
+
+### Detailed Validation
+Please acknowledge that the following validation checks have been performed prior to marking this PR as "ready for review":
+- [ ] You have validated these changes and assure this PR will address the respective Issue/Feature.
+- [ ] You are reasonably confident these changes will not impact any other components of this package or any dependent packages.
+- [ ] You have provided details below around the validation steps performed to gain confidence in these changes.
+<!--- Provide the steps you took to validate your changes below. -->
+
+### Standard Updates
+Please acknowledge that your PR contains the following standard updates:
+- Package versioning has been appropriately indexed in the following locations:
+    - [ ] indexed within dbt_project.yml
+    - [ ] indexed within integration_tests/dbt_project.yml
+- [ ] CHANGELOG has individual entries for each respective change in this PR
+    <!--- If there is a parallel upstream change, remember to reference the corresponding CHANGELOG as an individual entry.  -->
+- [ ] README updates have been applied (if applicable)
+    <!--- Remember to check the following README locations for common updates. →
+        <!--- Suggested install range (needed for breaking changes) →
+        <!--- Dependency matrix is appropriately updated (if applicable) →
+        <!--- New variable documentation (if applicable) -->
+- [ ] DECISIONLOG updates have been updated (if applicable)
+- [ ] Appropriate yml documentation has been added (if applicable)
+
+### dbt Docs
+Please acknowledge that after the above were all completed the below were applied to your branch:
+- [ ] docs were regenerated (unless this PR does not include any code or yml updates)
+
+### If you had to summarize this PR in an emoji, which would it be?
+<!--- For a complete list of markdown compatible emojis check our this git repo (https://gist.github.com/rxaviers/7360908)  --> 
+:dancer:

.github/pull_request_template.md

@@ -1,53 +1,33 @@
-Pull Request
-**Are you a current Fivetran customer?** 
-<!--- Please tell us your name, title and company -->
+**Please provide your name and company**
 
-**What change(s) does this PR introduce?** 
-<!--- Describe what changes your PR introduces to the package and how to leverage this new feature. -->
+**Link the issue/feature request which this PR is meant to address**
+<!--- If an issue was not created, please create one first so we may discuss the PR prior to opening one. -->
 
-**Did you update the CHANGELOG?** 
+**Detail what changes this PR introduces and how this addresses the issue/feature request linked above.**
+
+**How did you validate the changes introduced within this PR?**
+
+**Which warehouse did you use to develop these changes?**
+
+**Did you update the CHANGELOG?**
 <!--- Please update the new package version’s CHANGELOG entry detailing the changes included in this PR. -->
 <!--- To select a checkbox you simply need to add an "x" with no spaces between the brackets (eg. [x] Yes). -->
 - [ ] Yes
 
-**Does this PR introduce a breaking change?**
-<!--- Does this PR introduce changes that will cause current package users' jobs to fail or require a `--full-refresh`? -->
-<!--- To select a checkbox you simply need to add an "x" with no spaces between the brackets (eg. [x] Yes). -->
-- [ ] Yes (please provide breaking change details below.)
-- [ ] No  (please provide an explanation as to how the change is non-breaking below.)
-
-**Did you update the dbt_project.yml files with the version upgrade (please leverage standard semantic versioning)? (In both your main project and integration_tests)** 
+**Did you update the dbt_project.yml files with the version upgrade (please leverage standard semantic versioning)? (In both your main project and integration_tests)**
 <!--- The dbt_project.yml and the integration_tests/dbt_project.yml files contain the version number. Be sure to upgrade it accordingly -->
 <!--- To select a checkbox you simply need to add an "x" with no spaces between the brackets (eg. [x] Yes). -->
 - [ ] Yes
 
-**Is this PR in response to a previously created Bug or Feature Request**
-<!--- If an Issue was created it is helpful to track the progress by linking it in the PR. -->
-<!--- To select a checkbox you simply need to add an "x" with no spaces between the brackets (eg. [x] Yes). -->
-- [ ] Yes, Issue/Feature [link bug/feature number here]
-- [ ] No 
-
-**How did you test the PR changes?** 
-<!--- Proof of testing is required in order for the PR to be approved. -->
-<!--- To check a box, remove the space and insert an x in the box (eg. [x] Buildkite). --> 
-<!--- To select a checkbox you simply need to add an "x" with no spaces between the brackets (eg. [x] Yes). -->
-- [ ] Buildkite <!--- Buildkite testing is only applicable to Fivetran employees. --> 
-- [ ] Local (please provide additional testing details below)
-
-**Select which warehouse(s) were used to test the PR**
-<!--- To check a warehouse remove the space and insert an x in the box (eg. [x] Bigquery). --> 
-<!--- To select a checkbox you simply need to add an "x" with no spaces between the brackets (eg. [x] Yes). -->
-- [ ] BigQuery
-- [ ] Redshift
-- [ ] Snowflake
-- [ ] Postgres
-- [ ] Databricks
-- [ ] Other (provide details below)
-
 **Provide an emoji that best describes your current mood**
 <!--- For a complete list of markdown compatible emojis check our this git repo (https://gist.github.com/rxaviers/7360908)  --> 
 :dancer:
 
 **Feedback**
 
 We are so excited you decided to contribute to the Fivetran community dbt package! We continue to work to improve the packages and would greatly appreciate your [feedback](https://www.surveymonkey.com/r/DQ7K7WW) on our existing dbt packages or what you'd like to see next.
+
+**PR Template** 
+- [Community Pull Request Template](?expand=1&template=pull_request_template.md) (default)
+
+- [Maintainer Pull Request Template](?expand=1&template=maintainer_pull_request_template.md) (to be used by maintainers)

CHANGELOG.md

@@ -1,3 +1,25 @@
+# dbt_fivetran_utils v0.4.11
+
+## Feature Updates
+[PR #139](https://github.com/fivetran/dbt_fivetran_utils/pull/139) includes the following update:
+- Added the new `fivetran_union_relations()` macro.
+  - This was necessary for adding `union_data()` support to Zendesk the package, which works with some tables that have reserved-keywords for names (ie timezone).
+  - We heavily leveraged `dbt_utils.union_relations()` but adjusted it to work with these problematically named source tables.
+- Adjusted the `union_data()` macro:
+  - Now references `fivetran_union_relations()` instead of `dbt_utils.union_relations()`.
+  - Now includes a new optional argument, `connector_table_name_override`, which is the _actual_ name/identifier of the table as it appears in the connector schema. Currently, this is only used in our Netsuite2 data models, whose actual table names and defined `source` table names differ slightly due to our support of both the Netsuite1 and Netsuite2 endpoints.
+  - Includes a reference to a new `use_table_name_identifer_override` global variable. This variable was introduced to accommodate the use of `identifier` variables in packages where `connector_table_name_override` is used (again, just Netsuite for now).
+
+> Note: We already do have a macro called `union_relations()` in this package, but as we learned [recently](https://github.com/fivetran/dbt_fivetran_utils/releases/tag/v0.4.10), using the same exact name as a macro in `dbt_utils` is not a good idea. Thus, `fivetran_utils.union_relations()` will be deprecated in favor of `fivetran_utils.fivetran_union_relations()` in the future. Currently it is used in only one package (Marketo).
+
+[PR #145](https://github.com/fivetran/dbt_fivetran_utils/pull/145) includes the following update:
+- Added the new `fivetran_lookback()` macro. This generates SQL to simplify adding lookback windows to incremental models. See [README](https://github.com/fivetran/dbt_fivetran_utils/blob/releases/v0.4.latest/README.md#fivetran_lookback-source).
+- Added the new `fivetran_is_databricks_sql_warehouse()` macro. This determines if Databricks destination is a SQL Warehouse or All Purpose Cluster. See [README](https://github.com/fivetran/dbt_fivetran_utils/blob/releases/v0.4.latest/README.md#fivetran-is-databricks-sql-warehouse-source).
+
+## Under the Hood
+[PR #139](https://github.com/fivetran/dbt_fivetran_utils/pull/139) includes the following update:
+- Updated the pull request [templates](https://github.com/fivetran/dbt_fivetran_utils/tree/releases/v0.4.latest/.github) to align with our current format.
+
 # dbt_fivetran_utils v0.4.10
 
 ## Bug Fix

README.md

@@ -53,6 +53,7 @@ dispatch:
 - [📋 Contents](#-contents)
   - [Tests and helpers](#tests-and-helpers)
     - [collect\_freshness (source)](#collect_freshness-source)
+    - [fivetran\_is\_databricks\_sql\_warehouse (source)](#fivetran-is-databricks-sql-warehouse-source)
     - [seed\_data\_helper (source)](#seed_data_helper-source)
     - [snowflake\_seed\_data (source)](#snowflake_seed_data-source)
   - [Cross-database compatibility](#cross-database-compatibility)
@@ -79,11 +80,14 @@ dispatch:
     - [dummy\_coalesce\_value (source)](#dummy_coalesce_value-source)
     - [fill\_pass\_through\_columns (source)](#fill_pass_through_columns-source)
     - [fill\_staging\_columns (source)](#fill_staging_columns-source)
+    - [fivetran\_lookback (source)](#fivetran_lookback-source)
     - [persist\_pass\_through\_columns (source)](#persist_pass_through_columns-source)
     - [remove\_prefix\_from\_columns (source)](#remove_prefix_from_columns-source)
     - [source\_relation (source)](#source_relation-source)
     - [union\_data (source)](#union_data-source)
+      - [Using identifiers in packages that have `union_data` and include the `connector_table_name_override` argument (currently only Netsuite2)](#using-identifiers-in-packages-that-have-union_data-and-include-the-connector_table_name_override-argument-currently-only-netsuite2)
       - [Union Data Defined Sources Configuration](#union-data-defined-sources-configuration)
+    - [fivetran\_union\_relations (source)](#fivetran_union_relations-source)
     - [union\_relations (source)](#union_relations-source)
   - [Variable Checks](#variable-checks)
     - [empty\_variable\_warning (source)](#empty_variable_warning-source)
@@ -118,6 +122,17 @@ sources:
 **Args (sorta):**
 * `meta.is_enabled` (optional): The variable(s) you would like to reference to determine if dbt should include this table in freshness tests.
 
+----
+### fivetran_is_databricks_sql_warehouse ([source](macros/fivetran_is_databricks_sql_warehouse.sql))
+For Databricks destinations, this macro returns `true` if the Databricks target indicates it is a SQL Warehouse. It will return return `false` if it is an All-Purpose Cluster. 
+
+***Usage:**
+```yml
+    fivetran_utils.fivetran_is_databricks_sql_warehouse()
+```
+**Args:**
+* none
+
 ----
 ### seed_data_helper ([source](macros/seed_data_helper.sql))
 This macro is intended to be used when a source table column is a reserved keyword in a warehouse, and Circle CI is throwing a fit.
@@ -464,6 +479,28 @@ from source
 * `source_columns`  (required): Will call the [get_columns_in_relation](https://docs.getdbt.com/reference/dbt-jinja-functions/adapter/#get_columns_in_relation) macro as well requires a `ref()` or `source()` argument for the staging models within the `_tmp` directory.
 * `staging_columns` (required): Created as a result of running the [generate_columns_macro](https://github.com/fivetran/dbt_fivetran_utils#generate_columns_macro-source) for the respective table.
 
+----
+### fivetran_lookback ([source](macros/fivetran_lookback.sql))
+This macro takes a date expression and uses `dbt.dateadd` and backdates it by the specified interval. This is intended for use in incremental blocks to look backwards and catch late arriving records. Any date aggregate can be used as an input, but the typical usage is with the max date of the present model.
+
+**Usage:**
+```sql
+{% if is_incremental() %}
+  where date_day >= 
+    {{ fivetran_utils.fivetran_lookback(
+      from_date='max(date_day)',
+      interval=3,
+      datepart='day', 
+      safety_date='2010-01-01')
+    }}
+{% endif %}
+```
+**Args:**
+* `from_date`  (required): String to run against the current model. The expression used should be expected to return a single result.
+* `datepart` (required): The grain of the interval. 
+* `interval` (required): The number of units to look backwards.
+* `safety_date` (optional): This date will be used in the rare case that the `from_date` expression returns a null value. This only needs to be specified if you want to change the default value from '2010-01-01'.
+
 ----
 ### persist_pass_through_columns ([source](macros/persist_pass_through_columns.sql))
 This macro is used to persist pass through columns from the staging model to the **transform** package. This is particularly helpful when a `select *` is not feasible.
@@ -518,6 +555,12 @@ To create dependencies between the unioned model and its *sources*, you **must d
 
 If the source table is not found in any of the provided schemas/databases, `union_data` will return a **completely** empty table (ie `limit 0`) with just one string column (`_dbt_source_relation`). A compiler warning message will be output, highlighting that the expected source table was not found and its respective staging model is empty. The compiler warning can be turned off by the end user by setting the `fivetran__remove_empty_table_warnings` variable to `True`.
 
+```yml
+# in root dbt_project.yml file
+vars:
+  fivetran__remove_empty_table_warnings: true # false by default
+```
+
 **Usage:**
 ```sql
 -- in model.sql file
@@ -541,15 +584,27 @@ If the source table is not found in any of the provided schemas/databases, `unio
 * `default_variable`: The name of the variable that users should populate when they want to pass one specific relation to this model (mostly used for CI)
 * `union_schema_variable` (optional): The name of the union schema variable. By default the macro will look for `union_schemas`.
 * `union_database_variable` (optional): The name of the union database variable. By default the macro will look for `union_databases`.
+* `connector_table_name_override` (optional): The _actual_ name/identifier of the table as it appears in the connector schema. Currently, this is only used in our Netsuite2 data models, whose actual table names and defined `source` table names differ slightly due to our support of both the Netsuite1 and Netsuite2 endpoints.
+  * If this argument is used, refer to the below subsection.
+
+#### Using identifiers in packages that have `union_data` and include the `connector_table_name_override` argument (currently only Netsuite2)
+In our integration tests, we often have seed files that do not have the default names for their respective tables. They might have a `_data` suffix or something along those lines. In these cases, we make use of our identifier variables in the `integration_tests/dbt_project.yml` file. To ensure that the `union_data` macro picks these custom name configs up and does not use the `connector_table_name_override` (only a thing in Netsuite), set the following variable to `true` in the package's `integration_tests/dbt_project.yml` file:
+
+```yml
+vars:
+  use_table_name_identifer_override: true
+```
+
+This can also be used by customers, but is **only relevant to Netsuite2** because the `source` table names != actual connector table names.
 
 #### Union Data Defined Sources Configuration
+To create dependencies between the unioned model and its *sources*, you **must define** the source tables in a `.yml` file in your project and set the `has_defined_sources` variable (scoped to the source package in which the macro is being called) to `True` in your `dbt_project.yml` file. If you set `has_defined_sources` to true and do not define sources (at least adding the `name` of each table in the source), dbt will throw an error.
+
 ```yml
 # in root dbt_project.yml file
 vars:
-  shopify_source:
+  shopify_source: # or whatever source package
     has_defined_sources: true
-
-  fivetran__remove_empty_table_warnings: true # false by default
 ```
 
 ```yml
@@ -577,8 +632,36 @@ sources:
       - name: customer 
       ...
 ```
+
+----
+### fivetran_union_relations ([source](macros/union_relations.sql))
+Heavily adapted from [dbt_utils.union_relations()](https://github.com/dbt-labs/dbt-utils?tab=readme-ov-file#union_relations-source). The primary difference is that our version of the macro supports unioning relations that have reserved keywords as names (ex: table called `timezone`).
+
+This macro combines via a `union all` of an array of [Relations](https://docs.getdbt.com/reference/dbt-classes#relation), even when columns have differing orders in each Relation, and/or some columns are missing from some relations. Any columns exclusive to a subset of these relations will be filled with `null` where not present. A new column (`_dbt_source_relation`) is also added to indicate the source for each record.
+
+**Usage:**
+```sql
+{{ fivetran_utils.fivetran_union_relations(
+    relations=[ref('my_model'), source('my_source', 'my_table')],
+    exclude=["_loaded_at"],
+    aliases=['my_model_cte', 'my_source_table_cte']
+) }}
+```
+**Args:**
+* `relations`          (required): An array of [Relations](https://docs.getdbt.com/docs/writing-code-in-dbt/class-reference/#relation).
+* `aliases`            (optional): An override of the relation identifier. This argument should be populated with the overwritten alias for each relation (ie if we are selecting from a CTE). If not populated `relations` will be the default. This argument is not included in the `dbt_utils` version of this macro.
+* `exclude`            (optional): A list of column names that should be excluded from the final query.
+* `include`            (optional): A list of column names that should be included in the final query. Note the `include` and `exclude` parameters are mutually exclusive.
+* `column_override`    (optional): A dictionary of explicit column type overrides, e.g. `{"some_field": "varchar(100)"}`.``
+* `source_column_name` (optional, `default="_dbt_source_relation"`): The name of the column that records the source of this row. Pass `None` to omit this column from the results.
+* `where` (optional): Filter conditions to include in the `where` clause.
+
 ----
 ### union_relations ([source](macros/union_relations.sql))
+> TO BE DEPRECATED IN FAVOR OF `fivetran_union_relations`.
+>
+> Currenlty only used in [Marketo](https://github.com/fivetran/dbt_marketo/blob/main/models/intermediate/marketo__change_data_scd.sql#L47) transform package.
+
 This macro unions together an array of [Relations](https://docs.getdbt.com/docs/writing-code-in-dbt/class-reference/#relation),
 even when columns have differing orders in each Relation, and/or some columns are
 missing from some relations. Any columns exclusive to a subset of these
@@ -587,7 +670,7 @@ relations will be filled with `null` where not present. An new column
 
 **Usage:**
 ```sql
-{{ dbt_utils.union_relations(
+{{ fivetran_utils.union_relations(
     relations=[ref('my_model'), source('my_source', 'my_table')],
     exclude=["_loaded_at"]
 ) }}

dbt_project.yml

@@ -1,4 +1,4 @@
 name: 'fivetran_utils'
-version: '0.4.10'
+version: '0.4.11'
 config-version: 2
 require-dbt-version: [">=1.3.0", "<2.0.0"]