diff --git a/.github/PULL_REQUEST_TEMPLATE/maintainer_pull_request_template.md b/.github/PULL_REQUEST_TEMPLATE/maintainer_pull_request_template.md new file mode 100644 index 0000000..768ac3f --- /dev/null +++ b/.github/PULL_REQUEST_TEMPLATE/maintainer_pull_request_template.md @@ -0,0 +1,51 @@ +## PR Overview +**This PR will address the following Issue/Feature:** + +**This PR will result in the following new package version:** + + +**Please detail what change(s) this PR introduces and any additional information that should be known during the review of this PR:** + +## PR Checklist +### Basic Validation +Please acknowledge that you have successfully performed the following commands locally: +- [ ] dbt compile +- [ ] dbt run –full-refresh +- [ ] dbt run +- [ ] dbt test +- [ ] dbt run –vars (if applicable) + +Before marking this PR as "ready for review" the following have been applied: +- [ ] The appropriate issue has been linked and tagged +- [ ] You are assigned to the corresponding issue and this PR +- [ ] BuildKite integration tests are passing + +### Detailed Validation +Please acknowledge that the following validation checks have been performed prior to marking this PR as "ready for review": +- [ ] You have validated these changes and assure this PR will address the respective Issue/Feature. +- [ ] You are reasonably confident these changes will not impact any other components of this package or any dependent packages. +- [ ] You have provided details below around the validation steps performed to gain confidence in these changes. + + +### Standard Updates +Please acknowledge that your PR contains the following standard updates: +- Package versioning has been appropriately indexed in the following locations: + - [ ] indexed within dbt_project.yml + - [ ] indexed within integration_tests/dbt_project.yml +- [ ] CHANGELOG has individual entries for each respective change in this PR + +- [ ] README updates have been applied (if applicable) + +- [ ] DECISIONLOG updates have been updated (if applicable) +- [ ] Appropriate yml documentation has been added (if applicable) + +### dbt Docs +Please acknowledge that after the above were all completed the below were applied to your branch: +- [ ] docs were regenerated (unless this PR does not include any code or yml updates) + +### If you had to summarize this PR in an emoji, which would it be? + +:dancer: diff --git a/.github/pull_request_template.md b/.github/pull_request_template.md index 1b75bc7..b4e7e8e 100644 --- a/.github/pull_request_template.md +++ b/.github/pull_request_template.md @@ -1,49 +1,24 @@ -Pull Request -**Are you a current Fivetran customer?** - +**Please provide your name and company** -**What change(s) does this PR introduce?** - +**Link the issue/feature request which this PR is meant to address** + -**Did you update the CHANGELOG?** +**Detail what changes this PR introduces and how this addresses the issue/feature request linked above.** + +**How did you validate the changes introduced within this PR?** + +**Which warehouse did you use to develop these changes?** + +**Did you update the CHANGELOG?** - [ ] Yes -**Does this PR introduce a breaking change?** - - -- [ ] Yes (please provide breaking change details below.) -- [ ] No (please provide an explanation as to how the change is non-breaking below.) - -**Did you update the dbt_project.yml files with the version upgrade (please leverage standard semantic versioning)? (In both your main project and integration_tests)** +**Did you update the dbt_project.yml files with the version upgrade (please leverage standard semantic versioning)? (In both your main project and integration_tests)** - [ ] Yes -**Is this PR in response to a previously created Bug or Feature Request** - - -- [ ] Yes, Issue/Feature [link bug/feature number here] -- [ ] No - -**How did you test the PR changes?** - - - -- [ ] Buildkite -- [ ] Local (please provide additional testing details below) - -**Select which warehouse(s) were used to test the PR** - - -- [ ] BigQuery -- [ ] Redshift -- [ ] Snowflake -- [ ] Postgres -- [ ] Databricks -- [ ] Other (provide details below) - **Provide an emoji that best describes your current mood** :dancer: @@ -51,3 +26,8 @@ Pull Request **Feedback** We are so excited you decided to contribute to the Fivetran community dbt package! We continue to work to improve the packages and would greatly appreciate your [feedback](https://www.surveymonkey.com/r/DQ7K7WW) on our existing dbt packages or what you'd like to see next. + +**PR Template** +- [Community Pull Request Template](?expand=1&template=pull_request_template.md) (default) + +- [Maintainer Pull Request Template](?expand=1&template=maintainer_pull_request_template.md) (to be used by maintainers) diff --git a/CHANGELOG.md b/CHANGELOG.md index d42bd17..9dae9a4 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,3 +1,25 @@ +# dbt_fivetran_utils v0.4.11 + +## Feature Updates +[PR #139](https://github.com/fivetran/dbt_fivetran_utils/pull/139) includes the following update: +- Added the new `fivetran_union_relations()` macro. + - This was necessary for adding `union_data()` support to Zendesk the package, which works with some tables that have reserved-keywords for names (ie timezone). + - We heavily leveraged `dbt_utils.union_relations()` but adjusted it to work with these problematically named source tables. +- Adjusted the `union_data()` macro: + - Now references `fivetran_union_relations()` instead of `dbt_utils.union_relations()`. + - Now includes a new optional argument, `connector_table_name_override`, which is the _actual_ name/identifier of the table as it appears in the connector schema. Currently, this is only used in our Netsuite2 data models, whose actual table names and defined `source` table names differ slightly due to our support of both the Netsuite1 and Netsuite2 endpoints. + - Includes a reference to a new `use_table_name_identifer_override` global variable. This variable was introduced to accommodate the use of `identifier` variables in packages where `connector_table_name_override` is used (again, just Netsuite for now). + +> Note: We already do have a macro called `union_relations()` in this package, but as we learned [recently](https://github.com/fivetran/dbt_fivetran_utils/releases/tag/v0.4.10), using the same exact name as a macro in `dbt_utils` is not a good idea. Thus, `fivetran_utils.union_relations()` will be deprecated in favor of `fivetran_utils.fivetran_union_relations()` in the future. Currently it is used in only one package (Marketo). + +[PR #145](https://github.com/fivetran/dbt_fivetran_utils/pull/145) includes the following update: +- Added the new `fivetran_lookback()` macro. This generates SQL to simplify adding lookback windows to incremental models. See [README](https://github.com/fivetran/dbt_fivetran_utils/blob/releases/v0.4.latest/README.md#fivetran_lookback-source). +- Added the new `fivetran_is_databricks_sql_warehouse()` macro. This determines if Databricks destination is a SQL Warehouse or All Purpose Cluster. See [README](https://github.com/fivetran/dbt_fivetran_utils/blob/releases/v0.4.latest/README.md#fivetran-is-databricks-sql-warehouse-source). + +## Under the Hood +[PR #139](https://github.com/fivetran/dbt_fivetran_utils/pull/139) includes the following update: +- Updated the pull request [templates](https://github.com/fivetran/dbt_fivetran_utils/tree/releases/v0.4.latest/.github) to align with our current format. + # dbt_fivetran_utils v0.4.10 ## Bug Fix diff --git a/README.md b/README.md index 4588ad3..fa10857 100644 --- a/README.md +++ b/README.md @@ -53,6 +53,7 @@ dispatch: - [📋 Contents](#-contents) - [Tests and helpers](#tests-and-helpers) - [collect\_freshness (source)](#collect_freshness-source) + - [fivetran\_is\_databricks\_sql\_warehouse (source)](#fivetran-is-databricks-sql-warehouse-source) - [seed\_data\_helper (source)](#seed_data_helper-source) - [snowflake\_seed\_data (source)](#snowflake_seed_data-source) - [Cross-database compatibility](#cross-database-compatibility) @@ -79,11 +80,14 @@ dispatch: - [dummy\_coalesce\_value (source)](#dummy_coalesce_value-source) - [fill\_pass\_through\_columns (source)](#fill_pass_through_columns-source) - [fill\_staging\_columns (source)](#fill_staging_columns-source) + - [fivetran\_lookback (source)](#fivetran_lookback-source) - [persist\_pass\_through\_columns (source)](#persist_pass_through_columns-source) - [remove\_prefix\_from\_columns (source)](#remove_prefix_from_columns-source) - [source\_relation (source)](#source_relation-source) - [union\_data (source)](#union_data-source) + - [Using identifiers in packages that have `union_data` and include the `connector_table_name_override` argument (currently only Netsuite2)](#using-identifiers-in-packages-that-have-union_data-and-include-the-connector_table_name_override-argument-currently-only-netsuite2) - [Union Data Defined Sources Configuration](#union-data-defined-sources-configuration) + - [fivetran\_union\_relations (source)](#fivetran_union_relations-source) - [union\_relations (source)](#union_relations-source) - [Variable Checks](#variable-checks) - [empty\_variable\_warning (source)](#empty_variable_warning-source) @@ -118,6 +122,17 @@ sources: **Args (sorta):** * `meta.is_enabled` (optional): The variable(s) you would like to reference to determine if dbt should include this table in freshness tests. +---- +### fivetran_is_databricks_sql_warehouse ([source](macros/fivetran_is_databricks_sql_warehouse.sql)) +For Databricks destinations, this macro returns `true` if the Databricks target indicates it is a SQL Warehouse. It will return return `false` if it is an All-Purpose Cluster. + +***Usage:** +```yml + fivetran_utils.fivetran_is_databricks_sql_warehouse() +``` +**Args:** +* none + ---- ### seed_data_helper ([source](macros/seed_data_helper.sql)) This macro is intended to be used when a source table column is a reserved keyword in a warehouse, and Circle CI is throwing a fit. @@ -464,6 +479,28 @@ from source * `source_columns` (required): Will call the [get_columns_in_relation](https://docs.getdbt.com/reference/dbt-jinja-functions/adapter/#get_columns_in_relation) macro as well requires a `ref()` or `source()` argument for the staging models within the `_tmp` directory. * `staging_columns` (required): Created as a result of running the [generate_columns_macro](https://github.com/fivetran/dbt_fivetran_utils#generate_columns_macro-source) for the respective table. +---- +### fivetran_lookback ([source](macros/fivetran_lookback.sql)) +This macro takes a date expression and uses `dbt.dateadd` and backdates it by the specified interval. This is intended for use in incremental blocks to look backwards and catch late arriving records. Any date aggregate can be used as an input, but the typical usage is with the max date of the present model. + +**Usage:** +```sql +{% if is_incremental() %} + where date_day >= + {{ fivetran_utils.fivetran_lookback( + from_date='max(date_day)', + interval=3, + datepart='day', + safety_date='2010-01-01') + }} +{% endif %} +``` +**Args:** +* `from_date` (required): String to run against the current model. The expression used should be expected to return a single result. +* `datepart` (required): The grain of the interval. +* `interval` (required): The number of units to look backwards. +* `safety_date` (optional): This date will be used in the rare case that the `from_date` expression returns a null value. This only needs to be specified if you want to change the default value from '2010-01-01'. + ---- ### persist_pass_through_columns ([source](macros/persist_pass_through_columns.sql)) This macro is used to persist pass through columns from the staging model to the **transform** package. This is particularly helpful when a `select *` is not feasible. @@ -518,6 +555,12 @@ To create dependencies between the unioned model and its *sources*, you **must d If the source table is not found in any of the provided schemas/databases, `union_data` will return a **completely** empty table (ie `limit 0`) with just one string column (`_dbt_source_relation`). A compiler warning message will be output, highlighting that the expected source table was not found and its respective staging model is empty. The compiler warning can be turned off by the end user by setting the `fivetran__remove_empty_table_warnings` variable to `True`. +```yml +# in root dbt_project.yml file +vars: + fivetran__remove_empty_table_warnings: true # false by default +``` + **Usage:** ```sql -- in model.sql file @@ -541,15 +584,27 @@ If the source table is not found in any of the provided schemas/databases, `unio * `default_variable`: The name of the variable that users should populate when they want to pass one specific relation to this model (mostly used for CI) * `union_schema_variable` (optional): The name of the union schema variable. By default the macro will look for `union_schemas`. * `union_database_variable` (optional): The name of the union database variable. By default the macro will look for `union_databases`. +* `connector_table_name_override` (optional): The _actual_ name/identifier of the table as it appears in the connector schema. Currently, this is only used in our Netsuite2 data models, whose actual table names and defined `source` table names differ slightly due to our support of both the Netsuite1 and Netsuite2 endpoints. + * If this argument is used, refer to the below subsection. + +#### Using identifiers in packages that have `union_data` and include the `connector_table_name_override` argument (currently only Netsuite2) +In our integration tests, we often have seed files that do not have the default names for their respective tables. They might have a `_data` suffix or something along those lines. In these cases, we make use of our identifier variables in the `integration_tests/dbt_project.yml` file. To ensure that the `union_data` macro picks these custom name configs up and does not use the `connector_table_name_override` (only a thing in Netsuite), set the following variable to `true` in the package's `integration_tests/dbt_project.yml` file: + +```yml +vars: + use_table_name_identifer_override: true +``` + +This can also be used by customers, but is **only relevant to Netsuite2** because the `source` table names != actual connector table names. #### Union Data Defined Sources Configuration +To create dependencies between the unioned model and its *sources*, you **must define** the source tables in a `.yml` file in your project and set the `has_defined_sources` variable (scoped to the source package in which the macro is being called) to `True` in your `dbt_project.yml` file. If you set `has_defined_sources` to true and do not define sources (at least adding the `name` of each table in the source), dbt will throw an error. + ```yml # in root dbt_project.yml file vars: - shopify_source: + shopify_source: # or whatever source package has_defined_sources: true - - fivetran__remove_empty_table_warnings: true # false by default ``` ```yml @@ -577,8 +632,36 @@ sources: - name: customer ... ``` + +---- +### fivetran_union_relations ([source](macros/union_relations.sql)) +Heavily adapted from [dbt_utils.union_relations()](https://github.com/dbt-labs/dbt-utils?tab=readme-ov-file#union_relations-source). The primary difference is that our version of the macro supports unioning relations that have reserved keywords as names (ex: table called `timezone`). + +This macro combines via a `union all` of an array of [Relations](https://docs.getdbt.com/reference/dbt-classes#relation), even when columns have differing orders in each Relation, and/or some columns are missing from some relations. Any columns exclusive to a subset of these relations will be filled with `null` where not present. A new column (`_dbt_source_relation`) is also added to indicate the source for each record. + +**Usage:** +```sql +{{ fivetran_utils.fivetran_union_relations( + relations=[ref('my_model'), source('my_source', 'my_table')], + exclude=["_loaded_at"], + aliases=['my_model_cte', 'my_source_table_cte'] +) }} +``` +**Args:** +* `relations` (required): An array of [Relations](https://docs.getdbt.com/docs/writing-code-in-dbt/class-reference/#relation). +* `aliases` (optional): An override of the relation identifier. This argument should be populated with the overwritten alias for each relation (ie if we are selecting from a CTE). If not populated `relations` will be the default. This argument is not included in the `dbt_utils` version of this macro. +* `exclude` (optional): A list of column names that should be excluded from the final query. +* `include` (optional): A list of column names that should be included in the final query. Note the `include` and `exclude` parameters are mutually exclusive. +* `column_override` (optional): A dictionary of explicit column type overrides, e.g. `{"some_field": "varchar(100)"}`.`` +* `source_column_name` (optional, `default="_dbt_source_relation"`): The name of the column that records the source of this row. Pass `None` to omit this column from the results. +* `where` (optional): Filter conditions to include in the `where` clause. + ---- ### union_relations ([source](macros/union_relations.sql)) +> TO BE DEPRECATED IN FAVOR OF `fivetran_union_relations`. +> +> Currenlty only used in [Marketo](https://github.com/fivetran/dbt_marketo/blob/main/models/intermediate/marketo__change_data_scd.sql#L47) transform package. + This macro unions together an array of [Relations](https://docs.getdbt.com/docs/writing-code-in-dbt/class-reference/#relation), even when columns have differing orders in each Relation, and/or some columns are missing from some relations. Any columns exclusive to a subset of these @@ -587,7 +670,7 @@ relations will be filled with `null` where not present. An new column **Usage:** ```sql -{{ dbt_utils.union_relations( +{{ fivetran_utils.union_relations( relations=[ref('my_model'), source('my_source', 'my_table')], exclude=["_loaded_at"] ) }} diff --git a/dbt_project.yml b/dbt_project.yml index d366876..d024d9c 100644 --- a/dbt_project.yml +++ b/dbt_project.yml @@ -1,4 +1,4 @@ name: 'fivetran_utils' -version: '0.4.10' +version: '0.4.11' config-version: 2 require-dbt-version: [">=1.3.0", "<2.0.0"] diff --git a/integration_tests/dbt_project.yml b/integration_tests/dbt_project.yml index 36507f1..70461e8 100644 --- a/integration_tests/dbt_project.yml +++ b/integration_tests/dbt_project.yml @@ -1,5 +1,5 @@ name: 'fivetran_utils_integration_tests' -version: '0.4.10' +version: '0.4.11' config-version: 2 profile: 'integration_tests' diff --git a/integration_tests/packages.yml b/integration_tests/packages.yml index 1f95659..5403ade 100644 --- a/integration_tests/packages.yml +++ b/integration_tests/packages.yml @@ -2,8 +2,10 @@ packages: - package: fivetran/ad_reporting version: [">=1.0.0", "<2.0.0"] - - package: fivetran/shopify_holistic_reporting - version: [">=0.1.0", "<1.0.0"] + # - package: fivetran/shopify_holistic_reporting + # version: [">=0.1.0", "<1.0.0"] + - git: https://github.com/fivetran/dbt_shopify_holistic_reporting.git + revision: feature/bump-packages - package: fivetran/social_media_reporting version: [">=0.1.0", "<1.0.0"] diff --git a/macros/fivetran_is_databricks_sql_warehouse.sql b/macros/fivetran_is_databricks_sql_warehouse.sql new file mode 100644 index 0000000..052b4f0 --- /dev/null +++ b/macros/fivetran_is_databricks_sql_warehouse.sql @@ -0,0 +1,15 @@ +{% macro fivetran_is_databricks_sql_warehouse() %} + {% if target.type in ('databricks') %} + {% set re = modules.re %} + {% set path_match = target.http_path %} + {% set regex_pattern = "sql/.+/warehouses/" %} + {% set match_result = re.search(regex_pattern, path_match) %} + {% if match_result %} + {{ return(True) }} + {% else %} + {{ return(False) }} + {% endif %} + {% else %} + {{ return(False) }} + {% endif %} +{% endmacro %} \ No newline at end of file diff --git a/macros/fivetran_lookback.sql b/macros/fivetran_lookback.sql new file mode 100644 index 0000000..b42f237 --- /dev/null +++ b/macros/fivetran_lookback.sql @@ -0,0 +1,18 @@ +{% macro fivetran_lookback(from_date, datepart, interval, safety_date='2010-01-01') %} + +{{ adapter.dispatch('fivetran_lookback', 'fivetran_utils') (from_date, datepart, interval, safety_date='2010-01-01') }} + +{%- endmacro %} + +{% macro default__fivetran_lookback(from_date, datepart, interval, safety_date='2010-01-01') %} + + {% set sql_statement %} + select coalesce({{ from_date }}, {{ "'" ~ safety_date ~ "'" }}) + from {{ this }} + {%- endset -%} + + {%- set result = dbt_utils.get_single_value(sql_statement) %} + + {{ dbt.dateadd(datepart=datepart, interval=-interval, from_date_or_timestamp="cast('" ~ result ~ "' as date)") }} + +{% endmacro %} \ No newline at end of file diff --git a/macros/fivetran_union_relations.sql b/macros/fivetran_union_relations.sql new file mode 100644 index 0000000..75c77c1 --- /dev/null +++ b/macros/fivetran_union_relations.sql @@ -0,0 +1,128 @@ +{%- macro fivetran_union_relations(relations, aliases=none, column_override=none, include=[], exclude=[], source_column_name='_dbt_source_relation', where=none) -%} + {{ return(adapter.dispatch('fivetran_union_relations', 'fivetran_utils')(relations, aliases, column_override, include, exclude, source_column_name, where)) }} +{% endmacro %} + +{%- macro default__fivetran_union_relations(relations, aliases=none, column_override=none, include=[], exclude=[], source_column_name='_dbt_source_relation', where=none) -%} + + {%- if exclude and include -%} + {{ exceptions.raise_compiler_error("Both an exclude and include list were provided to the `union` macro. Only one is allowed") }} + {%- endif -%} + + {#-- Prevent querying of db in parsing mode. This works because this macro does not create any new refs. -#} + {%- if not execute %} + {{ return('') }} + {% endif -%} + + {%- set column_override = column_override if column_override is not none else {} -%} + + {%- set relation_columns = {} -%} + {%- set column_superset = {} -%} + {%- set all_excludes = [] -%} + {%- set all_includes = [] -%} + + {%- if exclude -%} + {%- for exc in exclude -%} + {%- do all_excludes.append(exc | lower) -%} + {%- endfor -%} + {%- endif -%} + + {%- if include -%} + {%- for inc in include -%} + {%- do all_includes.append(inc | lower) -%} + {%- endfor -%} + {%- endif -%} + + {%- for relation in relations -%} + + {%- do relation_columns.update({relation: []}) -%} + + {%- do dbt_utils._is_relation(relation, 'fivetran_union_relations') -%} + {%- do dbt_utils._is_ephemeral(relation, 'fivetran_union_relations') -%} + {%- set cols = adapter.get_columns_in_relation(relation) -%} + {%- for col in cols -%} + + {#- If an exclude list was provided and the column is in the list, do nothing -#} + {%- if exclude and col.column | lower in all_excludes -%} + + {#- If an include list was provided and the column is not in the list, do nothing -#} + {%- elif include and col.column | lower not in all_includes -%} + + {#- Otherwise add the column to the column superset -#} + {%- else -%} + + {#- update the list of columns in this relation -#} + {%- do relation_columns[relation].append(col.column) -%} + + {%- if col.column in column_superset -%} + + {%- set stored = column_superset[col.column] -%} + {%- if col.is_string() and stored.is_string() and col.string_size() > stored.string_size() -%} + + {%- do column_superset.update({col.column: col}) -%} + + {%- endif %} + + {%- else -%} + + {%- do column_superset.update({col.column: col}) -%} + + {%- endif -%} + + {%- endif -%} + + {%- endfor -%} + {%- endfor -%} + + {%- set ordered_column_names = column_superset.keys() -%} + {%- set dbt_command = flags.WHICH -%} + + + {% if dbt_command in ['run', 'build'] %} + {% if (include | length > 0 or exclude | length > 0) and not column_superset.keys() %} + {%- set relations_string -%} + {%- for relation in relations -%} + {{ relation.name }} + {%- if not loop.last %}, {% endif -%} + {%- endfor -%} + {%- endset -%} + + {%- set error_message -%} + There were no columns found to union for relations {{ relations_string }} + {%- endset -%} + + {{ exceptions.raise_compiler_error(error_message) }} + {%- endif -%} + {%- endif -%} + + {%- for relation in relations %} + + ( + select + + {%- if source_column_name is not none %} + cast({{ dbt.string_literal(relation) }} as {{ dbt.type_string() }}) as {{ source_column_name }}, + {%- endif %} + + {% for col_name in ordered_column_names -%} + + {%- set col = column_superset[col_name] %} + {%- set col_type = column_override.get(col.column, col.data_type) %} + {%- set col_name = adapter.quote(col_name) if col_name in relation_columns[relation] else 'null' %} + cast({{ col_name }} as {{ col_type }}) as {{ col.quoted }} {% if not loop.last %},{% endif -%} + + {%- endfor %} + + from {{ aliases[loop.index0] if aliases else relation }} as unioned_relation_{{ loop.index }} + + {% if where -%} + where {{ where }} + {%- endif %} + ) + + {% if not loop.last -%} + union all + {% endif -%} + + {%- endfor -%} + +{%- endmacro -%} \ No newline at end of file diff --git a/macros/union_data.sql b/macros/union_data.sql index a67fadb..c450a80 100644 --- a/macros/union_data.sql +++ b/macros/union_data.sql @@ -1,4 +1,4 @@ -{%- macro union_data(table_identifier, database_variable, schema_variable, default_database, default_schema, default_variable, union_schema_variable='union_schemas', union_database_variable='union_databases') -%} +{%- macro union_data(table_identifier, database_variable, schema_variable, default_database, default_schema, default_variable, union_schema_variable='union_schemas', union_database_variable='union_databases', connector_table_name_override=None) -%} {{ adapter.dispatch('union_data', 'fivetran_utils') ( table_identifier, @@ -8,7 +8,8 @@ default_schema, default_variable, union_schema_variable, - union_database_variable + union_database_variable, + connector_table_name_override ) }} {%- endmacro -%} @@ -21,7 +22,8 @@ default_schema, default_variable, union_schema_variable, - union_database_variable + union_database_variable, + connector_table_name_override ) -%} {%- if var(union_schema_variable, none) -%} @@ -39,7 +41,7 @@ {%- set relation=adapter.get_relation( database=source(schema, table_identifier).database if var('has_defined_sources', false) else var(database_variable, default_database), schema=source(schema, table_identifier).schema if var('has_defined_sources', false) else schema, - identifier=source(schema, table_identifier).identifier if var('has_defined_sources', false) else table_identifier + identifier=connector_table_name_override if connector_table_name_override else (source(schema, table_identifier).identifier if var('has_defined_sources', false) else table_identifier) ) -%} {%- set relation_exists=relation is not none -%} @@ -51,7 +53,7 @@ {%- endfor -%} {%- if relations != [] -%} - {{ dbt_utils.union_relations(relations) }} + {{ fivetran_utils.fivetran_union_relations(relations) }} {%- else -%} {% if execute and not var('fivetran__remove_empty_table_warnings', false) -%} {{ exceptions.warn("\n\nPlease be aware: The " ~ table_identifier|upper ~ " table was not found in your " ~ default_schema|upper ~ " schema(s). The Fivetran dbt package will create a completely empty " ~ table_identifier|upper ~ " staging model as to not break downstream transformations. To turn off these warnings, set the `fivetran__remove_empty_table_warnings` variable to TRUE (see https://github.com/fivetran/dbt_fivetran_utils/tree/releases/v0.4.latest#union_data-source for details).\n") }} @@ -69,7 +71,7 @@ {%- set relation=adapter.get_relation( database=source(schema, table_identifier).database if var('has_defined_sources', false) else database, schema=source(schema, table_identifier).schema if var('has_defined_sources', false) else var(schema_variable, default_schema), - identifier=source(schema, table_identifier).identifier if var('has_defined_sources', false) else table_identifier + identifier=connector_table_name_override if connector_table_name_override else (source(schema, table_identifier).identifier if var('has_defined_sources', false) else table_identifier) ) -%} {%- set relation_exists=relation is not none -%} @@ -81,7 +83,7 @@ {%- endfor -%} {%- if relations != [] -%} - {{ dbt_utils.union_relations(relations) }} + {{ fivetran_utils.fivetran_union_relations(relations) }} {%- else -%} {% if execute and not var('fivetran__remove_empty_table_warnings', false) -%} {{ exceptions.warn("\n\nPlease be aware: The " ~ table_identifier|upper ~ " table was not found in your " ~ default_schema|upper ~ " schema(s). The Fivetran dbt package will create a completely empty " ~ table_identifier|upper ~ " staging model as to not break downstream transformations. To turn off these warnings, set the `fivetran__remove_empty_table_warnings` variable to TRUE (see https://github.com/fivetran/dbt_fivetran_utils/tree/releases/v0.4.latest#union_data-source for details).\n") }} @@ -102,7 +104,7 @@ {%- set relation.value=adapter.get_relation( database=source(corrected_schema_name[1], table_identifier).database, schema=source(corrected_schema_name[1], table_identifier).schema, - identifier=var(identifier_var, table_identifier) + identifier=connector_table_name_override if connector_table_name_override else var(identifier_var, table_identifier) ) -%} {% endif %} {% endfor %} @@ -116,14 +118,15 @@ {%- set relation.value=adapter.get_relation( database=source(default_schema, table_identifier).database, schema=source(default_schema, table_identifier).schema, - identifier=var(identifier_var, table_identifier) + identifier=connector_table_name_override if (connector_table_name_override and not var('use_table_name_identifer_override', false)) else var(identifier_var, connector_table_name_override if connector_table_name_override else table_identifier) ) -%} {% endif %} {%- set table_exists=relation.value is not none -%} {%- if table_exists -%} - select * - from {{ relation.value }} + select + {{ dbt_utils.star(from=relation.value) }} + from {{ relation.value }} as {{ table_identifier }}_table {%- else -%} {% if execute and not var('fivetran__remove_empty_table_warnings', false) -%} {{ exceptions.warn("\n\nPlease be aware: The " ~ table_identifier|upper ~ " table was not found in your " ~ default_schema|upper ~ " schema(s). The Fivetran dbt package will create a completely empty " ~ table_identifier|upper ~ " staging model as to not break downstream transformations. To turn off these warnings, set the `fivetran__remove_empty_table_warnings` variable to TRUE (see https://github.com/fivetran/dbt_fivetran_utils/tree/releases/v0.4.latest#union_data-source for details).\n") }} diff --git a/macros/union_relations.sql b/macros/union_relations.sql index 56122e6..7076c61 100644 --- a/macros/union_relations.sql +++ b/macros/union_relations.sql @@ -1,3 +1,5 @@ +-- to be deprecated in favor of fivetran_union_relations + {%- macro union_relations(relations, aliases=none, column_override=none, include=[], exclude=[], source_column_name=none) -%} {%- if exclude and include -%}