Feature/ja updates

.github/workflows/prod-web-deploy.yml

@@ -18,7 +18,7 @@ jobs:
 
       - uses: actions/setup-node@v4
         with:
-          node-version: 18
+          node-version: 20
           cache: yarn
           cache-dependency-path: website/yarn.lock
 

.github/workflows/preview-web-deploy.yml → .github/workflows/test-web-deploy.yml

@@ -18,7 +18,7 @@ jobs:
 
       - uses: actions/setup-node@v4
         with:
-          node-version: 18
+          node-version: 20
           cache: yarn
           cache-dependency-path: website/yarn.lock
 

.gitignore

@@ -12,6 +12,9 @@ testcreds/
 *.log
 venv/
 .venv/
+nohup.out
+
+/.ruff_cache
 
 # Byte-compiled / optimized / DLL files
 __pycache__/
@@ -87,3 +90,4 @@ docs/_build/
 
 venv/
 .DS_Store
+myenv/

CHANGELOG.md

@@ -1,9 +1,14 @@
 # Changelog
 
+## 1.8.5 (2025-06-30)
+
+- Added support for resource scoped variables
+- Added developer credits in `info`
+
 ## 1.8.3 (2025-02-08)
 
 - Added walkthrough for databricks bootstrap on aws.
-- Bugfix for expport variables on dry run.
+- Bugfix for export variables on dry run.
 
 ## 1.8.2 (2025-01-16)
 

MANIFEST.in

@@ -1,3 +1,4 @@
 # MANIFEST.in
 include README.rst
 recursive-include stackql_deploy/templates *.template
+include stackql_deploy/inc/contributors.csv

README.md

@@ -265,14 +265,16 @@ To distribute **stackql-deploy** on PyPI, you'll need to ensure that you have al
 
 First, ensure you have the latest versions of `setuptools` and `wheel` installed:
 
-```
+```bash
+python3 -m venv venv
+source venv/bin/activate
 # pip install --upgrade setuptools wheel
 pip install --upgrade build
 ```
 
 Then, navigate to your project root directory and build the distribution files:
 
-```
+```bash
 rm dist/stackql_deploy*
 python3 -m build
 # or

examples/databricks/all-purpose-cluster/README.md → examples/databricks/classic/README.md

@@ -26,7 +26,7 @@ Now, is is convenient to use environment variables for context.  Note that for o
 ```bash
 #!/usr/bin/env bash
 
-export ASSETS_AWS_REGION='us-east-1' # or wherever you want
+export AWS_REGION='us-east-1' # or wherever you want
 export AWS_ACCOUNT_ID='<your aws account ID>'
 export DATABRICKS_ACCOUNT_ID='<your databricks account ID>'
 export DATABRICKS_AWS_ACCOUNT_ID='<your databricks aws account ID>'
@@ -46,28 +46,20 @@ export AWS_ACCESS_KEY_ID='<your aws access key id per aws cli>'
 Now, let us do some sanity checks and housekeeping with `stackql`.  This is purely optional.  From the root of this repository: 
 
 ```
-
 source examples/databricks/all-purpose-cluster/convenience.sh
-
 stackql shell
-
 ```
 
 This will start a `stackql` interactive shell.  Here are some commands you can run (I will not place output here, that will be shared in a corresponding video):
 
 
 ```sql
-
 registry pull databricks_account v24.12.00279;
-
 registry pull databricks_workspace v24.12.00279;
 
 -- This will fail if accounts, subscription, or credentials are in error.
 select account_id FROM databricks_account.provisioning.credentials WHERE account_id = '<your databricks account id>';
-
-
 select account_id, workspace_name, workspace_id, workspace_status from databricks_account.provisioning.workspaces where account_id = '<your databricks account id>';
-
 ```
 
 For extra credit, you can (asynchronously) delete the unnecessary workspace with `delete from databricks_account.provisioning.workspaces where account_id = '<your databricks account id>' and workspace_id = '<workspace id>';`, where you obtain the workspace id from the above query.  I have noted that due to some reponse caching it takes a while to disappear from select queries (much longer than disappearance from the web page), and you may want to bounce the `stackql` session to hurry things along.  This is not happening on the `stackql` side, but session bouncing forces a token refresh which can help cache busting. 
@@ -77,20 +69,20 @@ For extra credit, you can (asynchronously) delete the unnecessary workspace with
 Time to get down to business.  From the root of this repository:
 
 ```bash
-
+python3 -m venv myenv
 source examples/databricks/all-purpose-cluster/convenience.sh
-
-source ./.venv/bin/activate
-
-
+source venv/bin/activate
+pip install stackql-deploy
 ```
 
+> alternatively set the `AWS_REGION`, `AWS_ACCOUNT_ID`, `DATABRICKS_ACCOUNT_ID`, `DATABRICKS_AWS_ACCOUNT_ID` along with provider credentials `AWS_ACCESS_KEY_ID`, `AWS_SECRET_ACCESS_KEY`, `DATABRICKS_CLIENT_ID`, `DATABRICKS_CLIENT_SECRET`
+
 Then, do a dry run (good for catching **some** environmental issues):
 
 ```bash
 stackql-deploy build \
 examples/databricks/all-purpose-cluster dev \
--e AWS_REGION=${ASSETS_AWS_REGION} \
+-e AWS_REGION=${AWS_REGION} \
 -e AWS_ACCOUNT_ID=${AWS_ACCOUNT_ID} \
 -e DATABRICKS_ACCOUNT_ID=${DATABRICKS_ACCOUNT_ID} \
 -e DATABRICKS_AWS_ACCOUNT_ID=${DATABRICKS_AWS_ACCOUNT_ID} \
@@ -105,7 +97,7 @@ Now, let use do it for real:
 ```bash
 stackql-deploy build \
 examples/databricks/all-purpose-cluster dev \
--e AWS_REGION=${ASSETS_AWS_REGION} \
+-e AWS_REGION=${AWS_REGION} \
 -e AWS_ACCOUNT_ID=${AWS_ACCOUNT_ID} \
 -e DATABRICKS_ACCOUNT_ID=${DATABRICKS_ACCOUNT_ID} \
 -e DATABRICKS_AWS_ACCOUNT_ID=${DATABRICKS_AWS_ACCOUNT_ID} \
@@ -128,7 +120,7 @@ We can also use `stackql-deploy` to assess if our infra is shipshape:
 ```bash
 stackql-deploy test \
 examples/databricks/all-purpose-cluster dev \
--e AWS_REGION=${ASSETS_AWS_REGION} \
+-e AWS_REGION=${AWS_REGION} \
 -e AWS_ACCOUNT_ID=${AWS_ACCOUNT_ID} \
 -e DATABRICKS_ACCOUNT_ID=${DATABRICKS_ACCOUNT_ID} \
 -e DATABRICKS_AWS_ACCOUNT_ID=${DATABRICKS_AWS_ACCOUNT_ID} \
@@ -151,7 +143,7 @@ Now, let us teardown our `stackql-deploy` managed infra:
 ```bash
 stackql-deploy teardown \
 examples/databricks/all-purpose-cluster dev \
--e AWS_REGION=${ASSETS_AWS_REGION} \
+-e AWS_REGION=${AWS_REGION} \
 -e AWS_ACCOUNT_ID=${AWS_ACCOUNT_ID} \
 -e DATABRICKS_ACCOUNT_ID=${DATABRICKS_ACCOUNT_ID} \
 -e DATABRICKS_AWS_ACCOUNT_ID=${DATABRICKS_AWS_ACCOUNT_ID} \

examples/databricks/all-purpose-cluster/convenience.sh → examples/databricks/classic/convenience.sh

@@ -10,9 +10,9 @@ then
     source "${REPOSITORY_ROOT}/examples/databricks/all-purpose-cluster/sec/env.sh"
 fi
 
-if [ "${ASSETS_AWS_REGION}" = "" ];
+if [ "${AWS_REGION}" = "" ];
 then
-    ASSETS_AWS_REGION='us-east-1'
+    AWS_REGION='us-east-1'
 fi
 
 if [ "${AWS_ACCOUNT_ID}" = "" ];
@@ -57,7 +57,7 @@ then
     exit 1
 fi
 
-export ASSETS_AWS_REGION
+export AWS_REGION
 export AWS_ACCOUNT_ID
 export DATABRICKS_ACCOUNT_ID
 export DATABRICKS_AWS_ACCOUNT_ID

examples/databricks/all-purpose-cluster/resources/aws_iam_cross_account_role.iql → examples/databricks/classic/resources/aws/iam/iam_role.iql

@@ -49,7 +49,8 @@ AND path = '{{ path }}';
 
 /*+ exports, retries=3, retry_delay=5 */
 SELECT   
-arn as aws_iam_cross_account_role_arn
+'{{ role_name }}' as aws_iam_role_name,
+arn as aws_iam_role_arn
 FROM aws.iam.roles
 WHERE data__Identifier = '{{ role_name }}'
 

examples/databricks/classic/resources/aws/vpc/elastic_ip.iql

@@ -0,0 +1,56 @@
+/*+ exists */
+SELECT COUNT(*) as count FROM
+(
+SELECT allocation_id,
+json_group_object(tag_key, tag_value) as tags
+FROM aws.ec2.eip_tags
+WHERE region = '{{ region }}'
+GROUP BY allocation_id
+HAVING json_extract(tags, '$.Provisioner') = 'stackql'
+AND json_extract(tags, '$.StackName') = '{{ stack_name }}'
+AND json_extract(tags, '$.StackEnv') = '{{ stack_env }}'
+) t
+
+/*+ create */
+INSERT INTO aws.ec2.eips (
+ NetworkBorderGroup,
+ Tags,
+ ClientToken,
+ region
+)
+SELECT 
+'{{ region }}',
+'{{ tags }}',
+'{{ idempotency_token }}',
+'{{ region }}'
+
+/*+ statecheck, retries=3, retry_delay=5 */
+SELECT COUNT(*) as count FROM
+(
+SELECT allocation_id,
+json_group_object(tag_key, tag_value) as tags
+FROM aws.ec2.eip_tags
+WHERE region = '{{ region }}'
+GROUP BY allocation_id
+HAVING json_extract(tags, '$.Provisioner') = 'stackql'
+AND json_extract(tags, '$.StackName') = '{{ stack_name }}'
+AND json_extract(tags, '$.StackEnv') = '{{ stack_env }}'
+) t
+
+/*+ exports, retries=3, retry_delay=5 */
+SELECT allocation_id as eip_allocation_id, public_ip as eip_public_id FROM
+(
+SELECT allocation_id, public_ip,
+json_group_object(tag_key, tag_value) as tags
+FROM aws.ec2.eip_tags
+WHERE region = '{{ region }}'
+GROUP BY allocation_id
+HAVING json_extract(tags, '$.Provisioner') = 'stackql'
+AND json_extract(tags, '$.StackName') = '{{ stack_name }}'
+AND json_extract(tags, '$.StackEnv') = '{{ stack_env }}'
+) t
+
+/*+ delete */
+DELETE FROM aws.ec2.eips
+WHERE data__Identifier = '{{ eip_public_id }}|{{ eip_allocation_id}}'
+AND region = '{{ region }}'

examples/databricks/all-purpose-cluster/resources/aws_get_main_route_table_id.iql → examples/databricks/classic/resources/aws/vpc/get_main_route_table_id.iql

@@ -1,6 +1,6 @@
 /*+ exports, retries=3, retry_delay=5 */
 SELECT
-route_table_id
+route_table_id as main_route_table_id
 FROM aws.ec2.route_tables
 WHERE region = '{{ region }}'
 AND vpc_id = '{{ vpc_id }}';

examples/databricks/classic/resources/aws/vpc/nat_gateway.iql

@@ -0,0 +1,53 @@
+/*+ exists */
+SELECT COUNT(*) as count FROM
+(
+SELECT nat_gateway_id,
+json_group_object(tag_key, tag_value) as tags
+FROM aws.ec2.nat_gateway_tags
+WHERE region = '{{ region }}'
+GROUP BY allocation_id
+HAVING json_extract(tags, '$.Provisioner') = 'stackql'
+AND json_extract(tags, '$.StackName') = '{{ stack_name }}'
+AND json_extract(tags, '$.StackEnv') = '{{ stack_env }}'
+) t
+
+/*+ create */
+INSERT INTO aws.ec2.nat_gateways (
+ AllocationId,
+ SubnetId,
+ Tags,
+ region
+)
+SELECT 
+ '{{ eip_allocation_id }}',
+ '{{ nat_subnet_id }}',
+ '{{ tags }}',
+ '{{ region }}';
+
+/*+ statecheck, retries=3, retry_delay=5 */
+SELECT COUNT(*) as count FROM
+(
+SELECT nat_gateway_id,
+json_group_object(tag_key, tag_value) as tags
+FROM aws.ec2.nat_gateway_tags
+WHERE region = '{{ region }}'
+GROUP BY allocation_id
+HAVING json_extract(tags, '$.Provisioner') = 'stackql'
+AND json_extract(tags, '$.StackName') = '{{ stack_name }}'
+AND json_extract(tags, '$.StackEnv') = '{{ stack_env }}'
+) t
+
+/*+ exports, retries=3, retry_delay=5 */
+SELECT nat_gateway_id,
+json_group_object(tag_key, tag_value) as tags
+FROM aws.ec2.nat_gateway_tags
+WHERE region = '{{ region }}'
+GROUP BY allocation_id
+HAVING json_extract(tags, '$.Provisioner') = 'stackql'
+AND json_extract(tags, '$.StackName') = '{{ stack_name }}'
+AND json_extract(tags, '$.StackEnv') = '{{ stack_env }}'
+
+/*+ delete */
+DELETE FROM aws.ec2.nat_gateways
+WHERE data__Identifier = '{{ nat_gateway_id }}'
+AND region = '{{ region }}';

examples/databricks/classic/resources/aws/vpc/nat_inet_route.iql

@@ -0,0 +1,41 @@
+/*+ exists */
+SELECT COUNT(*) as count FROM
+(
+SELECT data__Identifier
+FROM aws.ec2.routes
+WHERE region = '{{ region }}'
+AND data__Identifier = '{{ route_table_id }}|0.0.0.0/0'
+) t
+
+/*+ create */
+INSERT INTO aws.ec2.routes (
+ DestinationCidrBlock,
+ NatGatewayId,
+ RouteTableId,
+ region
+)
+SELECT 
+ '0.0.0.0/0',
+ '{{ nat_gateway_id }}',
+ '{{ route_table_id }}',
+ '{{ region }}';
+
+/*+ statecheck, retries=5, retry_delay=5 */
+SELECT COUNT(*) as count FROM
+(
+SELECT data__Identifier
+FROM aws.ec2.routes
+WHERE region = '{{ region }}'
+AND data__Identifier = '{{ route_table_id }}|0.0.0.0/0'
+) t
+
+/*+ exports, retries=3, retry_delay=5 */
+SELECT data__Identifier as nat_inet_route_indentifer
+FROM aws.ec2.routes
+WHERE region = '{{ region }}'
+AND data__Identifier = '{{ route_table_id }}|0.0.0.0/0';
+
+/*+ delete */
+DELETE FROM aws.ec2.routes
+WHERE data__Identifier = '{{ inet_route_indentifer }}'
+AND region = '{{ region }}';