DILT-19: Quartz CLI Maintenance - Cleanup Improvements & Retry Logic

[padili-metrostar]

Summary

This PR contains significant improvements to the quartzctl CLI for better cluster cleanup reliability and error handling.

Changes

AWS Resource Cleanup (New)

• Implement proactive check for blocking AWS resources before Terraform destroy
• Add EC2 instance termination functionality
• Implement fallback AWS resource cleanup for orphaned resources
• Clean up Load Balancers, ENIs, Security Groups blocking VPC deletion

Kubernetes Cleanup Improvements

• Implement Kubernetes resource cleanup before EC2 termination
• Delete ALL validating and mutating webhooks (aggressive cleanup)
• Patch stuck namespaces with finalizer removal
• Add VirtualService listing and print functionality

Retry Logic Enhancements

• Enhanced logging and retry mechanisms for health checks
• Add DaemonSet readiness checks
• Cleanup for stuck terminating pods
• Recognize Helm-specific errors for smarter retries

Testing

• Add unit tests for isRetryableDestroyError() function (12 test cases)
• Add unit tests for isHelmReleaseError() function (7 test cases)
• All 19 tests passing

Key Functions Added

func isRetryableDestroyError(errStr string) bool { ... }
func isHelmReleaseError(errStr string) bool { ... }
func cleanupKubernetesBlockers() error { ... }
func ForceAWSCleanup() error { ... }

Testing

• ✅ Unit tests pass: go test -v -run "TestIs" ./internal/cmd/...
• ✅ Cleanup completed successfully on test cluster (14m 42s, 185 resources destroyed)
• ✅ No orphaned AWS resources after cleanup

Related PRs

• MetroStar/quartz#71 - DILT-19-Quartz-Maintenance
• MetroStar/quartz-cicd - DILT-19-Quartz-Maintenance

[codecov]

Codecov Report

❌ Patch coverage is 70.84871% with 79 lines in your changes missing coverage. Please review.
✅ Project coverage is 69.39%. Comparing base (e5a799b) to head (f2a9d9f).

Files with missing lines	Patch %	Lines
internal/cmd/install.go	68.68%	29 Missing and 2 partials ⚠️
internal/provider/kubernetes.go	71.73%	20 Missing and 6 partials ⚠️
internal/cmd/internal.go	47.61%	8 Missing and 3 partials ⚠️
internal/stages/daemonset_check.go	87.09%	2 Missing and 2 partials ⚠️
internal/provider/aws.go	57.14%	1 Missing and 2 partials ⚠️
internal/stages/check.go	75.00%	2 Missing and 1 partial ⚠️
internal/config/schema/environment.go	0.00%	1 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main      #62      +/-   ##
==========================================
+ Coverage   67.44%   69.39%   +1.95%     
==========================================
  Files          64       64              
  Lines        4457     3944     -513     
==========================================
- Hits         3006     2737     -269     
+ Misses       1228      967     -261     
- Partials      223      240      +17     

Files with missing lines	Coverage Δ
internal/cmd/util.go	82.71% <100.00%> (+6.28%)	⬆️
internal/terraform/operations.go	78.70% <100.00%> (+3.15%)	⬆️
internal/config/schema/environment.go	0.00% <0.00%> (ø)
internal/provider/aws.go	83.47% <57.14%> (+1.12%)	⬆️
internal/stages/check.go	86.72% <75.00%> (-0.38%)	⬇️
internal/stages/daemonset_check.go	87.09% <87.09%> (ø)
internal/cmd/internal.go	62.22% <47.61%> (-6.75%)	⬇️
internal/provider/kubernetes.go	70.60% <71.73%> (+7.48%)	⬆️
internal/cmd/install.go	66.28% <68.68%> (+9.85%)	⬆️

... and 54 files with indirect coverage changes

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[marshyski]

Update devbox.json golang version.