[New Feature]: Store DAAC responses somewhere before processing as a backup

[wphyojpl]

Checked for duplicates

Have you checked for duplicate issue tickets?

Yes - I've already checked

Alternatives considered

Have you considered alternative solutions to your feature request?

Yes - and alternatives don't suffice

Related problems

Some granules in UDS DB is showing there is "no" response from them.
There is uncertainty whether SNS / SQS dropped the message.
As a backkup, in SNS, store the raw message somewhere.
S3 would be a good location with lifecycle rule to drop it after some time. (1 year).

Describe the feature request

See Above

** Solution **

• SNS -> Firehose -> S3

# S3 bucket to store archived SNS messages
resource "aws_s3_bucket" "sns_archive" {
  bucket = "${var.prefix}-sns-archive"
  force_destroy = true
  tags = var.tags
}

# IAM role Firehose assumes to write to S3
resource "aws_iam_role" "firehose_role" {
  name = "${var.prefix}-firehose-role"

  assume_role_policy = jsonencode({
    Version = "2012-10-17"
    Statement = [
      {
        Effect = "Allow"
        Principal = {
          Service = "firehose.amazonaws.com"
        }
        Action = "sts:AssumeRole"
      }
    ]
  })
}

# IAM policy for Firehose to put objects to S3
resource "aws_iam_role_policy" "firehose_policy" {
  name = "${var.prefix}-firehose-policy"
  role = aws_iam_role.firehose_role.id

  policy = jsonencode({
    Version = "2012-10-17"
    Statement = [
      {
        Effect = "Allow"
        Action = [
          "s3:AbortMultipartUpload",
          "s3:GetBucketLocation",
          "s3:GetObject",
          "s3:ListBucket",
          "s3:ListBucketMultipartUploads",
          "s3:PutObject"
        ]
        Resource = [
          aws_s3_bucket.sns_archive.arn,
          "${aws_s3_bucket.sns_archive.arn}/*"
        ]
      }
    ]
  })
}

# Firehose delivery stream
resource "aws_kinesis_firehose_delivery_stream" "sns_firehose" {
  name        = "${var.prefix}-sns-firehose"
  destination = "s3"

  s3_configuration {
    role_arn   = aws_iam_role.firehose_role.arn
    bucket_arn = aws_s3_bucket.sns_archive.arn

    # Buffering controls: adjust for latency vs cost
    buffering_size     = 5   # MB
    buffering_interval = 60  # seconds
  }
}

# SNS subscription to Firehose
resource "aws_sns_topic_subscription" "sns_to_firehose" {
  topic_arn = var.sns_topic_arn
  protocol  = "firehose"
  endpoint  = aws_kinesis_firehose_delivery_stream.sns_firehose.arn
}