Transfer Family StartDirectoryListing throwing ThrottlingException Errors

[CarterSheehan]

I have a remote SFTP server with hundreds of remote folders and 1 to 2 TB of data to sync over to S3 which is causing ThrottlingException errors when calling the Transfer Family StartDirectoryListing API Action. Is there a solution to this? What good is a cloud-native reference architecture if it can't be used to sync large folder and files counts???

[pscheri]

Thanks for raising this.
Maybe we can add a custom retry strategy to the Step Function step that it's failing. Would you be able to test it? I don't currently have a big enough SFTP server to do it.

You could modify this in the AWS Console by editing the Step Function, clicking on the step failing and error handling on the right. There you can modify the exponential backoff approach, and I'd also enable Jitter.

If you want to do it on code, modify after this line and add:

file-transfer-sync-solution/transfer_sync_service/transfer_sync_service_stack.py

Line 360 in 5ee79ee

task_remote_server_list_lambda = sfn_tasks.LambdaInvoke(

task_remote_server_list_lambda.add_retry(
            backoff_rate=4,
            interval=Duration.seconds(4),
            max_attempts=6,
            errors=[
                "Lambda.ClientExecutionTimeoutException",
                "Lambda.ServiceException",
                "Lambda.AWSLambdaException",
                "Lambda.SdkClientException",
                "Lambda.TooManyRequestsException"
            ],
            jitter_strategy=sfn.JitterType.FULL
        )

If this works, feel free to raise a PR, or let me know and I'll apply the change.