Recursive download takes 5 minutes logging the same current directory before download begins

[denniswalker]

Am I doing something wrong or suboptimal? This isn't a complicated folder/file director structure - maybe 50 files total.

When I begin the get_r method, I see this repeatedly for 5 minutes before any transfer begins.
[2025-02-20 12:38:33,033] INFO - Current Working Directory: [/tmp]

Here are my connection settings:

cnopts.hostkeys = None
cnopts.timeout = 20
cnopts.allow_agent = False
cnopts.user_known_hosts_file = None
cnopts.compress = True
cnopts.knownhosts = None

and my transfer options:

    self.preserve_mtime = True
    self.prefetch = True
    self.max_concurrent_prefetch_requests = 64
    self.tries = 3
    self.resume = True
    self.callback = self.log_progress

[byteskeptical]

Howdy Dennis,

It does sound like something isn't quite right, are there a lot of files in /tmp otherwise? Your value for max_concurrent_prefetch_requests seems a bit high unless your running this on a threadripper. I would try lowering it to something between 4-8 to see if your thread locking the system. I would also not run cnopts.compress = True unless you are absolutely sure your server supports it and has it enabled[0] (probably not). Seeing the actual code (at least the connection/transfer portion) would help a great deal along with a debug log.

[0] - https://lists.mindrot.org/pipermail/openssh-unix-dev/2003-November/019734.html

[denniswalker]

Thank you @byteskeptical. My implementation attempt can be found here. https://github.com/denniswalker/tunnelgraf/blob/feature/22-add-scp-capability/src/tunnelgraf/transfer.py - Recommendations (or more) welcome.

I just tried without compression and lowering the max prefetch requests to 8 and still have a 5-minute delay. One relevant context is that I'm testing this through a multiple-hop bastion configuration and the connection is slower. That said, if possible, I'm aiming for something close to scp speed. Running rsync only takes about 10-15 seconds before it starts. I'll spend a bit of time later today or tomorrow with ipdb.set_trace, stepping through the code to better understand what's going on. Will also add debug logging here.

[byteskeptical]

This may be worth trying again with the latest version if your still interested.