Skip to content

Conversation

@akshit-git24
Copy link
Contributor

Hi @sr-857

This PR addresses a critical performance bottleneck in the FlowOrchestrator where synchronous file I/O was blocking the event loop during every step of a transform scan. By moving the execution log state to memory and offloading disk writes to a background thread, we significantly reduce overhead and prevent UI/API freezes during complex workflows.

[The Problem] :->

1.) Blocking I/O: The previous implementation read and wrote the entire JSON log file synchronously for every single step update.
2.) O(N²) Complexity: As the log file grew, reading/writing it took exponentially longer, causing massive slowdowns for long-running scans.
3.) Event Loop Starvation: These blocking operations paused all other concurrent tasks in the application.

[The Solution] :->

1.) In-Memory State: The execution log is now maintained in self.execution_log_data, eliminating the need to read from disk before every update.
2.)Async Writes: File writing is now handled by asyncio.to_thread(self._write_log_to_disk), effectively unblocking the main event loop.

_update_execution_log and _finalize_execution_log are now async functions that manage the state efficiently.

1.)Performance: Drastically reduced execution time for multi-step transforms.
2.) Scalability: The orchestrator can now handle much larger workflows without degradation.
3.)Responsiveness: The application remains responsive even during heavy logging activity.

…aces blocking synchronous file writes with in-memory state and background disk updates
@sr-857 sr-857 added the SWOC26 label Jan 9, 2026
@sr-857 sr-857 merged commit 699164c into sr-857:main Jan 9, 2026
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants