Skip to content

Conversation

@progval
Copy link
Contributor

@progval progval commented Oct 9, 2025

No description provided.

@progval progval force-pushed the par_sort_graph_rayon branch 2 times, most recently from a496c08 to 5d323a5 Compare October 9, 2025 12:33
@progval progval force-pushed the par_sort_graph_rayon branch from 5d323a5 to 06d7517 Compare October 9, 2025 12:37
@vigna
Copy link
Owner

vigna commented Oct 9, 2025

Well, the whole idea was to avoid Rayon as the bridge is horribly slow and prone to contention (that why we wrote ParFoldMap—performance with the bridge was abysmal).

@vigna
Copy link
Owner

vigna commented Oct 9, 2025

But we can try some benchmarks.

@progval
Copy link
Contributor Author

progval commented Oct 9, 2025

But here we're bridging an iterator with only a small number of elements, and the current code spawns a thread for each of these elements.

@vigna
Copy link
Owner

vigna commented Oct 9, 2025

Yeah, that's the idea—we know the parallelism we want, so we do it. And no Rayon overhead.

@progval
Copy link
Contributor Author

progval commented Oct 9, 2025

But Rayon's overhead is negligible on an iterator with less than a thousand items when processing each item takes minutes.

@vigna
Copy link
Owner

vigna commented Nov 14, 2025

This will be worked out in the current "de-parameterization" of the parallel code.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants