Questions on distributed index building

[yingjianwu98]

@chenghao-guo
Thanks for building the distributed indexing in ray, which is really powerful.

I am still new into lance and I have one architecture related questions:

Looks like right now some of the task planning, such as fragment balancing is built in the lance-ray layer. Do you see any benefits to move the logic into lance-core?

Similar to file_compaction, I am thinking we could have the planning logic reside in lance-core and then just get the taskPlan in lance-ray. One benefit I see in the future is that the same task planning logic can be re-usable across other engines like spark, and flink, etc.

Happy to hear what you think, thanks!

[chenghao-guo]

Hi @yingjianwu98,
Thank you for your insightful suggestions! I completely agree that integrating fragment balancing at the lance-core layer would offer significant benefits, especially in terms of reusability across all engines.
When I initially designed the distributed index in Ray, my primary goal was to minimize changes to avoid disrupting the lance-core interface. Introducing new APIs in Lance-Core can be challenging for the community to review and test, so I opted for a more incremental approach. Specifically, I added the merge_index_metadata() API to handle operations in the header and called create_scarlar_index() in the worker.

However, one challenge I encountered was determining an efficient way to split fragments within create_scarlar_index(). Perhaps we need an additional API like distribute_fragments() to manage this in the header, followed by create_scarlar_index() in the worker and merge_index_metadata() in the header. Alternatively, if there's a way to seamlessly integrate all engines with just a single create_scarlar_index() API, that would be even better.

Overall, I find your approach more logical and fully support your idea. Feel free to proceed with submitting a PR to lance-core. Once merged, I believe it will greatly simplify the Lance-Ray and Daft projects, making them more efficient and maintainable.
Looking forward to your thoughts!

[yingjianwu98]

Thanks for sharing your thoughts, and this makes sense.
I will take a look and see if we can refactor the code into lance-core.