Skip to content

Conversation

@bplatz
Copy link
Contributor

@bplatz bplatz commented Oct 10, 2025

Summary

Adds streaming query API that emits individual results as they are produced, rather than collecting all results into a vector. Supports all query types (SELECT, SELECT ONE, CONSTRUCT) and formats (FQL, SPARQL).

Key Features

  • Memory efficient streaming: Results emitted individually for large result sets
  • Metadata tracking: Optional :meta tracking emits final metadata map after results
  • CONSTRUCT streaming: Emits ungrouped graph nodes without @graph wrapper
  • Full query support: ORDER BY, GROUP BY, and aggregates (collect before emitting)
  • Cache validation: Early validation rejects :cache option for streaming queries

API Functions

  • fluree.db.api/query-stream - Stream query against database
  • fluree.db.api/query-connection-stream - Stream query via connection

Metadata Format

When :meta true is enabled, final result is:

{:_fluree-meta {:fuel 123, :time "5ms", :policy {...}, :status 200}}

Test Coverage

10 tests, 39 assertions covering:

  • Basic streaming (SELECT, LIMIT, SELECT ONE)
  • Metadata tracking (FQL and SPARQL)
  • Connection-based queries
  • CONSTRUCT queries (FQL and SPARQL)

Implementation Notes

  • Cache validation moved to public API layer
  • Internal functions simplified (validation upstream)
  • CONSTRUCT uses cat transducer for streaming ungrouped nodes
  • Metadata calculated from tracker counters without forcing realization

@zonotope
Copy link
Contributor

@bplatz it looks like this pull request is based on the bm25 ns branch, but the base branch is set to main so there are a lot of unrelated changes here.

Could you update the base branch to make reviewing easier?

@bplatz bplatz changed the base branch from main to feature/bm25-ns December 8, 2025 14:15
@bplatz
Copy link
Contributor Author

bplatz commented Jan 9, 2026

Closed in favor of #1186

@bplatz bplatz closed this Jan 9, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants