From c0895d4710215a7915ff2a5cdea625e10b31a982 Mon Sep 17 00:00:00 2001 From: meiravgri <109056284+meiravgri@users.noreply.github.com> Date: Thu, 11 Dec 2025 17:06:07 +0200 Subject: [PATCH 01/13] MOD-12263 vector iterator details Added detailed notes on vector iterator profiles, including vector search modes and batch execution statistics. --- content/commands/ft.profile.md | 27 +++++++++++++++++++++++++++ 1 file changed, 27 insertions(+) diff --git a/content/commands/ft.profile.md b/content/commands/ft.profile.md index 5554464a1e..b5f1c85ac9 100644 --- a/content/commands/ft.profile.md +++ b/content/commands/ft.profile.md @@ -121,6 +121,7 @@ Iterator types include: * `TAG` with `Term` * `NUMERIC` with `Term` * `VECTOR` +* `METRIC - VECTOR DISTANCE` * `EMPTY` * `WILDCARD` * `OPTIONAL` @@ -131,6 +132,32 @@ Counter is the number of times an iterator was interacted with. A very high valu `Size` is the size of the document set. `Counter` should always be equal or less than `Size`. +**Notes on Vector iterator profile** + +For vector queries, the iterator profile includes additional information specific to vector search execution. + +#### Vector search mode + +The `Vector search mode` field indicates the execution strategy used for both `VECTOR` and `METRIC - VECTOR DISTANCE` iterator types: + +| Mode | Description | +|:-- |:-- | +| `STANDARD_KNN` | Pure KNN vector search without filters. | +| `RANGE_QUERY` | Range-based vector search. used with `VECTOR_RANGE` queries. | +| `HYBRID_ADHOC_BF` | Ad-hoc brute force for filtered queries. Iterates through results that pass the filter and calculates distances on-the-fly. | +| `HYBRID_BATCHES` | Batch-based filtered search. Retrieves top vectors from the index in batches and checks which ones pass the filter. | +| `HYBRID_BATCHES_TO_ADHOC_BF`| Dynamic mode switching. Starts with batch-based search but switches to ad-hoc brute force if batch iterations yield insufficient results. | + +#### Batch execution statistics + +For queries using batch modes (`HYBRID_BATCHES` or `HYBRID_BATCHES_TO_ADHOC_BF`), additional batch statistics are included: + +| Returned field name | Definition | +|:-- |:-- | +| `Batches number` | Total number of batch iterations executed during the search. | +| `Largest batch size`| The maximum batch size used during execution. Batch sizes may vary dynamically based on the ratio of estimated matching documents to total index size. | +| `Largest batch iteration (zero based)` | The iteration number (0-based) when the largest batch occurred. | + ### Result processor profiles Result processors form a powerful pipeline in Redis Query Engine. They work in stages to gather, filter, score, sort, and return results as efficiently as possible based on complex query needs. Each processor reports `Time` information, which represents the total duration (in milliseconds, or ms) spent by the processor to complete its operation, and `Counter` information, which indicates the number of times the processor was invoked during the query. From a9056bb842740658af06a417b73d30901640fb18 Mon Sep 17 00:00:00 2001 From: meiravgri <109056284+meiravgri@users.noreply.github.com> Date: Thu, 11 Dec 2025 17:23:04 +0200 Subject: [PATCH 02/13] =?UTF-8?q?MOD-6056=20rename=20=E2=80=9Ccounter?= =?UTF-8?q?=E2=80=9D=20and=20=E2=80=9Csize=E2=80=9D?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- content/commands/ft.profile.md | 13 +++++++------ 1 file changed, 7 insertions(+), 6 deletions(-) diff --git a/content/commands/ft.profile.md b/content/commands/ft.profile.md index b5f1c85ac9..589edb2d2a 100644 --- a/content/commands/ft.profile.md +++ b/content/commands/ft.profile.md @@ -106,10 +106,10 @@ If there's only one shard, the label will be omitted. ### Iterator profiles -This section contains index iterator information, including `Type`, `Query Type`, `Term` (when applicable), `Time` (in ms), `Counter`, `Child iterator`, and `Size` information. +This section contains index iterator information, including `Type`, `Query Type`, `Term` (when applicable), `Time` (in ms), `Number of reading operations`, `Child iterator`, and `Estimated number of matches` information. Each iterator represents an executor for each part of the query plan, nested per the execution plan. The operation types mentioned below (`UNION`, etc) should match up with the provided query. -Inverted-index iterators also include the number of elements they contain. Hybrid vector iterators return the top results from the vector index in batches, including the number of batches. +Inverted-index iterators also include the number of elements they contain. Vector iterators include additional profiling information. See [Vector iterator profile](#vector-iterator-profile) below. Iterator types include: @@ -126,11 +126,11 @@ Iterator types include: * `WILDCARD` * `OPTIONAL` -**Notes on `Counter` and `Size`** +**Notes on `Number of reading operations` and `Estimated number of matches`** -Counter is the number of times an iterator was interacted with. A very high value in comparison to others is a possible warning flag. `NUMERIC` and `Child interator` types are broken into ranges, and `Counter` will vary depending on the range. For `UNION`, the sum of the counters in child iterators should be equal or greater than the child iterator’s counters. +`Number of reading operations` is the number of times an iterator was interacted with. A very high value in comparison to others is a possible warning flag. `NUMERIC` and `Child iterator` types are broken into ranges, and `Number of reading operations` will vary depending on the range. For `UNION`, the sum of the counters in child iterators should be equal or greater than the child iterator's counters. -`Size` is the size of the document set. `Counter` should always be equal or less than `Size`. +`Estimated number of matches` is the size of the document set. `Number of reading operations` should always be equal or less than `Estimated number of matches`. **Notes on Vector iterator profile** @@ -160,7 +160,8 @@ For queries using batch modes (`HYBRID_BATCHES` or `HYBRID_BATCHES_TO_ADHOC_BF`) ### Result processor profiles -Result processors form a powerful pipeline in Redis Query Engine. They work in stages to gather, filter, score, sort, and return results as efficiently as possible based on complex query needs. Each processor reports `Time` information, which represents the total duration (in milliseconds, or ms) spent by the processor to complete its operation, and `Counter` information, which indicates the number of times the processor was invoked during the query. +Result processors form a powerful pipeline in Redis Query Engine. They work in stages to gather, filter, score, sort, and return results as efficiently as possible based on complex query needs. Each processor reports `Time` information, which represents the total duration (in milliseconds, or ms) spent by the processor to complete its operation, and `Results processed` information, which indicates the number of times the processor was invoked during the query. + | Type | Definition | |:-- |:-- | From 8c47b85cd72ceaad261c82596848cb1b6fca789d Mon Sep 17 00:00:00 2001 From: meiravgri <109056284+meiravgri@users.noreply.github.com> Date: Thu, 11 Dec 2025 17:48:58 +0200 Subject: [PATCH 03/13] MOD-12816 update ResultProcessor GILTime --- content/commands/ft.profile.md | 2 ++ 1 file changed, 2 insertions(+) diff --git a/content/commands/ft.profile.md b/content/commands/ft.profile.md index 589edb2d2a..80ea5c32ef 100644 --- a/content/commands/ft.profile.md +++ b/content/commands/ft.profile.md @@ -102,6 +102,7 @@ If there's only one shard, the label will be omitted. | `Total` `profile` `time` | The total run time (ms) of the query. Normally just a few ms. | | `Parsing` `time` | The time (ms) spent parsing the query and its parameters into a query plan. Normally just a few ms. | | `Pipeline` `creation` `time` | The creation time (ms) of the execution plan, including iterators, result processors, and reducers creation. Normally just a few ms for `FT.SEARCH` queries, but expect a larger number for `FT.AGGREGATE` queries. | +| `Total` `GIL` `time` | The total time (ms) the query held the Global Interpreter Lock (GIL) during execution. Relevant for multi-threaded deployments where queries run in background threads. | | `Warning` | Errors that occurred during query execution. | ### Iterator profiles @@ -170,6 +171,7 @@ Result processors form a powerful pipeline in Redis Query Engine. They work in s | `Scorer` | The `Scorer` processor assigns a relevance score to each document based on the query’s specified scoring function. This function could involve factors like term frequency, inverse document frequency, or other weighted metrics. | | `Sorter` | The `Sorter` processor arranges the query results based on a specified sorting criterion. This could be a field value (e.g., sorting by price, date, or another attribute) or by the score assigned during the scoring phase. It operates after documents are fetched and scored, ensuring the results are ordered as required by the query (e.g., ascending or descending order). `Scorer` results will always be present in `FT.SEARCH` profiles. | | `Loader` | The `Loader` processor retrieves the document contents after the results have been sorted and filtered. It ensures that only the fields specified by the query are loaded, which improves efficiency, especially when dealing with large documents where only a few fields are needed. | +| `Threadsafe-Loader` | The `Threadsafe-Loader` processor safely loads document contents when the query is running in a background thread. It acquires the GIL to access document data. Reports an additional `GIL-Time` field showing how long (ms) it held the GIL. | | `Highlighter` | The `Highlighter` processor is used to highlight matching terms in the search results. This is especially useful for full-text search applications, where relevant terms are often emphasized in the UI. | | `Paginator` | The `Paginator` processor is responsible for handling pagination by limiting the results to a specific range (e.g., LIMIT 0 10).It trims down the set of results to fit the required pagination window, ensuring efficient memory usage when dealing with large result sets. | | `Vector` `Filter` | For vector searches, the `Vector Filter` processor is sometimes used to pre-process results based on vector similarity thresholds before the main scoring and sorting. | From b8bc7a1c420ceeb38da613c49125078a2cd9688e Mon Sep 17 00:00:00 2001 From: meiravgri <109056284+meiravgri@users.noreply.github.com> Date: Thu, 11 Dec 2025 17:58:22 +0200 Subject: [PATCH 04/13] MOD-12414 Add internal cursor reads to profile documentation --- content/commands/ft.profile.md | 1 + 1 file changed, 1 insertion(+) diff --git a/content/commands/ft.profile.md b/content/commands/ft.profile.md index 80ea5c32ef..5d6a31b6fc 100644 --- a/content/commands/ft.profile.md +++ b/content/commands/ft.profile.md @@ -104,6 +104,7 @@ If there's only one shard, the label will be omitted. | `Pipeline` `creation` `time` | The creation time (ms) of the execution plan, including iterators, result processors, and reducers creation. Normally just a few ms for `FT.SEARCH` queries, but expect a larger number for `FT.AGGREGATE` queries. | | `Total` `GIL` `time` | The total time (ms) the query held the Global Interpreter Lock (GIL) during execution. Relevant for multi-threaded deployments where queries run in background threads. | | `Warning` | Errors that occurred during query execution. | +| `Internal` `cursor` `reads` | The number of internal cursor read operations performed during a distributed `AGGREGATE` query in cluster mode. In cluster mode, the coordinator uses cursors to fetch results from shards - this counts the initial request plus any subsequent `FT.CURSOR READ`. | ### Iterator profiles From 917ab314d36227e1a85f05a67917a867da962ce7 Mon Sep 17 00:00:00 2001 From: meiravgri <109056284+meiravgri@users.noreply.github.com> Date: Sun, 14 Dec 2025 07:01:16 +0200 Subject: [PATCH 05/13] add missing iterators --- content/commands/ft.profile.md | 3 +++ 1 file changed, 3 insertions(+) diff --git a/content/commands/ft.profile.md b/content/commands/ft.profile.md index 5d6a31b6fc..9632de1579 100644 --- a/content/commands/ft.profile.md +++ b/content/commands/ft.profile.md @@ -124,9 +124,12 @@ Iterator types include: * `NUMERIC` with `Term` * `VECTOR` * `METRIC - VECTOR DISTANCE` +* `GEO` with `Term` * `EMPTY` * `WILDCARD` * `OPTIONAL` +* `OPTIMIZER` with `Optimizer mode` - Enabled by default in dialect 4+, or explicitly with `WITHOUTCOUNT`. +* `ID-LIST` - Iterator over specific document IDs (appears when using `INKEYS`) **Notes on `Number of reading operations` and `Estimated number of matches`** From 16001a301d34558a7e7e810104889435e0b029ae Mon Sep 17 00:00:00 2001 From: meiravgri <109056284+meiravgri@users.noreply.github.com> Date: Sun, 14 Dec 2025 07:25:55 +0200 Subject: [PATCH 06/13] update RP section should we remove vector filter? --- content/commands/ft.profile.md | 7 +++++-- 1 file changed, 5 insertions(+), 2 deletions(-) diff --git a/content/commands/ft.profile.md b/content/commands/ft.profile.md index 9632de1579..dd20b89c4e 100644 --- a/content/commands/ft.profile.md +++ b/content/commands/ft.profile.md @@ -167,7 +167,6 @@ For queries using batch modes (`HYBRID_BATCHES` or `HYBRID_BATCHES_TO_ADHOC_BF`) Result processors form a powerful pipeline in Redis Query Engine. They work in stages to gather, filter, score, sort, and return results as efficiently as possible based on complex query needs. Each processor reports `Time` information, which represents the total duration (in milliseconds, or ms) spent by the processor to complete its operation, and `Results processed` information, which indicates the number of times the processor was invoked during the query. - | Type | Definition | |:-- |:-- | | `Metrics` `Applier` | The `Metrics Applier` processor calculates or aggregates specific metrics related to the search results. For example, this might include applying a distance or similarity metric to vector search results, or calculating scores based on relevance or other parameters. | @@ -177,8 +176,12 @@ Result processors form a powerful pipeline in Redis Query Engine. They work in s | `Loader` | The `Loader` processor retrieves the document contents after the results have been sorted and filtered. It ensures that only the fields specified by the query are loaded, which improves efficiency, especially when dealing with large documents where only a few fields are needed. | | `Threadsafe-Loader` | The `Threadsafe-Loader` processor safely loads document contents when the query is running in a background thread. It acquires the GIL to access document data. Reports an additional `GIL-Time` field showing how long (ms) it held the GIL. | | `Highlighter` | The `Highlighter` processor is used to highlight matching terms in the search results. This is especially useful for full-text search applications, where relevant terms are often emphasized in the UI. | -| `Paginator` | The `Paginator` processor is responsible for handling pagination by limiting the results to a specific range (e.g., LIMIT 0 10).It trims down the set of results to fit the required pagination window, ensuring efficient memory usage when dealing with large result sets. | +| `Pager/Limiter` | The `Pager/Limiter` processor is responsible for handling pagination by limiting the results to a specific range (e.g., LIMIT 0 10).It trims down the set of results to fit the required pagination window, ensuring efficient memory usage when dealing with large result sets. | | `Vector` `Filter` | For vector searches, the `Vector Filter` processor is sometimes used to pre-process results based on vector similarity thresholds before the main scoring and sorting. | +| `Counter` | The `Counter` processor counts the total number of matching results. Used when running queries with `LIMIT 0 0` to return only the count. | +| `Grouper` | The `Grouper` processor groups results by field values. Used with `GROUPBY` in `FT.AGGREGATE` queries. | +| `Projector` | The `Projector` processor applies field transformations. Used with `APPLY` in `FT.AGGREGATE` queries. | +| `Network` | Collects and merges results from shards. Appears as the root processor in the coordinator's result processor chain for `FT.AGGREGATE` in cluster mode. | ### Coordinator From 6b461bd47a6a64cd6480ae65f6b2cc943b5737f4 Mon Sep 17 00:00:00 2001 From: meiravgri <109056284+meiravgri@users.noreply.github.com> Date: Sun, 14 Dec 2025 07:32:02 +0200 Subject: [PATCH 07/13] fix example with new names --- content/commands/ft.profile.md | 48 +++++++++++++++++----------------- 1 file changed, 24 insertions(+), 24 deletions(-) diff --git a/content/commands/ft.profile.md b/content/commands/ft.profile.md index dd20b89c4e..d49f9a28a9 100644 --- a/content/commands/ft.profile.md +++ b/content/commands/ft.profile.md @@ -133,7 +133,7 @@ Iterator types include: **Notes on `Number of reading operations` and `Estimated number of matches`** -`Number of reading operations` is the number of times an iterator was interacted with. A very high value in comparison to others is a possible warning flag. `NUMERIC` and `Child iterator` types are broken into ranges, and `Number of reading operations` will vary depending on the range. For `UNION`, the sum of the counters in child iterators should be equal or greater than the child iterator's counters. +`Number of reading operations` is the number of times an iterator was interacted with. A very high value in comparison to others is a possible warning flag. `NUMERIC` and `Child iterator` types are broken into ranges, and `Number of reading operations` will vary depending on the range. For `UNION`, the sum of the reading operations in child iterators should be equal or greater than the child iterator's reading operations. `Estimated number of matches` is the size of the document set. `Number of reading operations` should always be equal or less than `Estimated number of matches`. @@ -243,7 +243,7 @@ Here's an example of running the `FT.PROFILE` command for a compound query. 2) INTERSECT 3) Time 4) "0" - 5) Counter + 5) Number of reading operations 6) (integer) 6 7) Child iterators 8) 1) Type @@ -252,7 +252,7 @@ Here's an example of running the `FT.PROFILE` command for a compound query. 4) UNION 5) Time 6) "0" - 7) Counter + 7) Number of reading operations 8) (integer) 6 9) Child iterators 10) 1) Type @@ -261,7 +261,7 @@ Here's an example of running the `FT.PROFILE` command for a compound query. 4) UNION 5) Time 6) "0" - 7) Counter + 7) Number of reading operations 8) (integer) 4 9) Child iterators 10) 1) Type @@ -270,9 +270,9 @@ Here's an example of running the `FT.PROFILE` command for a compound query. 4) kids 5) Time 6) "0" - 7) Counter + 7) Number of reading operations 8) (integer) 4 - 9) Size + 9) Estimated number of matches 10) (integer) 4 11) 1) Type 2) TEXT @@ -280,9 +280,9 @@ Here's an example of running the `FT.PROFILE` command for a compound query. 4) +kid 5) Time 6) "0" - 7) Counter + 7) Number of reading operations 8) (integer) 4 - 9) Size + 9) Estimated number of matches 10) (integer) 4 11) 1) Type 2) TEXT @@ -290,9 +290,9 @@ Here's an example of running the `FT.PROFILE` command for a compound query. 4) small 5) Time 6) "0" - 7) Counter + 7) Number of reading operations 8) (integer) 2 - 9) Size + 9) Estimated number of matches 10) (integer) 2 9) 1) Type 2) UNION @@ -300,7 +300,7 @@ Here's an example of running the `FT.PROFILE` command for a compound query. 4) TAG 5) Time 6) "0" - 7) Counter + 7) Number of reading operations 8) (integer) 6 9) Child iterators 10) 1) Type @@ -309,9 +309,9 @@ Here's an example of running the `FT.PROFILE` command for a compound query. 4) new 5) Time 6) "0" - 7) Counter + 7) Number of reading operations 8) (integer) 4 - 9) Size + 9) Estimated number of matches 10) (integer) 10 11) 1) Type 2) TAG @@ -319,34 +319,34 @@ Here's an example of running the `FT.PROFILE` command for a compound query. 4) used 5) Time 6) "0" - 7) Counter + 7) Number of reading operations 8) (integer) 4 - 9) Size + 9) Estimated number of matches 10) (integer) 8 6) 1) Result processors profile 2) 1) Type 2) Index 3) Time 4) "0" - 5) Counter + 5) Results processed 6) (integer) 3 3) 1) Type 2) Scorer 3) Time 4) "0" - 5) Counter + 5) Results processed 6) (integer) 3 4) 1) Type 2) Sorter 3) Time 4) "0" - 5) Counter + 5) Results processed 6) (integer) 3 5) 1) Type 2) Loader 3) Time 4) "0" - 5) Counter + 5) Results processed 6) (integer) 3 {{< / highlight >}} @@ -403,32 +403,32 @@ Here's an example of running the `FT.PROFILE` command for a vector query. 2) VECTOR 3) Time 4) "0" - 5) Counter + 5) Number of reading operations 6) (integer) 3 6) 1) Result processors profile 2) 1) Type 2) Index 3) Time 4) "0" - 5) Counter + 5) Results processed 6) (integer) 3 3) 1) Type 2) Metrics Applier 3) Time 4) "0" - 5) Counter + 5) Results processed 6) (integer) 3 4) 1) Type 2) Sorter 3) Time 4) "0" - 5) Counter + 5) Results processed 6) (integer) 3 5) 1) Type 2) Loader 3) Time 4) "0" - 5) Counter + 5) Results processed 6) (integer) 3 {{< /highlight >}} From 7c3947b751cb6ef7692c57091a0467a225035289 Mon Sep 17 00:00:00 2001 From: meiravgri <109056284+meiravgri@users.noreply.github.com> Date: Sun, 14 Dec 2025 10:26:51 +0200 Subject: [PATCH 08/13] remove vector filter --- content/commands/ft.profile.md | 1 - 1 file changed, 1 deletion(-) diff --git a/content/commands/ft.profile.md b/content/commands/ft.profile.md index d49f9a28a9..6744bc5006 100644 --- a/content/commands/ft.profile.md +++ b/content/commands/ft.profile.md @@ -177,7 +177,6 @@ Result processors form a powerful pipeline in Redis Query Engine. They work in s | `Threadsafe-Loader` | The `Threadsafe-Loader` processor safely loads document contents when the query is running in a background thread. It acquires the GIL to access document data. Reports an additional `GIL-Time` field showing how long (ms) it held the GIL. | | `Highlighter` | The `Highlighter` processor is used to highlight matching terms in the search results. This is especially useful for full-text search applications, where relevant terms are often emphasized in the UI. | | `Pager/Limiter` | The `Pager/Limiter` processor is responsible for handling pagination by limiting the results to a specific range (e.g., LIMIT 0 10).It trims down the set of results to fit the required pagination window, ensuring efficient memory usage when dealing with large result sets. | -| `Vector` `Filter` | For vector searches, the `Vector Filter` processor is sometimes used to pre-process results based on vector similarity thresholds before the main scoring and sorting. | | `Counter` | The `Counter` processor counts the total number of matching results. Used when running queries with `LIMIT 0 0` to return only the count. | | `Grouper` | The `Grouper` processor groups results by field values. Used with `GROUPBY` in `FT.AGGREGATE` queries. | | `Projector` | The `Projector` processor applies field transformations. Used with `APPLY` in `FT.AGGREGATE` queries. | From ea606d5dbddea725811f47408a6ae7032e853b47 Mon Sep 17 00:00:00 2001 From: meiravgri <109056284+meiravgri@users.noreply.github.com> Date: Sun, 14 Dec 2025 16:51:04 +0200 Subject: [PATCH 09/13] refine internal cursor description --- content/commands/ft.profile.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/content/commands/ft.profile.md b/content/commands/ft.profile.md index 6744bc5006..1b888f24cb 100644 --- a/content/commands/ft.profile.md +++ b/content/commands/ft.profile.md @@ -104,7 +104,7 @@ If there's only one shard, the label will be omitted. | `Pipeline` `creation` `time` | The creation time (ms) of the execution plan, including iterators, result processors, and reducers creation. Normally just a few ms for `FT.SEARCH` queries, but expect a larger number for `FT.AGGREGATE` queries. | | `Total` `GIL` `time` | The total time (ms) the query held the Global Interpreter Lock (GIL) during execution. Relevant for multi-threaded deployments where queries run in background threads. | | `Warning` | Errors that occurred during query execution. | -| `Internal` `cursor` `reads` | The number of internal cursor read operations performed during a distributed `AGGREGATE` query in cluster mode. In cluster mode, the coordinator uses cursors to fetch results from shards - this counts the initial request plus any subsequent `FT.CURSOR READ`. | +| `Internal` `cursor` `reads` | The number of times the coordinator fetched result batches from a given shard during a distributed `AGGREGATE` query in cluster mode. Includes the initial request plus any subsequent batch fetches. | ### Iterator profiles From 2ecda016d8cc93b76991797fbe5b1888bf662f06 Mon Sep 17 00:00:00 2001 From: meiravgri <109056284+meiravgri@users.noreply.github.com> Date: Tue, 16 Dec 2025 07:22:54 +0200 Subject: [PATCH 10/13] fix GIL entreies & optimizer --- content/commands/ft.profile.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/content/commands/ft.profile.md b/content/commands/ft.profile.md index 1b888f24cb..0fa54695a2 100644 --- a/content/commands/ft.profile.md +++ b/content/commands/ft.profile.md @@ -102,7 +102,7 @@ If there's only one shard, the label will be omitted. | `Total` `profile` `time` | The total run time (ms) of the query. Normally just a few ms. | | `Parsing` `time` | The time (ms) spent parsing the query and its parameters into a query plan. Normally just a few ms. | | `Pipeline` `creation` `time` | The creation time (ms) of the execution plan, including iterators, result processors, and reducers creation. Normally just a few ms for `FT.SEARCH` queries, but expect a larger number for `FT.AGGREGATE` queries. | -| `Total` `GIL` `time` | The total time (ms) the query held the Global Interpreter Lock (GIL) during execution. Relevant for multi-threaded deployments where queries run in background threads. | +| `Total` `GIL` `time` | In multi-threaded deployments, the total time (ms) the query spent holding and waiting for the Redis Global Lock (referred to as GIL) during execution. Note: this value is only valid in the shards profile section. | | `Warning` | Errors that occurred during query execution. | | `Internal` `cursor` `reads` | The number of times the coordinator fetched result batches from a given shard during a distributed `AGGREGATE` query in cluster mode. Includes the initial request plus any subsequent batch fetches. | @@ -128,7 +128,7 @@ Iterator types include: * `EMPTY` * `WILDCARD` * `OPTIONAL` -* `OPTIMIZER` with `Optimizer mode` - Enabled by default in dialect 4+, or explicitly with `WITHOUTCOUNT`. +* `OPTIMIZER` with `Optimizer mode` - Query optimization wrapper for `FT.SEARCH` queries sorted by a numeric field. Enabled by default in dialect 4+, or explicitly with `WITHOUTCOUNT`. * `ID-LIST` - Iterator over specific document IDs (appears when using `INKEYS`) **Notes on `Number of reading operations` and `Estimated number of matches`** @@ -174,7 +174,7 @@ Result processors form a powerful pipeline in Redis Query Engine. They work in s | `Scorer` | The `Scorer` processor assigns a relevance score to each document based on the query’s specified scoring function. This function could involve factors like term frequency, inverse document frequency, or other weighted metrics. | | `Sorter` | The `Sorter` processor arranges the query results based on a specified sorting criterion. This could be a field value (e.g., sorting by price, date, or another attribute) or by the score assigned during the scoring phase. It operates after documents are fetched and scored, ensuring the results are ordered as required by the query (e.g., ascending or descending order). `Scorer` results will always be present in `FT.SEARCH` profiles. | | `Loader` | The `Loader` processor retrieves the document contents after the results have been sorted and filtered. It ensures that only the fields specified by the query are loaded, which improves efficiency, especially when dealing with large documents where only a few fields are needed. | -| `Threadsafe-Loader` | The `Threadsafe-Loader` processor safely loads document contents when the query is running in a background thread. It acquires the GIL to access document data. Reports an additional `GIL-Time` field showing how long (ms) it held the GIL. | +| `Threadsafe-Loader` | The `Threadsafe-Loader` processor safely loads document contents when the query runs in a background thread (relevant for multi-threaded deployments). It acquires the Redis Global Lock (referred to as GIL) to access document data. Reports an additional `GIL-Time` field representing the time (ms) spent holding the GIL and waiting to acquire it. | | `Highlighter` | The `Highlighter` processor is used to highlight matching terms in the search results. This is especially useful for full-text search applications, where relevant terms are often emphasized in the UI. | | `Pager/Limiter` | The `Pager/Limiter` processor is responsible for handling pagination by limiting the results to a specific range (e.g., LIMIT 0 10).It trims down the set of results to fit the required pagination window, ensuring efficient memory usage when dealing with large result sets. | | `Counter` | The `Counter` processor counts the total number of matching results. Used when running queries with `LIMIT 0 0` to return only the count. | From f1f2b52ea5f40eeea56123196ea4b609913d90a5 Mon Sep 17 00:00:00 2001 From: alon Date: Tue, 16 Dec 2025 14:54:21 +0200 Subject: [PATCH 11/13] add versions and format disclaimer + Shard ID field --- content/commands/ft.profile.md | 18 +++++++++++------- 1 file changed, 11 insertions(+), 7 deletions(-) diff --git a/content/commands/ft.profile.md b/content/commands/ft.profile.md index 0fa54695a2..d8babf2a16 100644 --- a/content/commands/ft.profile.md +++ b/content/commands/ft.profile.md @@ -87,6 +87,9 @@ is the query string, sent to `FT.SEARCH` or `FT.AGGREGATE`. ## Return +**Note**: This page contains the up-to-date profile output as of RediSearch v2.8.33, v2.10.26, v8.2.7, and v8.4.4. +The output format itself may differ between RediSearch versions, and RESP protocol versions. + `FT.PROFILE` returns a two-element array reply. The first element contains the results of the provided `FT.SEARCH` or `FT.AGGREGATE` command. The second element contains information about query creation, iterator profiles, and result processor profiles. Details of the second element follow in the sections below. @@ -97,14 +100,15 @@ This section contains query execution details for each shard. When more than one shard is in play, the shards will be labeled `Shard #1`, `Shard #2`, etc. If there's only one shard, the label will be omitted. -| Returned field name | Definition | -|:-- |:-- | -| `Total` `profile` `time` | The total run time (ms) of the query. Normally just a few ms. | -| `Parsing` `time` | The time (ms) spent parsing the query and its parameters into a query plan. Normally just a few ms. | -| `Pipeline` `creation` `time` | The creation time (ms) of the execution plan, including iterators, result processors, and reducers creation. Normally just a few ms for `FT.SEARCH` queries, but expect a larger number for `FT.AGGREGATE` queries. | +| Returned field name | Definition | +|:-- |:----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| +| `Shard ID` | String containing the unique shard ID. In OSS, this identifier is denoted as the node ID, and is recievend from `RedisModule_GetMyClusterID()` API. | +| `Total` `profile` `time` | The total run time (ms) of the query. Normally just a few ms. | +| `Parsing` `time` | The time (ms) spent parsing the query and its parameters into a query plan. Normally just a few ms. | +| `Pipeline` `creation` `time` | The creation time (ms) of the execution plan, including iterators, result processors, and reducers creation. Normally just a few ms for `FT.SEARCH` queries, but expect a larger number for `FT.AGGREGATE` queries. | | `Total` `GIL` `time` | In multi-threaded deployments, the total time (ms) the query spent holding and waiting for the Redis Global Lock (referred to as GIL) during execution. Note: this value is only valid in the shards profile section. | -| `Warning` | Errors that occurred during query execution. | -| `Internal` `cursor` `reads` | The number of times the coordinator fetched result batches from a given shard during a distributed `AGGREGATE` query in cluster mode. Includes the initial request plus any subsequent batch fetches. | +| `Warning` | Errors that occurred during query execution. | +| `Internal` `cursor` `reads` | The number of times the coordinator fetched result batches from a given shard during a distributed `AGGREGATE` query in cluster mode. Includes the initial request plus any subsequent batch fetches. | ### Iterator profiles From d36ef2734d5da370192001a7bf96cbb97ccc4b12 Mon Sep 17 00:00:00 2001 From: alon Date: Tue, 16 Dec 2025 14:57:10 +0200 Subject: [PATCH 12/13] fix version disclaimer --- content/commands/ft.profile.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/content/commands/ft.profile.md b/content/commands/ft.profile.md index d8babf2a16..15559173ab 100644 --- a/content/commands/ft.profile.md +++ b/content/commands/ft.profile.md @@ -87,7 +87,7 @@ is the query string, sent to `FT.SEARCH` or `FT.AGGREGATE`. ## Return -**Note**: This page contains the up-to-date profile output as of RediSearch v2.8.33, v2.10.26, v8.2.7, and v8.4.4. +**Note**: This page contains the up-to-date profile output as of RediSearch v2.8.33, v2.10.26, v8.2.7, and v8.4.3. The output format itself may differ between RediSearch versions, and RESP protocol versions. `FT.PROFILE` returns a two-element array reply. The first element contains the results of the provided `FT.SEARCH` or `FT.AGGREGATE` command. @@ -102,7 +102,7 @@ If there's only one shard, the label will be omitted. | Returned field name | Definition | |:-- |:----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| -| `Shard ID` | String containing the unique shard ID. In OSS, this identifier is denoted as the node ID, and is recievend from `RedisModule_GetMyClusterID()` API. | +| `Shard ID` | String containing the unique shard ID. In OSS, this identifier is denoted as the node ID, and is recievend from `RedisModule_GetMyClusterID()` API. (Available as of RediSearch v8.4.3) | | `Total` `profile` `time` | The total run time (ms) of the query. Normally just a few ms. | | `Parsing` `time` | The time (ms) spent parsing the query and its parameters into a query plan. Normally just a few ms. | | `Pipeline` `creation` `time` | The creation time (ms) of the execution plan, including iterators, result processors, and reducers creation. Normally just a few ms for `FT.SEARCH` queries, but expect a larger number for `FT.AGGREGATE` queries. | From f886a42c74725ab9c0831c19dac794a51f675b28 Mon Sep 17 00:00:00 2001 From: "David W. Dougherty" Date: Tue, 16 Dec 2025 07:00:33 -0800 Subject: [PATCH 13/13] DWD copy-edit pass --- content/commands/ft.profile.md | 22 ++++++++++++---------- 1 file changed, 12 insertions(+), 10 deletions(-) diff --git a/content/commands/ft.profile.md b/content/commands/ft.profile.md index 15559173ab..8092e2be1f 100644 --- a/content/commands/ft.profile.md +++ b/content/commands/ft.profile.md @@ -87,8 +87,10 @@ is the query string, sent to `FT.SEARCH` or `FT.AGGREGATE`. ## Return -**Note**: This page contains the up-to-date profile output as of RediSearch v2.8.33, v2.10.26, v8.2.7, and v8.4.3. +{{< note>}} +This page contains the up-to-date profile output as of RediSearch v2.8.33, v2.10.26, v8.2.7, and v8.4.3. The output format itself may differ between RediSearch versions, and RESP protocol versions. +{{< /note >}} `FT.PROFILE` returns a two-element array reply. The first element contains the results of the provided `FT.SEARCH` or `FT.AGGREGATE` command. The second element contains information about query creation, iterator profiles, and result processor profiles. @@ -102,7 +104,7 @@ If there's only one shard, the label will be omitted. | Returned field name | Definition | |:-- |:----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| -| `Shard ID` | String containing the unique shard ID. In OSS, this identifier is denoted as the node ID, and is recievend from `RedisModule_GetMyClusterID()` API. (Available as of RediSearch v8.4.3) | +| `Shard ID` | String containing the unique shard ID. In Redis Open Source, this identifier is denoted as the node ID, and is received from the `RedisModule_GetMyClusterID()` API. (Available as of RediSearch v8.4.3) | | `Total` `profile` `time` | The total run time (ms) of the query. Normally just a few ms. | | `Parsing` `time` | The time (ms) spent parsing the query and its parameters into a query plan. Normally just a few ms. | | `Pipeline` `creation` `time` | The creation time (ms) of the execution plan, including iterators, result processors, and reducers creation. Normally just a few ms for `FT.SEARCH` queries, but expect a larger number for `FT.AGGREGATE` queries. | @@ -112,10 +114,10 @@ If there's only one shard, the label will be omitted. ### Iterator profiles -This section contains index iterator information, including `Type`, `Query Type`, `Term` (when applicable), `Time` (in ms), `Number of reading operations`, `Child iterator`, and `Estimated number of matches` information. -Each iterator represents an executor for each part of the query plan, nested per the execution plan. The operation types mentioned below (`UNION`, etc) should match up with the provided query. +This section contains index iterator information, including `Type`, `Query Type`, `Term` (when applicable), `Time` (ms), `Number of reading operations`, `Child iterator`, and `Estimated number of matches` information. +Each iterator represents an executor for each part of the query plan, nested per the execution plan. The operation types mentioned below (for example, `UNION`) should match up with the provided query. -Inverted-index iterators also include the number of elements they contain. Vector iterators include additional profiling information. See [Vector iterator profile](#vector-iterator-profile) below. +Inverted-index iterators also include the number of elements they contain. Vector iterators include additional profiling information. See [Notes on Vector iterator profile](#notes-onvector-iterator-profile) below. Iterator types include: @@ -132,16 +134,16 @@ Iterator types include: * `EMPTY` * `WILDCARD` * `OPTIONAL` -* `OPTIMIZER` with `Optimizer mode` - Query optimization wrapper for `FT.SEARCH` queries sorted by a numeric field. Enabled by default in dialect 4+, or explicitly with `WITHOUTCOUNT`. +* `OPTIMIZER` with `Optimizer mode` - Query optimization wrapper for `FT.SEARCH` queries sorted by a numeric field. Enabled by default in [DIALECT 4]({{< relref "/develop/ai/search-and-query/advanced-concepts/dialects##dialect-4-deprecated" >}}) or explicitly with `WITHOUTCOUNT`. * `ID-LIST` - Iterator over specific document IDs (appears when using `INKEYS`) **Notes on `Number of reading operations` and `Estimated number of matches`** -`Number of reading operations` is the number of times an iterator was interacted with. A very high value in comparison to others is a possible warning flag. `NUMERIC` and `Child iterator` types are broken into ranges, and `Number of reading operations` will vary depending on the range. For `UNION`, the sum of the reading operations in child iterators should be equal or greater than the child iterator's reading operations. +`Number of reading operations` is the number of times an iterator was interacted with. A very high value in comparison to others is a possible warning flag. `NUMERIC` and `Child iterator` types are broken into ranges, and `Number of reading operations` will vary depending on the range. For `UNION`, the sum of the reading operations in child iterators should be equal to or greater than the child iterator's reading operations. -`Estimated number of matches` is the size of the document set. `Number of reading operations` should always be equal or less than `Estimated number of matches`. +`Estimated number of matches` is the size of the document set. `Number of reading operations` should always be equal to or less than `Estimated number of matches`. -**Notes on Vector iterator profile** +#### Notes on Vector iterator profile {#notes-onvector-iterator-profile} For vector queries, the iterator profile includes additional information specific to vector search execution. @@ -152,7 +154,7 @@ The `Vector search mode` field indicates the execution strategy used for both `V | Mode | Description | |:-- |:-- | | `STANDARD_KNN` | Pure KNN vector search without filters. | -| `RANGE_QUERY` | Range-based vector search. used with `VECTOR_RANGE` queries. | +| `RANGE_QUERY` | Range-based vector search. Used with `VECTOR_RANGE` queries. | | `HYBRID_ADHOC_BF` | Ad-hoc brute force for filtered queries. Iterates through results that pass the filter and calculates distances on-the-fly. | | `HYBRID_BATCHES` | Batch-based filtered search. Retrieves top vectors from the index in batches and checks which ones pass the filter. | | `HYBRID_BATCHES_TO_ADHOC_BF`| Dynamic mode switching. Starts with batch-based search but switches to ad-hoc brute force if batch iterations yield insufficient results. |