Skip to content

Why are chunks still passed in the prompt when using KV cache? #7

@Wailsafi

Description

@Wailsafi

First of all, thank you for your excellent work!

I wanted to understand why the chunks are still being passed in the prompt even when using the KV cache. From my understanding, the purpose of KV caching is to avoid passing chunks in the prompt and to prevent redundant computations in the attention matrices .

Could you please clarify this? am i missing something here ?

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions