Reuse KV cache of prefixes#484

Draft

tohtana wants to merge 11 commits intomainfrom

tohtana/cache_prefix

Commits on Apr 23, 2024

reuse kv cache prefix
Masahiro Tanaka
committed

Commits on Apr 26, 2024

fix argument
Masahiro Tanaka
committed

Commits on Apr 28, 2024

fix alloc arg
Masahiro Tanaka
committed

Commits on May 8, 2024

add option to prefix cache
Masahiro Tanaka
committed

Commits on May 17, 2024

flush the last request
Masahiro Tanaka
committed

Commits on May 25, 2024

add debug func
Masahiro Tanaka
committed

Commits on May 27, 2024

remove debug function
Masahiro Tanaka
committed
Merge branch 'main' into tohtana/cache_prefix
tohtana
authored

Commits on May 30, 2024

save prefix cache at every iteration
Masahiro Tanaka
committed

Commits on May 31, 2024

fix token count for prefix cache
Masahiro Tanaka
committed

Commits on Jun 3, 2024

fix prefix cache target
Masahiro Tanaka
committed