VAF 0.1.0a26: lower cost per request through prompt caching
VAF's daily spend limit stopped too late because the cached share went uncounted on Anthropic; 0.1.0a26 counts it.
0.1.0a26 is out. The agent carries a fixed set of tools, and its instructions are byte-for-byte identical from one message to the next. The clock, the guidance for the task at hand and the tool list used to sit at the front and changed with every message. They now ride behind the conversation instead. Measured in live use, seven in ten tokens of a chat request are reused, and the same conversation costs a third less.
pip install -U --pre vafTwo bugs are fixed:
- The cost estimate prices cached tokens at the rate the provider publishes for them. On Anthropic they did not count at all: the estimate ran low, and VAF's daily spend limit stopped too late. On OpenAI-compatible providers the whole prompt was charged in full instead. Without a published rate VAF charges in full, so the estimate stays an upper bound.
- The usage view books a call's tokens against the call that spent them. Providers report what a request cost in a closing part of the response that carries no text, and VAF read it too early: every call carried the previous call's figures. Older records are not corrected.
Four model-provider settings can no longer be changed from every account on the network, among them the endpoint prompts leave the machine through. All four are admin-only now. Anyone who set one from a non-admin account has an administrator set it again.
The usage view arrived in 0.1.0a24; since 0.1.0a26 every line belongs to the call it bills.