How to build a local-first AI infrastructure
With VAF, the AI agent and its language model run on a machine in your network, and from outside you reach it through a VPN, not an open port.
Local-first means your data and your work live on your own devices first, and the network stays optional. The term was proposed in a 2019 essay from the lab Ink & Switch, then for software in general. Applied to AI, the agent, its memory, its built-in tools and the language model run on a machine in your own network.
Four parts fit together. A machine in your network runs around the clock, and the language model runs on it. VAF is the agent, with memory, tools and an account for each person. Your team reaches it over HTTPS in the office and through a VPN from outside.
Which machine is enough
How much memory you need depends on the model. At 16-bit precision a model takes about 2 GB per billion parameters, according to the Hugging Face documentation. In their example, a model with just over 15 billion parameters measured about 29 GB, and about 9.5 GB at 4-bit, and the running conversation takes memory on top. How fast the model writes its answer token by token depends mostly on memory bandwidth when it serves few requests at once, as NVIDIA and Databricks explain in their inference guides.
- An existing PC or server is enough for a smaller local model. By default VAF picks the model tier to fit the hardware and needs roughly 3 to 18 GB of RAM or VRAM depending on the tier.
- Compact machines with large memory shared by processor and graphics hold big models too. Examples are the NVIDIA DGX Spark with 128 GB and mini PCs with an AMD Ryzen AI Max+ 395 and, according to AMD, up to 128 GB. NVIDIA rates the DGX Spark at 273 GB/s of memory bandwidth.
- A workstation with a graphics card computes in the card's own memory as long as the model fits into it. NVIDIA rates the RTX PRO 6000 at 96 GB with 1,792 GB/s.
If the machine is not strong enough for the model your work needs, you connect one over an API, for example the Veyllo API. More on that below.
Setting up VAF and sharing it on your network
On Linux the installer asks for the mode. In server mode it sets up a service that starts with the machine, even with nobody logged in, generates the certificates and opens the firewall for the access port on the local network.
git clone https://github.com/Veyllo-Labs/VAF.git && cd VAF
chmod +x install.sh && ./install.sh --serverIn server mode the access address, the certificate and the firewall rule are bound to the IP address. Give the server a static IP or reserve it in the router.
On a desktop, a switch in the settings or vaf server on makes VAF reachable over HTTPS on your local network; on Windows this can fail at the firewall without administrator rights. Your team opens the machine's address in a browser, usually on port 8443, and on the first visit accepts the self-generated certificate or installs VAF's certificate authority as trusted.
Anyone opening the web interface over the network signs in, with a second factor by default. An administrator creates the accounts and sets their rights in the settings.
The local model
For local models, VAF starts its own llama.cpp server with a model in GGUF format. On Linux it computes with Vulkan on NVIDIA, AMD and Intel graphics cards, or on the processor when there is none; on Windows with CUDA, ROCm, SYCL or Vulkan; on Apple-silicon Macs with Metal. VAF sends no telemetry.
For machines with limited GPU power or VRAM, the setup wizard explicitly offers to take a key for the Veyllo API. The model then no longer runs locally, and your prompts and tool results go to that provider. The wizard also switches image recognition and, unless another speech service was chosen, speech-to-text to Veyllo. VAF still stores memory, credentials and files in ~/.vaf/ and in your own database container.
From outside, through a VPN
Do not open the access port to the internet; VAF's security policy rules that out explicitly. A public network cannot be admitted either, and VAF refuses API requests from addresses outside the admitted networks, though the login page is still served. You bring the VPN yourself, because VAF does not set one up.
Devices on WireGuard or OpenVPN with addresses in the 10.x range are admitted by default, provided the VPN reports a network of its own; a WireGuard with /32 addresses is admitted with vaf server networks allow. Tailscale, Headscale and NetBird get in after one command. On a server without a screen, VAF needs passwordless sudo for the firewall, otherwise you open the port yourself.
vaf server networks tailscale onvaf server vpn-only on locks the local network out. On a rented server, keep the access port closed in the provider's firewall, open only the VPN's own port there, and set VAF up with vaf server provision --no-firewall.
Limits worth knowing
By default a single worker handles chat requests one after another, so one person's long chat run holds up the others' chat requests. VAF is alpha software, and the project's security policy calls for private and internal use, not production.