Getting started
The kalavai client is the main tool to interact with the Kalavai platform, to create and manage both local and public pools and also to interact with them (e.g. deploy models). Let's go over its installation.
From release v0.5.0, you can now install kalavai client in non-worker computers. You can run a pool on a set of machines and have the client on a remote computer from which you access the LLM pool. Because the client only requires having python installed, this means more computers are now supported to run it.
Requirements to run the client
- Python 3.6+
- For seed and workers: Docker engine installed (for linux, Windows and MacOS) with privilege access.
Install the client
The client is a python package and can be installed with one command:
pip install kalavai-client
Createa a local, private LLM pool
Kalavai is free to use, no caps, for both commercial and non-commercial purposes. All you need to get started is one or more computers that can see each other (i.e. within the same network), and you are good to go. If you are interested in join computers in different locations / networks, contact us or book a demo with the founders.
You can create and manage your pools with the new kalavai GUI, which can be started with:
$ kalavai gui start
[+] Running 1/1
✔ Container kalavai_gui Started0.1s
Loading GUI, may take a few minutes. It will be available at http://localhost:3000
This will expose the GUI and the backend services in localhost. By default, the GUI is accessible via http://localhost:3000
1. Create an LLM pool
After you have started the GUI, you can create your LLM pool by clicking on the circle-plus button. Give the pool a name, and select an IP to use as the pool address. Note that this address will need to be visible by worker machines that want to join in.
You can also create a pool using the CLI command:
kalavai pool start NAME
Note: Currently seed nodes are only supported in Linux x86_64.
2. Add worker nodes
Important: only nodes within the same network as the seed node (the one that created the pool) can be added successfully. If you are interested in join computers in different locations / networks, contact us or book a demo with the founders.
Increase the power of your AI pool by inviting others to join. For that, you need to generate a joining token. Use the navigation panel to go to Devices, and then click the circle-plus button to add new devices. You can select the Access mode, which determine the level of access new nodes will have over the pool:
- admin: Same level of access than the seed node, including generating new joining tokens and deleting nodes.
- user: Can deploy jobs, but lacks admin access over nodes.
- worker: Workers carry on jobs, but cannot deploy their own jobs.
Or use the CLI:
kalavai pool token --worker
Copy the joining token and share it with others. On the machines you want to add to the pool, after logging in to kalavai GUI, paste the joining token in the text field under Access with token, and click join
Or join with the CLI:
kalavai pool join <TOKEN>
Kalavai asks you if you want to join (run workloads in the local machine) or attach (use the node to access and control the pool, without running workloads) to the pool.
3. Explore resources
For both seed and worker nodes, the dashboard shows a high level view of the LLM pool: resources available, current utilisation and active devices and deployments.
Use the navigation bar to see more details on key resources:
- Devices: every machine connected to the pool and its current status
- GPUs: list of all available and utilised GPUs
- Jobs: all models and deployments active in the pool
4. Leave the pool
Any device can leave the pool at any point and its workload will get reassigned. To leave the pool, click the circle-stop button on the dashboard, under Local status card. Nodes can rejoin at any point following the above procedure.
Or do so with the CLI:
kalavai pool stop
What's next
Now that you know how to get a pool up and running, check our end to end tutorial on how to self-host an LLM Pool.