1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
|
# Privacy-First Command-Line AI for Linux
Unlock the power of AI—right from your Linux terminal.
This project delivers a fully local AI environment, running open source language models directly on your machine.
No cloud. No GAFAM. Just full privacy, control, and the freedom to manipulate commands in your shell.
## How it works
* [Ollama](https://ollama.com/) run language models on the local machine.
* [openedai-speech](https://github.com/matatonic/openedai-speech) provides text-to-speech capability.
* [nginx](https://nginx.org/en/) add an authentication to the API.
* [AIChat](https://github.com/sigoden/aichat) is used as LLM CLI tool featuring Shell Assistant, Chat-REPL, RAG, AI Tools & Agents.
Everything is free, open-source, and automated using Docker Compose and shell scripts.
It doesn't require an internet connection to work once the models have been downloaded.
## Requirements
To run this project efficiently, a powerful computer with a recent NVIDIA GPU is required.
As an example, I achieved good performance with an Intel(R) Core(TM) i7-14700HX, a GeForce RTX 4050, and 32GB of RAM using the [qwen2.5:7b](https://ollama.com/library/qwen2.5) model.
The model [qwen2.5-coder:32b](https://ollama.com/library/qwen2.5-coder:32b) is usable but slow with this configuration.
Note that more modest models like [llama3.2:3b](https://ollama.com/library/llama3.2) require much fewer resources and still allow you to do a lot of things.
On GNU/Linux, you must use the [NVIDIA Container Toolkit](https://github.com/NVIDIA/nvidia-container-toolkit).
Note that it is probably possible to run the project on other GPUs or modern MacBooks, but this is not the purpose of this project.
## How to launch the server
Choose the models you wish to use in the docker-compose.yaml file.
```
environment:
- MODELS=....
```
Add an API key to secure server access by adding a `.env` file like this:
```
LLM_API_KEY=1234567890
```
Create a user authentication for aichat Web UI:
```
htpasswd -c src/nginx/htpasswd user
```
Next, start the servers and their configuration with Docker Compose:
```bash
docker compose up --build -d
```
Then wait for the models to finish downloading using the following command to display the status:
```
docker-compose logs -f ollama_provision
```
## How to use
The `setup_desktop.sh` script allows for copying a compiled static version of [AIChat](https://github.com/sigoden/aichat) from a container to your host and configuring the tool.
### AIChat essentials
A request to populate a demo database:
```bash
aichat "10 fictitious identities with username, firstname, lastname and email then display in json format. The data must be realistic, especially from known email domains."
```
Request a snippet of code:
```
aichat -m ollama:qwen2.5-coder:32b -c "if a docker image exist in bash"
```
To launch a chatbot while maintaining context:
```bash
aichat -s
```
Pipe a command and interpret the result:
```
ps aux | aichat 'which process use the most memory'
```
Using roles:
```
aichat -r short "tcp port of mysql"
./tools/speech.sh synthesize --play --lang en --voice bryce "$(aichat -r english-translator 'Bienvenue dans le monde de l AI et de la ligne de commande.')"
```
Go to the [AIChat](https://github.com/sigoden/aichat) website for other possible use cases.
### Text To Speech
For this features, use the speech.sh script like this:
```
./speech.sh synthesize --play --lang fr --voice pierre "Bonjour, aujourd'hui nous somme le $(date +%A\ %d\ %B\ %Y)."
```
## How to Use Remotely
The API authentication via Nginx allows you to open the API on the internet and use it remotely.
By adding a reverse proxy like Caddy in front of it, you can also add TLS encryption.
This way, you can securely use this environment remotely.
To use script tools in a remote context, use the environment variables TTS_API_HOST and modify AIChat config (~/.config/aichat/config.yaml) .
Example:
```
export TTS_API_HOST="https://your-remote-domain"
./tools/speech.sh ...
```
## Web UI
A web application to interact with supported LLMs directly from your browser is available at [http://127.0.0.1:8001/playground](http://127.0.0.1:8001/playground).
A web platform to compare different LLMs side-by-side is available at [http://127.0.0.1:8001/arena](http://127.0.0.1:8001/arena).
|
