Building The Initial Server

I was thinking recently that there could be a lot of value in deploying a local server loaded with a local LLM, MCP and automation tools. Specifically one that could assist with various tasks and whatnot around my home network. Maybe it could handle some automations as well, get some schedules, review security footage. Regardless of the outcome and final usage, the overall experience would help me learn a lot about deploying such a system.

The benefits of rolling your own local LLM server are many but one of the most important is privacy, and the close second is controlled context. Keeping all, or at least most of my information, data, and output behind my firewall means I have greater control over the information I wish to use, and make my server specialized for tasks and projects specific to me and my working environment.

The Hardware

I commandeered an old gaming rig that I pieced together over COVID. The machine had been a reliable workhorse for a few years, and although it is WAAYY underpowered, it would have to do for now. Besides, I like seeing what I can squeeze out of a humble set of hardware to make a stable server.

Below are the server specs specifications:

As you can see, very much underpowered for an LLM server, but it has slightly more than the minimum needed to perform with efficiency. I will eventually swap out the GPU with something more powerful. Most likely I’ll find a refurb or something after the holidays.

Now that I had the hardware, I needed to choose a stable OS that can easily run a server and be lightweight. My first thought was to dust off my FreeBSD cheatsheets, but I wanted to work with something that would, or at least could, be more widely used and would not require too much tweaking. The only choice for me was Debian.

The Initial Setup: Debian and Beyond

I could have gone with Ubuntu, or another feature rich Linux distro like Arch (I see you Arch peeps), or Mint, but Debian is a distro I know well, it’s lightweight and more importantly, it’s stable. If done right, this machine can be a reliable working LLM and automated task server with solid resilience allowing me to upgrade hardware without much trouble.

With Debian installed, I went with XFCE as the desktop, again because it is lightweight and I can easily customize it to my liking. A few more things needed to be added, which were the specific hardware drivers for the AMD chipsets I was using. These I found at the AMD website and I installed them manually.*

Note: When adding AMD drivers or any proprietary driver, you may find instructions about adding sources to your sources.list that are specific to your distro. This can cause issues when you do a distro upgrade. This is why, when using Debian, it is better to manually install those drivers and update them manually or with a shell script.

Adding Docker for Easy Deployment

Using Docker was part of the initial plan for this server. In a previous post I explained the benefits of using Docker for development. In this case, I need to make specific LLMs, tools and other services communicate with other nodes on my network. I also need to make these services available to users on my network. All that is to say, I need a web server. Instead of installing all that is needed for running a web server on my network, I will use Docker containers configured with the specifications I need to run my server and the services.

Two main containers are going to be running. The first will be a n8n container, along with a Nginx for reverse proxy connection. This will allow me to have a secure connection over my network and avoid any “man-in-the-middle” vulnerabilities that could arise.

The second container will run a PostgreSQL database to store information output by n8n automations. This database will also allow me to use the content within for other projects and automations, as well as various MCPs I intend to use.

What’s Next

Configuring n8n for tasks and making the server available on my network with some DNS routing using my PFSense router.

Please leave a comment if you have any feedback or advice. Happy making!