2024 end of year update

Its time to summarize what happened in my basement during the year... Some things are gone, some new stuff has arrived , power consumption increased by 30% and almost 2k has been spent... again (i really need to stop this)...

On the power side...

I have sold the old and power hungry Eaton 9130 and replaced it with a newer Eaton 5PX 2200i which is slightly less power hungry, but still consuming more than expected despite being significantly newer, line-interactive instead of online and has a lower capacity...

So this has pushed by homelab consumption almost permanently above 200 Watts. Also had some "fun" with the network add-on card, that required some special (although in hindsight very obvious) cable wiring to be reset and usable.

But OMG the display on that thing is simply gorgeous...

Also I have acquired a switched power distribution unit, with remotely controllable outlets. This makes it possible to save some power by cutting off equipment i rarely use, but still consumes significant power while off (greetings to the server BMCs). The thing I managed to find is a ServerTech CW-8HE-C20 which is old as fuck, but has the last firmware from 2021 and ServerTech support was generous enough to provide me the license keys to unlock smart load shedding for free. It also came with an environmental probe, so now I have in-rack temperature monitoring.

Servers came and went

This year I have burned trough a record amount of server hardware. Around May I got myself a HPE Proliant DL80 Gen9 with 12 drive bays. This was supposed to become my new NAS but has been a disappointment altogether.

The machine has several issues:

• the fancy looking drive caddies with "chips" are cheaply made and flimsy and I did not manage to get any light effects out of those many LEDs on them.
• using this as a Proxmox virtualization host is basically impossible due to the HP P840 RAID controller. So this has limited the usage of the server as a bare-metal TrueNAS or OMV system. Which is a pity as I am already heavily invested in Proxmox. I was thinking about replacing the controller with a LSI one, but there are proprietary looking cables running from the controller to the backplane, so I concluded it would be too much hassle.
• This is a really non-standard server (basically a dumbed down DL380) with components that are really hard to get. So a potential extension for dual-CPU server was basically not possible, due to heatsinks and fans being an unobtanium.
• Also the rails for the server are slightly sticking out from the rack, so I cannot close the back doors.

Anyhow, this will up for sale as soon as I fix the issue with one of the memory slots not being recognized, as I have likely managed to bend a pin in the CPU socket....

On the brighter side, gone is my Dell PowerEdge R430 that I didn't really have a use-case for, and was too power hungry with the dual E5-2680v4 CPUs. The buyer also did not want a power hog, so he wanted only one processor. So I made some experiments and was surprised that with a single-CPU and no drives connected the consumption dropped below 60W.

Unfortunately I did not resist long and had to jump on a deal for another machine - the SuperMicro 6038R-E1CR16L that can hold 16 3.5 drives with 2 additional SFF slots on the back (although SATA only) with the potential to hide some additional drives on the inside. The interesting part of this machine is the chassis, where I may move the contents of my existing SuperMicro NAS. A thing I will probably have to do, as this machine has a surprisingly high power consumption, idling at around 100 Watts without any drives.

But hopefully I am done buying servers for the foreseeable time.

Networking got upgraded too...

I long had the feeling I am growing out my UniFi US-24-250W switch where I constantly had to disconnect something when I wanted to plug in something new. Also all the ports cramped on the right side were annoying when trying to manage the cables.

So what I found is a Brocade ICX-7250-48P that is coming with 8 SFP+ ports. Setting this up was journey, starting with making the proper serial cable (as the mini-USB port on port just looks like mini-USB, otherwise its a plain old RS-232 port) continuing with learning the command line interface. Honestly I was expecting the switch to be a bit more configurable via the WebUI. It was upgradable to the new FastIron 9.0 firmware, which looks good at the first glance, but even after almost 3 years its in a alpha state at best where trying to change any setting via the web is resulting in "not implemented" errors. So I am still thinking about downgrading to version 8.0 (which is recommended by Brocade/Ruckus anyhow) just to have some more options in the Webui. Despite looking like from the early 90s...

This change has again bumped by homelab idle consumption by 50 W and I have lost the switch data from the single pane of glass provided by the UniFi controller... But heck I have paid like 30% of what a 48 port UniFi PoE switch would cost and still have 6 extra SFP+ ports.

ZFS has finally entered the game

This was the single most significant and expensive upgrade my homelab has ever got. I was slowly running out of storage in my MergerFS setup consisting of a bunch of different drives with different speeds and capacities. ZFS was something I wanted to try out for some time, so again I have jumped on a deal of 8x 12TB HGST SAS drives. 6 drives went into 2 striped vdevs with 1 redundant drive in each. 2 drives were supposed to be hot spares, but of them was DoA. Thankfully the seller iuppiter.nl was kind enough to refund the fault drive in warranty.

The ZFS setup was also a bumpy journey, staring off with TrueNAS, that worked nice, but was a bit too limiting in its nature and unwillingness to allow installation of custom apps. So went back to OpenMediaVault, which has its quirks, but can run all my dockers and custom stuff directly on the host. Pool performance has some issues with scrubs running literally for days, but hopefully thats some misconfigured recordsize on some of the pools.

Besides the price, this also managed to add something like 30 W to the power consumption, due to the previously idling and spinned down drives now continuously doing something.

Finally the software stuff...

Software is ever changing and I have trouble keeping track of everything. However here's the biggest additions:

Zabbix is finally monitoring almost everything in my homelab, overloading me Telegram notifications when some device goes down. But with 100 hosts (including smarthome IoT stuff) this was a really overdue update. Definitely less fancy than a InfluxDB/Grafana stack, but works very well and is perfectly documented and supported.

Deploying Gitea was also a significant quality-of-life update, so now I can finally keep proper history of my docker compose and other config files

Also I have moved by backups to Restic (via the Backrest frontend) what is making them infinitely more managable compared to the rsnapshot thingy I was using before.

I have also made the habit of documenting the stuff I am doing in my homelab via notes in Silverbullet. Its very practical and extremely useful but I still wish there would be a proper web version of Obsidian that I really love for work, but for its off-line nature is not suited for my private documentation.

Given with the new ZFS setup I ended up with some excess storage space, I have allocated 10 TBs to a StorJ storagenode that should give me some cashback after some time. Not sure how much it was worth it, as it looks like the ZFS pool is having issues handling the amount of data without a pair of ZFS special devices and its giving me the constant stress of keeping my node up at any costs.

Finally I have made full use of the excellent OpenWebUI as the self-hosted ChatGPT frontend, providing me premium API access and private storage of my chat history. I am also hosting some local LLM models, but basically never using them.

Plans for 2025

For next year I would really like to keep it simple (and cheap).

So the goals are:

• do not spend more than 500 on homelab equipment
• do not increase power consumption (i still need to estabilish a baseline, but I'd like to keep it below 300 W)
• make more posts

So see you around next year...