Kubernetes on Roland Huß

Blog winter is over

Sat, 04 Apr 2026 00:00:00 +0000

The last post on this blog was about Jib, Google’s daemonless Java image builder. That was July 2018. Almost eight years ago. Anybody remember when that was the latest hotness?

Before that, I wrote about Docker when Docker was still exciting and built a Kubernetes cluster on Raspberry Pi 3 nodes when that was still a weekend adventure. I spent way too many words on Jolokia and JMX. 27 posts between 2010 and 2018, then silence. If you’ve been reading tech blogs long enough, you know how that goes.

So what breaks eight years of silence?

Why the silence
#

There’s no dramatic story here, no burnout or life crisis. I just stopped at some point because my writing energy went elsewhere. I co-authored Kubernetes Patterns with Bilgin Ibryam (first edition in 2019, second in 2023). More recently, I finished Generative AI on Kubernetes with Daniele Zonca in 2026.

Writing books is a strange experience. You pour months into a manuscript, and when it ships, you’re proud and drained at the same time. But once the last book was out in March 2026, something shifted. The pressure was gone, the gap between having something to say and sitting down to write it started closing, and I wanted to write shorter, more opinionated pieces again.

What changed
#

The tech world of 2026 looks nothing like 2018. AI-first engineering isn’t a conference buzzword anymore. It’s how I work every day. The way I write code, design systems, and think about developer experience has shifted more in the last year than in the decade before. Last month I scaffolded a complete operator in two days that would have taken two weeks in 2018, and most of my work was writing the specification, not the code.

Context engineering became something I care about deeply. Not in the abstract “prompt engineering” sense, but in the practical “how do you structure specifications so that AI agents produce useful output” sense. I’ve been deep into Spec Driven Development, particularly spec-kit from GitHub. I have opinions about it, even as the whole field is still taking shape.

When you find yourself explaining the same ideas in conversations, Slack threads, and pull request descriptions over and over again, that’s usually a sign you should write them down properly.

What’s coming
#

The topic I keep coming back to is context engineering and Spec Driven Development: how I structure specifications for agentic coding workflows, what works, what fails, and why the “just give it a prompt” approach misses the point. This will probably be a recurring theme, because the field is moving fast and there’s a lot to figure out in public.

Close behind is AgentOps on Kubernetes. Running agentic workloads on Kubernetes and OpenShift is a different beast than classic web services. Agents are unpredictable, long-running, and resource-hungry. They talk to the outside world in ways that make security teams nervous. I’m ramping up on this topic professionally, figuring out how to operate these workloads in a secure and scalable way. Expect posts about the particular demands of AI agents and why your existing Deployment patterns won’t cut it.

Beyond those two, expect posts about AI-first engineering in daily practice (the surprising wins, the things that still don’t work), agentic coding projects and tools, the home K3s cluster that’s been running on five Raspberry Pi 4 nodes for over five years, book-adjacent Kubernetes patterns, and whatever else catches my attention.

On AI and this blog
#

Since AI changed how I work, it naturally changed how I write too. I use AI tools for ideation, polishing, and managing the publishing workflow. The thinking and the opinions stay mine. Every post carries an AI attribution tag at the bottom so you know exactly what role AI played. I apply the same principle I advocate in the Responsible Vibe Coding Guide: use AI as a tool, but own the result. I’ll write more about the process in a future post.

Let’s see
#

I’m not going to promise a posting schedule. That kind of commitment didn’t work last time, and I see no reason to repeat it. But I have more things to write about now than at any point during the old blog’s run. Eight years ago, Jib was the latest hotness. The next post won’t take that long.

AIA HAb CeNc Hin R Claude Opus 4.6 v1.0

Bringing Octobox to OpenShift Online

Sun, 25 Mar 2018 00:00:00 +0000

Octobox is for sure one of my favourite tools in my GitHub centred developer workflow. It is incredible for GitHub notification management which allows me to ignore all the hundreds of GitHub notification emails I get daily.

Octobox is a Ruby-on-Rails application and can be used as SaaS at octobox.io or installed and used separately. Running Octobox in an own account is especially appealing for privacy reasons and for advanced features which are not enabled in the hosted version (like periodic background fetching or more information per notification).

This post shows how Octobox can be ported to the free “starter” tier of OpenShift Online.

Application setup
#

An Octobox installation consists of three parts:

Octobox itself, a Rails application
Redis as an ephemeral cache, used as a session store
Postgresql as the backend database

Naturally, this would lead to three services. However, as I’m not striving for an HA setup and the sake of simplicity, I decided to combine Octobox and Redis in a single pod. Since a combined lifecycle for Octobox and Redis is a reasonable, fair choice, this reduces the number of OpenShift resource objects considerably.

As persistent store for Postgres, we use a plain PersistentVolume which is good enough for our low-footprint database requirements.

Docker Images
#

To get an application onto OpenShift, you first need to package all parts of your application into Docker images which eventually become container during runtime.

There are some restrictions for Docker images to be usable on OpenShift. The most important one is that all containers run under a random UID, which is part of the Unix group root. This restriction has the consequence that all directories and files to which the application process want to write should belong to group root and must be group writable.

Octobox already is distributed as a Docker image and has recently be [updated][octobox-dockerfile-chgrp] to be OpenShift compatible. The Postgres image is directly picked up from an OpenShift provided ImageStream, so there is no issue at all. The Redis Imagee is also already prepared for OpenShift However, when using Redis from this image in an ephemeral mode (so not using persistence) there is a subtle issue which prevents starting the Pod: As the Dockerfile declares a VOLUME and even though in our setup we don’t need it, we have to declare a volume in the Pod definition anyway. Otherwise, you end up with a cryptic error message in the OpenShift console (like can't create volume ...). An emptyDir volume as perfectly good enough for this.

Template
#

To install the application an OpenShift Template has been created. It contains the following objects

DeploymentConfigs for “Octobox with Redis” and “Postgres”
Services for Octobox and Postgres
PersistentVolumeClaim for Postgres

A route for accessing the app is created later on the OpenShift console. Please refer to these installation instructions for more details how to use this templats.

OpenShift Online Starter
#

OpenShift Online Starter is the free tier of OpenShift online which is very useful for learning OpenShift concept and get one’s feet wet. However, it has some quite restrictive resource limitations:

1 GB Memory
1 GB Storage

This budget is good enough for small applications like Octobox, but if you need more horsepower than you can easily upgrade to OpenShift Online Pro.

The challenge is now to distribute the three parts (Octobox, Postgres, Redis) over these 1 GB. As Octobox as rails application is quite a memory hog, we want to dedicate as much memory as possible to it. For Postgres, we do not need much Memory at all, so 50 to 100 MB is good enough. The same for Redis as an initial guess. We can always tune this later if we found out that our initial guess a wrong.

Ok, let’s start with:

875 MB Octobox
50 MB Redis
75 MB Postgres

When trying out these limits, I quickly found out that this doesn’t work. The reason is that OpenShift Online has a minimum size for a container which is 100 MB. Also, you can’t choose requests and limits freely, but there is a fixed ratio of 50% to calculate the request from a given limit (the request specified is always ignored). This fact not only means that you get a Burstable QoS class, but also that you have to specify 200 MB as limit to get at least 100 MB request to exceed the required minimum.

So we end up with:

600 MB Octobox
200 MB Redis
200 MB Postgres

Apparently, this is not optimal, but that’s how it works for OpenShift Online Starter tier (and probably also the Pro Tier). For other OpenShift cluster it, of course, depends on the setup of this specific cluster. We could put Redis and Octobox in the same container, and start two processes in the container. This change would free up another 150 MB for Octobox but is ugly design. So we won’t do it ;-)

tl;dr
#

Porting an application to OpenShift is not difficult. Especially the free OpenShift Online Starter is very appealing for such experiments. The challenges are mostly around creating proper Docker images and getting resource limits right. As a result, you get a decent running and managed installation.

For the full installation instructions, please refer to the OpenShift specific Octobox installation instructions.

RasPi 3 Kubernetes Cluster - An Update

Wed, 05 Apr 2017 00:00:00 +0000

Our Ansible Playbooks for installing Kubernetes on a Raspberry Pi Cluster have been constantly updated and are now using the awesome kubeadm. The update to Kubernetes 1.6. was a bit tricky, though.

Recently I had the luck to meet Mr. @kubernetesonarm Lucas Käldström at the DevOps Gathering where he demoed his multi-arch cluster. That was really impressing. Lucas really squeezes out the maximum what is possible these days with Raspberry Pis and other SOC devices on the Kubernetes platform. Please follow his Workshop on GitHub for a multi-platform setup with ingress controller, persistent volumes, custom API servers and more.

Needless to say that after returning home one of the first task was to update our Ansible playbooks for updating to Kubernetes 1.6 on my RasPi cluster. The goal of these playbooks are a bit different than Lucas workshop setup: Instead of living at the edge, the goal here is to provide an easy, automated and robust way to install a standard Kubernetes installation on a Raspberry Pi 3 cluster. kubeadm is a real great help and makes many things so much easier. However there are still some steps to do in addition.

After following the workshop instructions it turned out soon, that it was probably not the best time for the update. Kubernetes 1.6. has just been released and it turned out that last minute pre-release changes broke kubeadm 1.6.0. Luckily these were fixed quickly with 1.6.1. However the so called self hosted mode of kubeadm broke, too (and is currently still broken in 1.6.1 but should be fixed soon). So the best bet for the moment is to use a standard install (with external processes for api-server et. al).

Also this time I wanted to use Weave instead of Flannel as the overlay network. In turned out that this didn’t worked on my cluster because every of my nodes got the same virtual Mac address assigned by Weave. That’s because this address is calculated based on /etc/machine-id. And guess what. All my nodes had the same machine id 9989a26f06984d6dbadc01770f018e3b. This it what the base Hypriot 1.4.0 system decides to install (in fact it is derived by systemd-machine-id-setup from /var/lib/dbus/machine-id). And every Hypriot installation out there has this very same machine-id ;-) For me it wasn’t surprising, that this happened (well, developing bugs is our daily business ;-), but I was quite puzzled that this hasn’t been a bigger issue yet, because I suspect that especially in cluster setups (may it be Docker Swarm or Kubernetes) at some point the nodes need their unique id. Of course most of the time the IP and hostname is enough. But for a more rigorous UUID /etc/machine-id is normally good fit.

After knowing this and re-creating the UUID on my own (with dbus-uuidgen > /etc/machine-id) everything works smoothly now again, so that I have a base Kubernetes 1.6 cluster with DNS and proper overlay network again. Uff, was quite a mouthful of work :)

You find the installation instructions and the updated playbooks at https://github.com/Project31/ansible-kubernetes-openshift-pi3. If your router is configured properly, it takes not much more than half an hour to setup the full cluster. I did it several times now since last week, always starting afresh with flashing the SD cards. I can confirm that its reproducible and idempotent now ;-)

The next steps are to add persistent volumes with Rook, Træfik as ingress controller and an own internal registry.

Feel free to give it a try and open many issues ;-)

A Raspberry Pi 3 Kubernetes Cluster

Wed, 27 Apr 2016 00:00:00 +0000

Let’s build a Raspberry Pi Cluster running Docker and Kubernetes. There has been already a handful of good recipes, however this howto is a bit different and provides some unique features.

My main motivation for going the Raspberry Pi road for a Kubernetes cluster was that I wanted something fancy for my Kubernetes talk to show, shamelessly stealing the idea from others (kudos to @KurtStam, @saturnism, @ArjenWassink and @kubernetesonarm for the inspiration ;-)

I.e. the following Pi-K8s projects already existed:

kubernetes-installer-rpi : A set up shell scripts and precompiled ARM binaries for running Kubernetes by @KurtStam on top of the Hypriot Docker Image for Raspberry Pi.
Kubernetes on ARM : An opinionated approach by @kubernetesonarm with an own installer for setting up Kubernetes no only for the Pi but also for other ARM based platforms.
K8s on Rpi : Another shell based installer for installing a Kubernetes cluster by @ArjenWassink and @saturnism

When there are already multiple recipes out there, why then trying yet another approach ?

My somewhat selfish goals were:

Using (and learning on the way) Ansible for not only a one-shot installation but also maintainance and upgrades.
Learning myself how to setup a Kubernetes cluster. This setup includes flannel as an overlay network, the SkyDNS extension and soon also a registry. Using Ansible helps me to incrementally add on top of things already installed.
Want to use WiFi for connecting the cluster. See below for the reason.
Get OpenShift Origin running and be able to switch between Ansible and OpenShift via Ansible.
Create a demonstration platform for my favourite development and integration platform fabric8.

As it turns out the whole experience was very enlightening to me. Its one thing to start Kubernetes on a single node within a VM (because multiple VM-based nodes kill soon your machine resourcewise) or having a small bare metal cluster, which blinks red and green and where you can plug wires at will. Not to mention the the geek factor :)

Shopping List
#

Here’s my shopping list for a Raspberry Pi 3 cluster, along with (non-affiliate) links to (German) shops, but I’m sure you can find them elswhere, too.

Amount	Part	Price
4	Raspberry Pi 3	4 * 38 EUR
4	Micro SD Card 32 GB	4 * 11 EUR
1	WLAN Router	22 EUR
4	USB wires	9 EUR
1	Power Supply	30 EUR
1	Case	10 EUR
3	Intermediate Case Plate	3 * 7 EUR

All in all, a 4 node Pi cluster for 288 EUR (as of April 2016). Not so bad.

Some remarks:

Using WiFi for the connection has the big advantage that the Raspberry Pi 3 integrated BCM43438 WiFi chip doesn’t go over USB and saves valuable bandwidth used for IO in general. That way you are able to to get ~ 25 MB/s for disk IO and network traffic, respectively. And also less cables, of course. You can alway plug the power wire for demos, too ;-)
A class 10 Mirco SD is recommended but it doesn’t have to be the fastest on the world as the USB bus only allows around 35 MB/s anyway.

Initial Pi Setup
#

Most of the installation is automated by using Ansible. However the initial setup is a bit more involved. It certainly can be improved (e.g. automatic filesystem expanding of the initial Raspian setup). If you have ideas how to improve this, please open issues and PRs on Project31/ansible-kubernetes-openshift-pi3. Several base distributions has been tried out. It turned out that the most stable setup is based on a stock Raspian. Unfortunately it doesn’t provide a headless WLAN setup as it is possible with the latest Hypriot images, but for the moment it much more stable (I had strange kernel panics and 200% CPU load issues with the Hypriot image for no obvious reasons). Since this is a one time effort, let’s use Raspbian. If you want to try out the Hypriot image, there’s an experimental branch for the Ansible playbooks which can be used with Hypriot. I will retry Hypriot OS for sure some times later.

Download the latest Raspian image and store it as raspbian.zip :

 curl -L https://downloads.raspberrypi.org/raspbian_lite_latest \
 -o raspbian.zip

Install Hypriots’ flash installer script. Follow the directions on the installation page.
Insert you Micro-SD card in your Desktop computer (via an adapter possibly) and run
```
 flash raspbian.zip
```
You will be asked to which device to write. Check this carefully, otherwise you could destroy your Desktop OS if selecting the the wrong device. Typically its something like /dev/disk2 on OS X, but depends on the number of hard drives you have.
Insert the Micro SSD card into your Raspberry Pi and connect it to a monitor and keyboard. Boot up. Login in with pi / raspberry. Then:
```
 raspi-config --expand-rootfs
 vi /etc/wpa_supplicant/wpa_supplicant.conf
```
and then add your WLAN credentials
```
 network={
 ssid="MySSID"
 psk="s3cr3t"
 }
```
Reboot
Repeat step 2. to 5. for each Micro SD card.

Network Setup
#

It is now time to configure your WLAN router. This of course depends on which router you use. The following instructions are based on a TP-Link TL-WR802N which is quite inexepensive but still absolutely ok for our purposes since it sits very close to the cluster and my notebook anyway.

First of all you need to setup the SSID and password. Use the same credentials with which you have configured your images.

My setup is, that I span a private network 192.168.23.0/24 for the Pi cluster which my MacBook also joins via its integrated WiFi.

The addresses I have chosen are :

| 192.168.23.1 | WLAN Router | | 192.168.23.100 | MacBook’s WLAN | | 192.168.23.200 … 192.168.23.203 | Raspberry Pis |

The MacBook is setup for NAT and forwarding from this private network to the internet. This script helps in setting up the forwarding and NAT rules on OS X.

In order to configure your WLAN router you need to connect to it according to its setup instructions. The router is setup in Access Point mode with DHCP enabled. As soon as the MAC of the Pis are known (which you can see as soon as they connect for the first time via WiFi), I configured them to always use the same DHCP lease. For the TL-WR802N this can be done in the configuration section DHCP -> Address Reservation. In the DHCP -> DHCP-Settings the default gateway is set to 192.168.23.100, which my notebook’s WLAN IP.

Startup all nodes, you should be able to ping every node in your cluster. I added n0 … n3 to my notebook’s /etc/hosts pointing to 192.168.23.200 … 192.168.23.203 for convenience.

You should be able to ssh into every Pi with user pi and password raspberry. Also, if you set up the forwarding on your desktop properly you should be able to ping from within the pi to the outside world.

Ansible Playbooks
#

After this initial setup is done, the next step is to initialize the base system with Ansible. You will need Ansible 2 installed on your desktop (e.g. brew install ansible when running on OS X)

Ansible Configuration
#

Checkout the Ansible playbooks:

 git clone https://github.com/Project31/ansible-kubernetes-openshift-pi3.git k8s-pi
 cd k8s-pi

Copy over hosts.example and adapt it to your needs
```
 cp hosts.example hosts
 vi hosts
```
There are three Ansible groups which are refered to in the playbooks:

| pis | All cluster node | n0, n1, n2, n3 | | master | Master node | n0 | | nodes | All nodes which are not master | n1, n2, n3|
Copy over the configuration and adapt it.
```
 cp config.yml.example config.yml
 vi config.yml
```
You should at least put in your WLAN credentials, but you are also free to adapt the other values.

Basic Node Setup
#

If you have already created a cluster with these playbooks and want to start a fresh, please be sure that you cleanup your ~/.ssh/known_hosts from the old host keys. You should be able to ssh into each of the nodes without warnings. Also you must be able to reach the internet from the nodes.

In the next step the basic setup (without Kubernetes) is performed. This is done by

ansible-playbook -k -i hosts setup.yml

When you are prompted for the password, use raspberry. You will probably also need to confirm the SSH authentity for each host with yes.

The following steps will be applied by this command (which may take a bit):

Docker will be installed from the Hypriot repositories
Your public SSH key is copied over to pi’s authenticated_keys and the users password will be taken from config.yml
Some extra tools are installed for your convenience and some benchmarking:
- hdparm
- iperf
- mtr
Hostname is set to the name of the node configured. Also /etc/hosts is setup to contain all nodes with their short names.
A swapfile is enabled (just in case)

With this basic setup you have already a working Docker environment.

Now its time to reboot the whole cluster since some required boot params has been added. Plug the wire.

Kubernetes Setup
#

The final step for a working Kubernetes cluster is to run

ansible-playbook -i hosts kubernetes.yml

This will install one master at n0 and threed additional nodes n1, n2, n3.

The following features are enabled:

etcd, flanneld and kubelet are installed as systemd services on the master
kubelet and flanneld are installed as systemd services on the nodes
Docker is configured to use the Flannel overlay network
kubectl is installed (and an alias k)

If there are some issues when restarting services in the master, don’t worry. However you should best restart the master node n0 when this happens, because when setting up the other nodes the would fail if not all services are running on the master.

After an initial installation it may take a bit until all infrastructure docker images has been loaded. Eventually should be able to use kubectl get nodes from e.g. n0. When this wotks but you see only one node, please reboot the cluster since some services may have not been started on the nodes (plug the cables when n0 is ready).

Install SkyDNS
#

For service discovery via DNS you should finally install the SkyDNS addon, but only when the cluster is running, i.e. the master must be up and listening. For this final step call:

ansible-playbook -i hosts skydns.yml

Wrap Up
#

This has become a rather long recipe. I re-did everything from scratch within 60 minutes, so this could be considered as a lower boundary (because I already did it several times :). The initial setup might be a bit flaky, but should be easy to fix. I’d love to hear your feedback on this, and maybe we get it more stable afterwards. Remember, that’s my first Ansible playbook :)

Now go out, buy and setup your Kubernetes cluster and have fun :-)

Kubernetes on Roland Huß

Blog winter is over

Why the silence #

What changed #

What’s coming #

On AI and this blog #

Let’s see #

Bringing Octobox to OpenShift Online

Application setup #

Docker Images #

Template #

OpenShift Online Starter #

tl;dr #