NOINDEX

Building a Kubernetes Homelab

This Site (late 2019/2020)

I’ve been working for the last 10 years in web/software development, project management, and product development. Being the head of the company for a tiny 4 person firm often meant that I filled in wherever there was a need. With the closure of my company Y-Designs, I felt the need to revisit all of my capabilities and to reflect on my past 10 years.

I learned a lot along the way from my peers and clients alike and I am thankful.

The point of what I’m building here is to prove that my technical capabilities have depth despite how wide a net I seem to cast in my resume. Besides getting parts for the physical hardware, I was able to finish all of this in relative leisure time in under 6 weeks (less than 5 hours a day, not every day, definitely not on sunny days).

The Project

The project is to build a dynamic portfolio site from the hardware up all the way to branding and layouts of my own. While I may have some design input from some friends, everything I’ve done is by my own hands. For the sake of direction and order of operation, I’ll start with the hardware and end with the site development.

Contents

Getting the Right Hardware

Since I had some time and a need to move to “free” hosting, I invested in building a rack-based server. I looked on eBay, few other used server market places and found that they were typically overpriced.

The most eco-conscious and easy to source option was to go to a PC recycler every few days to see if they turn up anything good. As luck would have it, I found an older, but still good R520 dell rack mount server for the grand total of $150.

While the R520 isn’t the most advanced piece of hardware, it’s definitely more than enough once kitted with 2 CPUs (32 threads) and 96GB of RAM. In addition, four 4 terabyte drives were installed as main storage.

One of the trickier parts of this build was the HBA (host bus adapter) board that was included. After some research, it appeared that my best bet was to buy another HBA board. The board would be flashed with IT mode so that it could handle the simpler storage configuration for home and homelab use (none raid, sata pass through configuration).

My house has a gigabit internet connection with a static IP. CenturyLink made me call them 6 times before they would give me a static IP though. At the same time as the server install, I put together an old Dell with pfSense to act as my PPPoE client and also as a firewall. This will come into play as we set up our internal network and Kubernetes.

Server Software and Architecture

The server is meant to serve two purposes:

Because of the secondary requirements and the ease of use, I decided to go with Unraid as the base OS for this server. Proxmox was a close second, but it seemed overkill for the moment. Once the BIOS and a few other Dell firmware were updated, I was able to get everything running quickly. Pro-tip, look for the Dell service tag. That will tell you all that you need to download.

With Unraid, it’s just a quick install to a USB stick then in it goes into the server. Once the server is up and running you’re all good to start setting up your “unraid” disk arrays. I won’t go into full details about the “unraid” array, but it is a JBOD style disk array with a one or two disk failure redundancy. This suits most home users especially considering the giant disk sizes of today’s hard drives (4 terabytes plus is a norm). I’ve installed a few key plugins into Unraid including the community application (app store).

Unraid supports having both VMs and Docker containers. This allows us to create a few generic but important services very quickly.

For the below services, I used the docker containers right on top of Unraid:

In addition to those containers, I’ve ran a few VMs for various purposes (mostly CentOS based):

The docker based services run directly on the server IP. The VM services all run their own kernels and are more isolated; they even have their own IPs assigned statically by pfSense (firewall/router).

Kubernetes with Kubespray

For setting up Kubernetes, I chose to use Kubespray. Kubespray is a set of Ansible scripts designed to help facilitate Kubernetes installations into common Linux distros.

I worked on creating three nearly identical VMs on the Unraid server first. I followed many of the instructions and ideas from this post. The most important part is to create the right size VMs for your needs. This obviously depends on your needs and your availability of hardware, but my master VM has 4GB of RAM and 2 cores while the workers have 2GB of RAM and 2 cores each.

From that point on, there’s a lot of automagic that’s already built into Kubespray. It lets you set up your K8S environment relatively quickly. Their official docs help in describing what all needs to be taken into consideration. If you’ve taken a similar route as me, I would suggest assigning these K8S VMs static IPs within your network range. In my case, it’s done within pfSense and its DHCP service, but that depends on your router.

Once setup, I still didn’t have a true load-balancer on my Kubernetes setup. Most homelabs use NodePorts which is fine, but not if you’re looking to scale things. This is where MetalLB comes in handy along with our pfSense router/firewall.

Since I wanted each service on Kubernetes to have its own internal IP, I turned to BGP. What is BGP? Border Gateway Protocol lets MetalLB and pfSense communicate about a range of IPs they would like to assign and route. Thanks to another wonderful post I was able to figure out some of the required configuration points for both Kubernetes and pfSense quickly.

Please note that use of MetalLB made sense in my setup, but may not be ideal if you’re running everything in the cloud (hence MetalLB = bare-metal load balancer).

With the Kubernetes environment and IP resolution figured out, I could move to application development and deployment.

Private Docker Registry and Quirks

I thought I could move into development and deployment, but I was wrong. An issue arose once I found that some Docker images run fine in normal Docker, but have trouble under Kubernetes after some trial. Because of this, I needed to set up a private Docker registry. Note, my Docker registry lives directly on top of Unraid since I didn’t want to provision a specific volume size against the registry.

Being a security-minded individual (though not a pro), I didn’t want to just leave the registry open for use by anyone on the network. Since I wanted to K.I.S.S. as much as possible, I set up the Docker registry with a htpasswd basic auth. Turns out this isn’t super basic.

  1. Make your own cert for TLS. Without this cert, you can’t do basic auth on Docker registries. Make sure to use a domain name you’re planning to use for the registry should you expose it externally or internally (something like registry.mydomain.local) 
    openssl req -newkey rsa:4096 -nodes -sha256 -keyout certs/domain.key -x509 -days 365 -out certs/domain.crt
  2. Make a bcrypt based htpasswd for your user. 
    htpasswd -Bn registryuser > htpasswd
  3. Copy htpasswd and cert files as part of the Docker volume and ensure that you’re passing environmental vars with their paths. Restart your instance (or if you’re running command line, it might go something like below) 
    docker run -d \
  -p 5000:5000 \
  --restart=always \
  --name registry \
  -v "$(pwd)"/auth:/auth \
  -e "REGISTRY_AUTH=htpasswd" \
  -e "REGISTRY_AUTH_HTPASSWD_REALM=Registry Realm" \
  -e REGISTRY_AUTH_HTPASSWD_PATH=/auth/htpasswd \
  -v "$(pwd)"/certs:/certs \
  -e REGISTRY_HTTP_TLS_CERTIFICATE=/certs/domain.crt \
  -e REGISTRY_HTTP_TLS_KEY=/certs/domain.key \
  registry:2
  4. Either in your router’s DNS or on your “/etc/hosts” file, point the IP of the server to your registry domain (ex: 10.10.10.10 to registry.mydomain.local)
  5. Add your “domain.crt” file as “ca.crt” to your Docker certificates, typically located at: /etc/docker/certs.d (this is in your local Linux machine, not the servers) 
    sudo mkdir -p /etc/docker/certs.d/registry.mydomain.local:5000/
sudo cp domain.crt /etc/docker/certs.d/registry.mydomain.local:5000/ca.crt
sudo systemctl reload docker
  6. Try to login to your registry, type in your username/pass set in the htpasswd file. 
    docker login registry.mydomain.local:5000
    Success!

From here, you can re-tag and push your local builds to the private registry. To integrate this with Kubernetes, you’ll need to follow these instructions.

Running Strapi on K8S - A Headless CMS

For the past few years, I’ve been playing with headless content management systems (CMS). I’ve used WordPress in this capacity before, but find that it is really made more for server-generated HTML. Because I’ve developed GraphQL and REST-based backends in the past, I didn’t want to repeat the efforts that I’ve already done as examples. During my research for a headless CMS, Strapi came up as a good choice at this time.

Docker Image Issues

While the Docker container image worked great on my local machine, it kept on having various issues once it was translated into a Helm chart for Kubernetes.

One of the issues inevitably forced me to rebuild the docker image with some modifications. I’ve updated the code with some insights from other users who also forked strapi-docker. You can check it out here in Github. Most of the changes are in the entrypoint.sh file.

Replicas, Scaling, and Persistent Volume (PV) via NFS

Another issue with Strapi within Kubernetes is that Strapi relies on a directory to store images and other data. If the datastore isn’t shared across multiple instances of the app, it would cause issues when some workers have the right data and others don’t.

To get around this, you can use PV and PVC to assign specific storage to deployments within Kubernetes. Persistent Volumes are storage volume that stays put across restarts and replicas. You can have it be backed by the likes of an AWS EBS store, NFS, or GlusterFS (your pick really). Persistent Volume Claims are the assignments and divisions of those volumes. You could have a 100G volume that gets divided into multiple claims for example.

For my purposes, I decided to use the native Linux NFS share from Unraid as the volume source with limits to the specific IP range of my 3 cluster nodes.

Writing your own Helm Chart

Because everyone has different deployment needs, sometimes it’s just faster to write your own Helm chart. I’ve published a simple NFS PV version of my Helm Chart as a reference. It’s not perfect, but it sure does get the job done that I need. Just a caveat, I’m using an external SQL instance since I already have one running. I presume this would be similar for many folks who would be running RDS or CloudSQL as the relational data store.

Debugging Strapi in Docker and Kubernetes

Unlike debugging server software on your own machine, you have to type in a few commands to see what’s going on within Kubernetes. Here are a few that are pretty basic, but will get you going:

Reading the status of all of the pods:

kubectl get pods

Attaching to the docker container within the pod:

kubectl exec -it strapi-5bc44bc48b-q88k7 -- /bin/bash

Reading logs for the pod:

kubectl logs strapi-5bc44bc48b-q88k7 | less

Reading events (by timestamp):

kubectl get events --sort-by='.metadata.creationTimestamp'

Because of some issues with my initial Strapi setup and NFS mounting, I had a lot of restarts to get this running. The most basic things still count as I did not realize that the base VM for the Kubernetes nodes did not have NFS utilities installed!

Site Structure and Purpose

The purpose of the site is to showcase some of my skills and to talk about who I am. Being on the web, the best thing you can do is to show images and videos as a showcase of technology. After all, photos are worth a thousand words.

I broke it down into four major pages/types: Home, What I do, What I enjoy, and Projects.

This set allows me to show the various roles I’ve played for different projects and also my personal human side of things.

The Basic Site Structure

Below is what you’ll see when you visit the site visually:

I’ve set up three Content Types within Strapi with these fields:

Simple Branding and Visuals

I am not a designer by trade. That said, I think I have an eye for design. All that time I spent with designers must have rubbed off on me at some level.

The Brief

Either way, as I started this project, I knew that I needed to develop a personal brand. My goal wasn’t to focus very hard on this aspect, but just enough to cover my bases.

As with any design project, you have to start with a set of priorities:

After going through a few iterations, I asked around to see what worked and what didn’t, below are some of the iterations and the final logo. I chose League Spartan as the font. I particularly enjoy stronger sans-serif fonts like this and Bebas Neue. That said, I wanted to show impact and squareness. It’s similar in thought to something like Supreme where it’s very simple but very effective.

The Fonts

Since the logo already contained a nice heading font, I decided to keep that for H1 elements. Besides that, I wanted to use an easy to read and flexible font to cover the rest. Knowing that Open Sans has Light, Semi-Bold, and few other in-between options, it’s flexibility made it a no brainer.

React Gatsby and Hybrid Dynamic components

Because I don’t want to physically host the files on my home server, I decided that the best way was to use React Gatsby and to make a mostly static website. That said, some of the dynamic functions still rely on the homelab for its use.

Getting Gatsby and Strapi to communicate

Gatsby can connect with any number of CMSes to aggregate data as long as you do it correctly on the React component side. Gatsby pre-aggregates the data from data sources, then hydrates the data to pages/components. Once those pages are built, they stay as static files until they’re served by your webserver.

In my trial with integrating Strapi, I quickly moved from using the REST-based integration “gatsby-source-strapi” to using the more native GraphQL implementation “gatsby-source-graphql”. One of the advantages is that by using the native GraphQL implementation in Gatsby, you can access GraphQL Fragments within Strapi’s dynamic fields (flexible fields where you can have images and text inter-mixed as blocks). This makes it possible for us to render the corresponding React components a lot easier.

GraphQL Source and Gatsby Image Sharp

I still had to figure out the issue between GraphQL and Gatsby Image Sharp. Gatsby has a great transform plugin for local images that resizes and reformats the images based on the client’s needs. Tying this together with GraphQL source plugin is a bit difficult so I look at “gatsby-plugin-remote-images”. This pulls down the image from a remote source and puts it through the image processor. Typically, this plugin does not work with the native GraphQL source, but I was able to find a PR/fork that works for my current needs. With this figured out, I could finally start web development.

React-Gatsby Web Development

Since I’ve already laid out most of the structure, images, content, and API/GraphQL down first, development was very smooth. React components in general aren’t too difficult in this instance since all you’re doing is placing static content into them from the Gatsby GraphQL sources. For the most part, I used Flexbox to place all things and kept things as simple as possible by using Bootstrap class names. If things got complex, sub-components or wrapper components were quickly created to solve the issue.

Hybrid Dynamic Components - Simple day-by-day Seattle weather app

To showcase that I can work in React beyond Gatsby, I built a simple React-Redux based web app that fetches REST-API results. Weather, daily events, and historical days are displayed in a dashboard-like setting for Seattle. Check it out here: React Redux Daily Seattle Weather

Because some of the APIs didn’t allow requests from a web source (CORS based rules), I also created an API reflector to serve all of the APIs to the React-Redux app.

Previous Post
Client-Side AI with edge-llm