It's been one year since I deployed hydra, my production Kubernetes cluster.

This post is going to be a write-up of the things I did right, and the things I did wrong, warts and all. I'll close off by talking about my plans for the future.

An unhealthy pod is a dead pod

In the old world, if I deployed a virtual machine, or a container; they would become my pet. It was my responsibility to manage, it was my responsibility to treat with care and affection. I gave it a name and I wouldn't dare delete it. In Kubernetes, that's not how things work. The 'closest' [1] analogue to a container is a Pod. How do you restart the pod if the process fails to start properly? You don't.

Pods get removed. They get deleted. They're not pets anymore, not here.

There isn't time to name your pets. They're cattle now. If one cow dies, whatever. If the whole herd does? you've got a problem. This kind of thinking is what I had to switch to when I moved to Kubernetes.  You don't restart pods. You destroy them.

I think the greatest realization of this was when I lost my lovingly crafted dashboards because I deleted a pod.

This has greater implications than just deleting pods. Any changes you want to make permanent? It needs to be mutable either with a PVC or Configmap or whatever. This forces you to think more carefully about what files and settings are important, because if you just mount the entire container as a PVC, you won't be able to reap the benefits of updating. If you don't mount the right files, when the pod gets deleted, you've lost those changes forever. This kind of understanding is made a lot easier with documentation from the developers of the service you're deploying; but sometimes you find that some edge cases (my Nextcloud deployment has some CM mounts in the /etc dir to increase the PHP filesize, for example).

[1] One could probably argue that a container (within a pod) is the closest to a container within docker, but a pod is the lowest 'level' one works within Kubernetes.

Get your secrets right from the start

Secrets, API keys, passwords, all of that. It's sensitive (obviously). Make sure the management of secrets is done in a sane way from day 0.  I made the mistake of deploying stuff using secrets stored in git. While secrets aren't encrypted in Kubernetes[2], it's an incredibly easy method of attack if your secrets are stored in CI.

This is the biggest criticism I can make of my own infrastrucutre. I didn't do secrets properly, and as a result they're scattered about within my hydra repo. Anyone that gets granted access to the repo, gets the keys to the kingdom.

One of the tasks on my TODO list is to remove the secrets entirely; but it's something I should have avoided from the start.

[2] Base64 is not encryption.

Status checking is only as good as the check

Status checking provides a great insight into the health of a cluster, from a user perspective. It's a nice, blackbox way of saying "Is everything running fine?".

The way my network and DNS is designed means that internally my services can function absolutely fine; but externally it's fucked. This makes status checking a little bit difficult, since I can't rely on checking from within Kubernetes.  

To solve this, I use uptime robot; it's not ideal since there's a pretty hefty delay between knowing the status of something, and uptime robot updating.

That being said, having these checks is still useful. Knowing the status of your estate is one of the most important things to have when you're running distributed services.

Use a dedicated OS for the cluster

Currently my nodes use a mixture of OpenSuse Kubic and Fedora. I started with Fedora, and over time i've transitioned to Kubic; due to the ease of use.

Ease of use? What do you mean?

Think about maintaining a virtual machine. Keeping it updated. Making sure that config doesn't change. Making modifications. Now multiply that by 9. It's not fun. Even using tools like Ansible to automate most of it. Especially on version based operating systems that release fairly frequently.

Using a dedicated operating system for the job that's purpose built for Kubernetes has saved a number of headaches.

I don't need to worry about cgroup configuration, repo setup, all of that. It's not a particularly easy task to prepare a node for use in a cluster, and maintaining the OS on top of the cluster itself isn't a particularly fun thing either. Keeping Fedora nodes in the cluster makes sense if your nodes are pets, since you can afford to maintain them. My nodes aren't pets. They're cattle. They're part of the system.

Avoid dependency locks

When building distributed systems, you will have dependencies. Don't make those dependencies circular. I intentionally built my cluster to avoid these circular locks.

Example (albeit not a great one):

You set your controlplane endpoint to be 'api.example.com', and put the IP in /etc/hosts temporarily.
You deploy the cluster, and then deploy the DNS that provides the NS authority for 'example.com'.
You're then relying on the DNS being hosted by Kubernetes, to provide Kubernetes critical function.

Don't do this. It will result in something being cluster-external; but that's a tradeoff that is absolutely worth it. Soft dependencies are easier to deal with, such as deploying a CI system to deploy changes to the cluster itself; as you can still do things manually if it fails.

It never stops

As is the case in a DevOps world, the work never stops. It will never be done. The mental laundry list of things I want to change and fix grows bigger with every day. That's the reality of Kubernetes, and it's one I've learnt over the past year of maintaining a kubernetes cluster in production.

The future

My current stack sits ontop of oVirt; which is fantastic. I've noticed that the development is a bit slow, and a lot of resources has gone into other RH products; so i'm considering the replacements.

The one I'm planning (and currently developing); is a dual stack cluster. This is a Kubernetes in Kubernetes cluster using KubeVirt. Taking over the key points highlighted above, the thing I'm most eager to work with, is secrets. Once the design is more concrete, I'll write it up. Don't think anyone has gotten KinKy before :p

Hello world. Lets talk about honeypots, and the helpful insights they give into the state of botting on the internet.

But first, what is a honeypot?

Honeypots, explained

A honeypot (or honeytrap); is essentially a decoy computer or container that you run to trap and track attackers.  It can be a real machine, or a fake one designed to emulate a real one. Often honeypots are made to look appealing by appearing as though they run exploitable software. The purpose of a honeypot is to gain an insight into what attackers would do, what they would target, and what they would exploit to gain access. This better equips you to understand what methods attackers use to gain access, and enables you to better defend from future attacks.

An example of a honeypot, is the Citrix Honeypot. The Citrix Honeypot aims to emulate a Citrix Gateway that is vulnerable to CVE-2019-19781. This vulnerability would allow an unauthenticated user to perform remote code execution on a Citrix Gateway (among other Citrix products). This is incredibly appealing to attackers, as if you can perform remote code execution, you can compromise firewalls, install cryptocoin miners, release malware and more. It's bad, and the Citrix vulnerability was even worse because an unauthenticated user could do it. That means that anyone could do it.

The Citrix honeypot is useful to people running Citrix systems, as they can safely^[1] run the honeypot without risking their real infrastructure, and with the data gathered (stuff like attacker's IP addresses), they can strengthen the security of their real infrastructure by IP banning the attackers. This might not stop them in their tracks, but it would certainly slow them down.

The majority of attacks performed on the internet aren't done by hackers in some dimly lit basement somewhere. Most of them are done by automated bots, scanning the internet for exposed ports and running scripts to break in. These kinds of attacks are frequent, and annoying.

Honeypots, especially when used alongside monitoring and automated banning software; provide an excellent way to filter out 90% of these automated attacks; significantly reducing the attacks on your real systems.

[1] Honeypots are not always safe. Be aware that letting anyone access a system, real or not, is a risk.

Enter Tpot

Because I am a nerd and I love seeing bots attempt to break in, I set up a honeypot using the open source project Tpot. Tpot is special, in that it is a preconfigured collection of honeypots listening on a number of different ports to simulate a range of different services. This casts a very wide net to catch all manner of bots.

Tpot utilizes the sandboxing capabilities of docker to make it easy to upgrade, and safer^[1] than running things bare metal. The best thing about tpot is that it aggregates the data from all the running services and presents the data in various forms using kibana.

To get this running, I created a VM inside my oVirt cluster and installed tpot using the ISO. I didn't want to expose my residential IP address to the internet, so instead I launched a VPS on DigitalOcean running VyOS, and setup a wireguard link on my firewalled netork. All ports are then forwarded from the VPS to the virtual machine, exposing it directly to the internet. This gives the highest opportunity for bots and attackers to fall into my trap.

I've been running this publicly for a few weeks now and not noticed any issues so far, so lets hope for the best. Now, lets discuss the things learnt from running a public honeypot.

Insights Gained

Lets take a look at the statistics. Over a 15 day period, there were a total of 2,768,090 Attacks from 7,423 different IP addresses. That's Two attacks, per second, of every hour, of every day.

Samba

The most popular honeypot by far, was Dionaea. This honeypot aims to emulate the popular network file sharing^[2] protocol Samba (SMB), among other things. 95% of the attacks on Dionaea were targeting this protocol.

Why is the SMB protocol so popular? Well, it's tightly integrated into Windows operating systems, and has been a core component for a very long time. There are a huge number of security vulnerabilities in different versions of SMB; and it's enticing to attackers, as these vulnerabilities could easily grant an attacker with root (administrator) level access to a windows machine.

A notable example of SMB being exploited, is the famous WannaCry series of attacks. These attacks exploited a vulnerability in the SMB protocol, which allowed an unauthenticated user to execute code on a windows machine. This code would infect the machine with the worm, and spread to other machines on the networks. Did I mention it encrypted all the files on the host, demanding a ransom for the files to be decrypted? It's a trivial attack, and an easy payday for cybercriminals looking to get some bitcoin. That's probably why it's so popular.

My suggestion? Don't ever, ever expose SMB to the internet.

[2] SMB covers more than just filesharing; but that's beyond the scope of this article.

SSH & Telnet

Next up, is Cowrie. This honeypot aims to emulate both SSH and Telnet. Both of these are incredibly popular protocols, used for establishing a commandline shell over the internet.

I won't be addressing Telnet much, as it's not as popular nowadays; and it's advised to just use SSH instead^[3]

SSH is one of the most well established tools for initiating a remote connection over the internet. It's also fairly easy to misconfigure to make it vulnerable. The problem with misconfigured SSH servers, is that if you get in; you can do a lot. Often you need to bruteforce passwords to get in, but once you are, game over.

The Cowrie honeypot was dealt over 290,430 attacks over the 15 day observation period. The most common attack being on the SSH implementation. admin and root were the most common usernames attempted; with 16,000 and 10,200 attacks respectively. The most common passwords were 1234, admin, root, user and support. Basic passwords really.

What does this mean for a server operator though?

It means, that when you have SSH exposed to the internet; you need to be careful. It won't take long for bots to begin attacking you, and you cannot use easy to guess passwords if you care about that machine not being compromised.

Another alternative to this, and one I highly suggest, is to forgo passwords altogether and switch to key based authentication. This will stop 99% of attacks outright.

[3] In the majority of cases, ssh is suggested. Telnet is still used for debugging purposes due to it's simplicity.

There are a more honeypots to cover than just these two, but I don't want to make this post too long. If there's significant interest I'll do some more writeups covering some of the more unique honeypots.

Now what?

Now, I have a fairly large collection of data (50GB+ and growing). What can I do with it?

For a start I can feed the IP addresses collected from tpot, and send them to my frontline loadbalancers to drop connections from attackers. This is a work in progress, but I will be following on from this post when it is implemented.

In addition, the data collected from tpot is automatically submitted to sicherheitstacho. The aim of this project is to build a realtime visualization of attacks going on around the world.

I've also made the data public. You can take a look at the dashboards here. Give it a play! There's a lot that I didn't cover here.

Thanks.

(Alternative title: The importance of infrastructure as code)

Hello everyone. I hope you're doing well. I'm writing this blog post to share with you a recent adventure I had, and something that helped reinforce my belief that you should aim to have everything defined as code (where possible).

Kubernetes is the hot new thing at the moment, and I am currently migrating my infrastructure off of docker and onto this behemoth of a system for a multitude of reasons. They're the typical K8S (Kubernetes) reasons, so I won't repeat them.

One of the most important to me though was abstraction. The abstraction that is given to the end-user makes things a lot easier to deploy, manage, upgrade and more. On the downside, when you don't do abstraction right, it can be a major kick in the teeth when things come crashing down.

I very recently lost roughly a week's worth of work in the space of 4 seconds. This blog post is going to discuss the mistakes I made, and hopefully, you can learn a few things from it.

The Seven days.

To give a bit of context, I've been running a mix of Kubernetes and Openshift clusters on my at-home server farm. My first (long term) deployment was K8s, the second was Openshift origin 4.4, and the most recent has been K8s. I call this iteration Apollo.

Apollo is special. It's been the cluster where I try to define my Kubernetes infrastructure as code and make monitoring and tracing a core feature in the cluster.

To manage the infrastructure as code (IaC), I was using a mixture of Helm and Kustomize. Helm allows for quick wins while deploying infrastructure as code, as you can deploy parameterized infrastructure, only changing the parameters as needed.

Kustomize requires you to deploy your entire infrastructure as code. No ifs, no buts. You want a secret, you create the YAML.

My first mistake was using a mixture of configuration management tools. It leads to fragmentation of my codebase, and as a result, maintaining quality was a little harder. When you are aiming to maintain and improve quality, try to stick to a minimal codebase. This scales with team size, so find whatever is appropriate for your team and stick with it.

Once the cluster was lovingly deployed with Ansible, I began investigating how I could monitor the state of the cluster. Having a view over the state of your environment is paramount to quality, and so any QA should push for good dashboarding and monitoring as a core component. With monitoring, you understand the health of your environment better. You recognise the weaknesses, the bottlenecks, the defects.

Dashboards give you an understanding of the state of the system as a whole and give you visibility into where you need to test.

Within the K8S realm, Prometheus is one of the most popular monitoring tools at the moment. Combine that with a graphing tool like Grafana, and you've got a killer combo.

I used Helm to deploy the Prometheus operator, which deployed Prometheus, Grafana and a few other components into your cluster. I saved the variables I used to deploy initially so that I could redeploy the infrastructure again, and again, and get repeatable results.

The Prometheus operator configures some Grafana dashboards by default, giving some great insight into the health of a Kubernetes cluster.

Once there, I logged into Grafana and began building some extra dashboards by hand. I wrote one to monitor the health of my Hypervisor cluster.

From there, I continued building and wrote one to Monitor the heatlh of my FreeNAS system.

These manual changes provided me with even greater visibility of the health of my entire system, not just Kubernetes. I began to understand some of the defects within my system in a much better way.

One example of this was learning that I had a high level of queued IO requests on FreeNAS. This was causing performance issues and as a result, was causing timeouts on components reliant on FreeNAS. I had discovered a defect through the use of monitoring, was able to trace the cause back to queued IO, and fix it with creating a cache device.

Making the manual changes and creating dashboards proved to be undoubtedly useful. Although, with these manual changes lies my next mistake. Manual changes. I made the mistake of manually creating dashboards onto an automatic deployment. If I were to redeploy the Prometheus operator, I would lose all of these manual changes, and my dashboards.

I spent about a week creating these dashboards, working into the evening to get them configured to how I liked them.

I defined most of my infrastructure as code. Not all of it. (Foreshadowing a little bit here).

Four seconds.

Four seconds is all it took to bring it all crashing down.

I had built up my dashboards to a point where I wanted to experiment with alerting me when issues are found. Stuff like CPU temperatures spiking above 80°C for more than a few minutes. I want to be able to act quickly when an issue is found and resolve it. To do this, I wanted to configure Grafana to send me an email when an alert is triggered.

To do this, I went to my private repo, modified the helm values to set an email server and its credentials, and updated the deployment. I had assumed that the dashboards were saved into backing storage (like I would have done with Kustomize), and so updating the deployment would just load the dashboards again once it rolls out.

I pressed enter and 4 seconds later, I was notified that the deployment had completed successfully. I opened up the Grafana dashboard and was presented with a login screen.

Odd. I don't usually need to login to Grafana as I had a session token.

I logged in and noticed that the theme had changed back to the default (dark mode).

This is where my mind begins to panic.

I take a look at the dashboard list and realise they're all there; except for the ones I had created myself. The 10 minutes setting up the operator was there, but the 7 days making my dashboards were gone. S**t.

Unbeknownst to me, Helm had redeployed the entire operator. Meaning it wiped the slate clean and treated it like it was a completely new deployment. I had misunderstood the helm chart and assumed that it saved the configuration into storage, resulting in an iterative deployment.

This is my third mistake. Using tools I do not understand properly. Whether this is a misunderstanding of Helm, Kubernetes or something else; there is still a misunderstanding made.

The takeaway from all this.

So. I made a few mistakes. Let's cover them in a bit more depth so that you can recognise them in your team, and prevent them from happening.

Multiple tools for the same purpose

The first mistake I made was the use of multiple tools to perform the same task. While I'm not saying you should use one tool for everything, you should not aim to use multiple tools that do the same thing. An example of this would be using Ansible and Puppet. They are both configuration management tools. Unless you are doing something highly specialised that requires the use of multiple tools, try to stick to one. This helps prevent knowledge of one leaking into the other and causing confusion and mistakes. In my case, I had assumed that the helm chart created storage for the Graphana dashboards (allowing them to persist) like I would have done for Kustomize.

90% IaC is not acceptable.

The next mistake was manual changes ontop of automated changes. I had defined 90% of my infrastructure as code, but the remaining 10% was what mattered the most.

The 10% in my case, was the custom dashboards I had created. For others, 10% could be code pipelines to deploy code. This kind of infrastructure is often easy to create, and incredibly hard to spot. Sometimes you make assumptions that the remaining 10% is not that important, but you won't realise that until it's gone. I'm not going to claim I'm a master at this by any stretch, but I'll try and give some tips on recognising it.

The first steps to recognising the hidden 10% are to deploy your infrastructure into a test environment with only the parts that are defined as code and compare it to what is actually deployed. This will give you a start into recognising what work needs to be done, and do not stop until you reach 100% and can prove it is 100%.

For some cases, 100% is not possible, and that can be ok. Mitigate against it by documenting the manual changes needed, and make sure that you can prove it.

Understand the tools you are using

This links back to my first mistake somewhat. But it is important to understand the infrastructure tools you are using. I'm not saying that you need to be the all-seeing guru of all things helm, but you should at least build up an understanding that allows you to do the most common tasks without needing to consult a guide. If the tool you are working with does something you did not expect, investigate why that happened and learn from it. Tools are not people, they will always do what you tell them to do. Do not sit idly and just think "oh nothing will come of that", because it can.

Next steps

Where do I go from here? I'm going to drop helm. It's great for some quick and small wins, but I do not want to be in a position where I am not in control of what I am deploying, and cause an issue like this again.
I'm probably going to wait for Openshift origin 4.4 to release (officially), and give it another go. Maybe I'll call this one Skylab. All I know is that from here I can only learn, and I hope you learnt something from this adventure too.

Hello everyone.

I hope you had a good holiday season.

I took time off over the holiday to redeploy my infrastructure to make it more automated, more reliable and to be more organised.

This multi-part blog post will cover the entire journey, from planning to execution and finally, the firefighting.

TLDR: I redeployed my infrastructure and documented it. Link here

Before we begin, I want to cover the reasons why I did all this.

My infrastructure in 2019 was a single Desktop computer. Through the year it grew from that to 1 server, then to two, then to 5.

While I had done my best to manage them as separate entities, my infrastructure had become large enough that I should begin treating them the way an enterprise would. Additionally, the way my infrastructure had grown had resulted in many issues arising from the gradual expansion.

Drift

As my infrastructure slowly grew from one server to two, then to five; they began to drift heavily. To me, drift is the slight changes in configuration between machines that make them incompatible with one another when attempting to automate tasks.

One such example of drift is filesystem layout relating to Docker.

On machine A, the mount points for docker volumes might be /srv/docker. On machine B, this would be /opt/docker. While this wasn't an issue when managing 2 or 3 machines, it became an issue when I wanted to automate the deployment of docker images onto these machines using ansible. While I could have hardcoded the values to point to the correct locations, it would have resulted in much unnecessary complexity in my ansible scripts.

This kind of drift was present all over my machines, from sysctl variables to daemon configurations.

Drift happens because of a lack of proper change control and planning, and in its infancy, I didn't feel the need to enforce either.

Bloat & Complexity

Bloat became a problem for my hypervisors because I wasn't treating them as hypervisors; I was treating them as general-purpose do all servers.

One served the purpose of a DNS server and a web server anda build server. Each of these purposes should have been segregated into a virtual machine; so that in the event that I need to shut the hypervisor down, the virtual machine could be moved onto a different hypervisor.

In hindsight, I saw it as easier to install the necessary packages and be on my way; instead of setting up a virtual machine and then the packages. This approach lacked the understanding that I would need to maintain an additional component on top of the hypervisor; rather than maintain a component next to the hypervisor. The system is a lot less complex when it doesn't have 20 other services running with it.

This bloat became a big problem when I had to deal with a cascading failure of one service, causing the hypervisor to enter emergency mode, resulting in the entire system being unusable until I fixed the issue. If I had put the service into a virtual machine, then I wouldn't have been in that mess.

Bloat also added to the total number of packages installed, and thus, updates also became more of an issue as the hypervisors grew in complexity.

Bloat happens when systems become too monolithic, and from a failure to segregate services properly.

(a lack of) Abstraction

This is somewhat related to my previous two points, but it's different enough that I feel the need to point it out. In my infrastructure, I had failed to abstract my services properly. This failure to abstract meant that (for example), my NFS server was hydrogen.srv.oxide.one, not nfs1.nix.oxide.one.

In an ideal world, I shouldn't have to care that NFS is served by hydrogen; I can point my services to nfs1, and I'll be fine.

While this isn’t so much of an issue, it does become annoying when attempting to keep track of my infrastructure and keep things running smoothly.

Lack of repeat-ability

One of the problems that I realised I had was that my hypervisors were not repeatable. I had failed to document how to set up the machines, along with any fixes or configuration changes I had done to make that machine into what it was. Some configuration changes were useful and needed, but others were not.

What this means in practice is that if I suddenly lost the OS on the hypervisor, I would be lost when attempting to rebuild it. I had initially built it by googling my way forward, but this was not sustainable.

The goal

With these issues in mind, I concluded that it would generally be easier to redeploy my infrastructure, rather than retrofit any fixes. I wanted this redeployment to be the lastone I need to do. To make sure that the redeployment was the last, I need to make sure that the process was Documented and Automated, where possible.

I gave myself some goals to be able to mark this infrastructure deployment as ‘successful’. They are as follows:

• Be Repeatable
• Do not treat any machines differently.
• Keep the hypervisors as simple as possible.

First, I wanted my infrastructure to be repeatable. If I wipe a hypervisor, take a fresh install of the OS and make the changes defined in the Documentation, I should end up in the same place as I was before. If not, then I need better documentation. Ensuring that this is kept, I can also eliminate drift.

Second, I should not treat my machines differently; unless there is a justifiable reason to do so. This will help to eliminate drift and complexity.

Third, keep the hypervisors as simple as possible. If something doesn't need to run on the hypervisor, don't run it. Ensure that they do not depend on each other so that they can run independently. Anything that creates a cross-machine dependency should have checks in place to ensure that cascading failures do not happen. This will eliminate my lack of abstraction and bloat.

So, I had my issues; I had my goal. Now to plan.

The plan

I began to document everything about my machines. I had little visibility of what my infrastructure actually was. If I wanted to redeploy, I should at least document the mistakes I made so I don’t make them again.This task alone took up the first week of my holiday, but it was worth it.

Having a good scope of everything in my domain gave me clarity on how much drift had been going on, both on the hardware scale and software scale.  One of my hypervisors had over 100GB of ram, yet the other had 32. On the software scale, one of my hypervisors was setup and synchronised with LDAP; yet the other was not.

It was impressive that everything was running as smoothly as it was, in all honesty.

Once I began building out the documentation; I began to have a nagging thought in my head:

Am I creating a spec, or am I creating documentation?

A spec states what the requirements are for a system. It states how a system should function and behave, whereas documentation states how the system currently functions. While it was in essence Documentation, it felt a lot like a spec, and so I treated it as such.

For each component I wrote about, I determined whether it was useful to the end goal. If it was, I documented it, and it became part of the spec. If the component wasn't relevant to my end goal, and I couldn't justify the component existing, then it wouldn't become part of the spec.

One such example of this is the use of Nvidia drivers in the kernel. At some point in my infra, I had installed a GPU into each server so that it could perform hardware encoding of video. This resulted in the Nvidia kernel drivers being installed.

If I were treating this as Documentation, I would point this out and include instructions for how to set this up on a new hypervisor.

I determined that I shouldn’t really use out of tree kernel drivers if I wanted to ensure stability; so it didn’t become part of the spec. This kind of approach lead to me writing a hybrid Spec/Doc sheet; that would take the best parts of my old infra and build a rock-solid foundation for future expansion.

Once redeployed, the spec will become the documentation, and all will be well.

I call this infrastructure oxide one. It’s currently at version 1, but will be updated well into the future as it is now the single source of truth for all of my machines.

The next blog post will cover how I got around to turning the spec into actions using Ansible.

The wiki is available here

(after you fulfill the prereqs first)

Hello everyone!

In this blog post I'm going to be doing a write-up on creating an Ansible script to deploy a Kubernetes cluster.

The Why

I'm trying to learn Kubernetes, and I tend to learn by breaking things.

I don't want to spend a significant amount of time setting up individual virtual machines, installing the packages and then setting the cluster up only to break something and tear it down again. Even with the help of the script I wrote in my previous blog post it'd still take a significant amount of time to get back to a usable cluster.

I wanted to build something that was able to produce repeatable builds, while still not taking out a sizeable part of my day waiting for the cluster to be ready.

There are already some fantastic projects out there that handle deploying and provisioning a cluster, such as Kubespray or Kops; but it feels like they're more suited for 'actual' cloud deployments rather than my at home setup. They also add some complexity that I don't feel comfortable trying to diagnose if things go wrong.

Additionally, by automating the task I am able to boost my automation skills (which is something I love to do).

The How

The scripts are made up of two Ansible playbooks that handle different things respectively. There is the 'init' playbook, and the deployment playbook.

'init' playbook involves downloading a fedora cloud image, installing the required dependencies onto that cloud image (such as Docker, Kubeadm and Container Networking plugins) and finally packaging it up into a 'base' image. This ideally should only need to be run once and copied over to your libvirt hosts.

The deployment playbook is concerned with taking that base image, and setting up the cluster using that image as a base to build the cluster from.

I've done it this way for a few reasons; the first being that it's a slow playbook. On each run it downloads a new fedora cloud image, installs and updates all packages, then packages it up ready for use. Overall taking about 4 minutes to complete.

The second reason is that I don't want to have to download the same packages on each node multiple times since the 'base' image has all the packages installed already. I want to reduce the amount of unneeded repetition as much as possible to make deployment fast. I feel that downloading packages several times onto what is realistically the 'same' machine, is unneeded.

The third reason being that doing it this way allows me to base virtual machines off of a single base image using QEMU's backing file ability. This gives the benefit of being able to save a significant amount of space, which can be in the realm of 50GB depending on cluster size.

Overview of deployment

On the deployment side of things, the script does essentially the same things as my provisioning script; but I've rewritten it to run all commands in parallel instead of one at a time.

Once the virtual machines are ready, the script logs into the first master and runs kubeadm init; before copying the join token over and running it on each of the nodes.

Finally, it copies the kubeconf over to /tmp/kubeconf.config and applies both Flannel and MetalLB as those are pretty much required dependancies for an 'at home' cluster.

And thats it! You've got a cluster deployed and ready to go.

Time to break stuff!

Challenges + What I learnt

I ran into a few challenges, one of which being that Fedora 31 refused to work with both ContainerD and Cri-o; meaning I was stuck with Docker for the time being. The Cri-o issue resulted from it not accepting that fedora 31 is a thing and reported it as an unknown Operating System.

The other challenge I had was rewriting the script to run in parallel, as I had initially written it to run one at a time. It proved to be incredibly useful to; as it now takes less time to deploy 6 virtual machines than it would have been to deploy one!

Running all the scripts in parallel also taught me some best practices in expansible. I'm now more confident using roles, facts and playbooks in Ansible!

Conclusion

I hope you enjoyed reading this writeup. Source code of this playbook is available here, and I will be adding a README so you can run this at home :)

I almost forgot, the gif. Enjoy :p

Traditional Virtual Machine (VM) deployments are usually a pain to deal with on libvirt.

You open up virt-manager, click through the menus, select how many vCPUs you want, the memory, the storage and the ISO you install from.

Once that's done, you then need to actually install the operating system onto your VM. This is a slow process, and I imagine large cloud providers aren't secretly doing this in the background when you are spinning up instances.

Having to do all these steps manually makes deployment velocity slow, and increases Toil. It also limits my avenues of attack when looking at ways I can automate my work. If I have to manually click through 'next' several times before getting to a usable machine, there's not much point in me automating the rest of the stack... is there?

In this blog post, I take a look at different ways of automating the VM provisioning process, alongside eliminating a lot of the repetitive work involved with setting up a VM.

Part 1: Approaches

PXEBooting

PXE booting, commonly referred to as 'Pixie booting' is a method of sending a boot image over the network to a target, and booting an image from that. Dating back to 1985, PXEbooting is well established and just about every machine with network capabilities can do it; from Rasberry Pis to Mainframes.

While PXEbooting in and of itself is not used to provision virtual machines as all it really does is send a bootloader to a machine over the network; it can be used in combination with several different tools to make it an effective method to do Virtual (and physical) machine installs.

Some of those tools are:

• Foreman
• Cobbler
• MaaS Project
• FOG Project
• Kickstart files (for RPM based distros)
• Preseed files (for APT baed distros)

Kickstart and Preseed files are less PXE Solutions and more predefined 'defaults'; but they are used commonly with PXE to do unattended installs of operating systems.

Foreman, Cobbler, MaaS and FOG are PXE solutions that all offer a good experience all around; and I've given the former 3 a try before. Foreman is useful if you're interested in managing the entire lifecycle of a machine, as it also installs puppet onto the target machine.

While they all have their respective benefits; PXE Provisioning solutions all have a common downside. Time.

When you provision a virtual machine using PXE, it has to download the base operating system from a mirror, and install all the packages. Even on the fastest of internet connections, this will take time.

When I was using foreman, it took about 10 minutes from clicking 'done' to having a system ready for use.

How is it that Cloud services can have a machine ready for me in less than a minute? Uber fast mirrors? No. If you want a Cloud-like experience, you need a Cloud-like solution.

Cloud-init

Cloud-init was born out of the idea that instead of re-downloading and installing the operating system loads of different times; why not just have one 'base' image with the base OS installed and work from that?

What cloud-init does, is apply a set configuration changes you want to make to that base OS to get it to the state you want. You're always going to need to install the base operating system anyway, so why not start from when you actually make changes that make it different from any other system.

How does it work though?

On any given OS with cloud-init installed and enabled, it will check to see if an additional ISO is mounted to the Virtual Machine. In that ISO it will check for two files, user-data and meta-data.

#cloud-config
users:
  - name: okami
    gecos: okami
    lock-passwd: false
    sudo: ALL=(ALL) NOPASSWD:ALL
    ssh_authorized_keys:
      - ssh-rsa AAA...6eNKCKYJ8iXuHUpZJ7EOesnCR9 okami@pegasus
runcmd:
  - sed -i 's/SELINUX=enforcing/SELINUX=disabled/' /etc/selinux/config

timezone: Europe/London
locale: en_GB.UTF-8

write_files:
  - path: /etc/yum.repos.d/kubernetes.repo
    content: |
      [kubernetes]
      name=Kubernetes
      baseurl=https://packages.cloud.google.com/yum/repos/kubernetes-el7-x86_64
      enabled=1
      gpgcheck=1
      repo_gpgcheck=1
      gpgkey=https://packages.cloud.google.com/yum/doc/yum-key.gpg https://packages.cloud.google.com/yum/doc/rpm-package-key.gpg

  - path: /etc/sysctl.d/kubernetes.conf
    content: |
      net.bridge.bridge-nf-call-ip6tables = 1
      net.bridge.bridge-nf-call-iptables = 1
      net.ipv4.ip_forward = 1

  - path: /etc/modules-load.d/br_netfilter.conf
    content: |
      br_netfilter

power_state:
  mode: reboot
  message: Bye Bye
  timeout: 30
  condition: True

runcmd:
  - touch /etc/cloud/cloud-init.disabled  

user-data is the file which tells cloud-init what you want done to the operating system. Some examples of things you can put into a user-data file are as follows:

• Adding users (and authorized SSH keys)
• Setting the Locale
• Adding Repos
• Installing Packages
• Updating the System
• Running arbitrary commands

Feel free to check out the documentation on cloud-init as there is a lot more you can do with it, than with what I just listed.

The meta-data file is where you set the machine's hostname. There are other things you can put in meta-data but for now that's all you need to know.

This allows you to create customized images, and have them ready in only a few minutes. From what I understand, this is what cloud providers use to spin up instances, as it allows you to have a very short time-to-live.

Using Cloud-init ties in very well with the use of Cloud images from Distribution vendors. A Cloud image is the 'base' OS I mentioned earlier. Typically only a few hundred MB in size; and on startup resize themselves to cover the size of the disk you give it. These cloud-images have already got cloud-init installed and enabled so you're ready to go very quickly.

You might be thinking "that's great and all, but surely I would need to generate the cloud-init ISOs manually, right?"

Well, sorta? There's another weapon up my sleeve that handles all of that for you.

Part 2. Ansible

Ansible is a tool developed by Red Hat, that is essentially clever shell scripting on steroids. I use it heavily in my lab to automate processes I feel need automating, and since my adoption of Ansible; I cannot praise the tool enough.

I took an Ansible script I found on github a while back (and cannot find the source [sorry]); and modified it fairly heavily to suit my needs.

I'll give you a high level overview of what it does.

1. Copy the base disk image from my Rsync share to a local location, renaming it {FQDN}.qcow2
2. Generates the hostname from one I give it on the command line and chucks it into meta-data
3. Generates a cloud-init ISO using the machine specific hostname, and a base user-data file
4. Resizes the new disk image from 200MB to 10GB
5. Runs virt-install, importing the new disk image and attaching the cloud-init ISO
6. Ejects the cloud-init ISO
7. Deletes the cloud-init ISO and meta-data file (no longer needed)

This is all done from a single command.

ansible-playbook -i inventory.ini image_provisioner.yml --extra-vars="runon=blackbird host_name=shuttlefish"

After a minute or two, I can see the hostname shuttlefish.vm.oxide.one has been registered to my DNS server, and I can ssh into the machine.

This has dramatically increased the TTL for my virtual machines, and reduced the amount of Toil relating to VM provisioning to almost none.

This script can very easily be used in an automated setting, which I plan on doing soon (keep watch of this blog)

Downsides of Cloud-init + Ansible

There are downsides to taking the cloud-init approach. The biggest being that you can't use this method to provision physical machines.

If that's what you're after, you're better off going for a PXE approach. Forman is pretty damn good for this.

Conclusion

Ansible is awesome; and when used in addition to tools like cloud-init, I am able to create Cloud-like experiences at home.

I hope you enjoyed this exploration into how I do my deployments. I mostly wrote this so I can write more blog posts that link back to this, so I hope you'll forgive if it lacks substance.

Thanks,

Okami.

ansible-playbooks

ansible-playbooks

GitDashokami

Hello world. It's currently 08:30 on a Tuesday; and I'm sat freezing on the floor in an overcrowded train to London. This is my new blog. I'm trying to get better at writing about my projects and I figured a blog was a decent-ish way of doing that.

Lets begin by talking about how I rebuilt my homepage.

For me, a homepage is a place to advertise yourself. It's almost a portal into your digital identity, and although my previous homepage was alright to showcase myself; I wanted something a little more unique and interactive.

I wanted to build a site that allowed me to really show off. One of the biggest examples of a site that shows off is a site called doqk.ml. It's got that cool ASCII art, hacker in the 90s type feel. I guess that's what I wanted to go for with the original okami.dev, that I never quite pulled off.

I remember that the feeling I wanted to give with the original okami.dev was a feeling that you're in a terminal. That's why I went for the console font, the redhat prompt at the top and README.md type layout.

Terminals excite me. To me a terminal is one of the best tools in anyone's arsenal. I wanted to build something that almost felt like you're in one, and in a way that felt realistic. Dropping the user into a terminal would also give them a peer into who I am in a much more interactive way than just "here's who I am". I could build cool little Easter eggs and custom functions to print info about me.

I figured that if I'm redesigning okami.dev I should really try harder to make the user feel that they're in a terminal than I did originally.

With this in mind; I began planning out how I was going to present a 'terminal' to the end user. There are a few approaches to doing this but in the end I went with displaying a terminal using xterm.js and connecting it to some form of back-end.

I initially went along with benjojo's use of virtual machines to build the back-end, but I I got about 4 hours into compiling Linux kernels before I realized that any edits I would make to the site would require a recompile.

I figured I could probably do it a lot easier and a lot faster with Docker.

Docker

Docker seemed like a good choice to do this kind of task, but giving people shell access to a Docker container is a bit of a security nightmare; especially given that they'd be able to exploit it fairly easily.

I've been using Docker as the workhorse for a lot of my infrastructure for quite some time. I've enjoyed using it given how much flexibility it gives me over Virtual Machines. Automating, testing and deploying is made significantly easier.

If I wanted to go with docker I'd need to put a lot of effort into hardening the image.

Implementation and Security

So, we have the front-end and the back-end. Lets build.

GoTTY ended up being what I used to connect to the Docker image as it allows you to spawn a docker container when a user connects, and eventually kills it once they disconnect.

Once I got some really basic image setup through GoTTY I shared a link with some friends to find flaws. (There were a lot).

After the first few compliments I told them to find ways to exploit this. Within a few minutes, I guess I got my wish.

Obviously, I had some work to do before releasing this.

First on the list was removing all network access. I did this before the initial 'alpha' release as I realized that giving people network access is megadumb regardless of how much I trust them. This is done fairly easily by spawning the container with -net none.

The next thing I learnt was to restrict how the user interacts with the filesystem. After a few minutes of playing, someone managed to fill up the filesystem with random data. This was done with dd if=/dev/urandom of=/filename.txt. I fixed that by removing removing dd at build time, and mounting the container's filesystem as --read-only.

Third; was limiting memory and cpu limits resources. This is fairly easy to do by running the container with the following. -m 32m --cpus=0.5. What this does is limit the container to 32mb of ram and 0.5 of the CPU max. This prevents stuff like fork bombs as with this the container will now just crash and drop the connection if such a heinous command is entered.

Once those were out of the way, the immediate flaws were fixed. People started to get creative.

To limit how long people had to play with the container per session, I made the container timeout after 10 minutes and close. This meant people had less time to break stuff before their container would die and they'd need to get a new shell.

Someone figured out you could SIGKILL the timeout command to drop the limit on the user, so I changed it to run as root and drop the shell to a different user.

Since then, it's been (fairly) smooth sailing. I'm thankful to the people who took the time to test the image so my dumb-ass doesn't get fucked over by malicious actors.

Making it look fancy

After most of the bugs were fixed; I got to work with making it look cool. I wrote a few functions to print out information and skills, and built in my Zsh config to make the syntax highlighting look nicer; but otherwise it was done.

I am pretty happy with the results. I set off making something that looked like a terminal, and I feel that the end result is pretty close.

Future plans involve putting some of my code into the image so people can actually view how god awful I am at coding, but for now I'm going to leave it as is.

Redesigning okami.dev taught me a lot of lessons, and it's especially taught me some great ways of securing Docker images for public use. Hopefully I'll be able to take this into future endeavors.

The code is available below:

GitHub - okamidash/okami.dev: Code for website okami.dev

Code for website okami.dev. Contribute to okamidash/okami.dev development by creating an account on GitHub.

GitHubokamidash

The finished site is available here

Thanks,

Okami.