Virtual Machine deployments made easy

Traditional Virtual Machine (VM) deployments are usually a pain to deal with on libvirt.

You open up virt-manager, click through the menus, select how many vCPUs you want, the memory, the storage and the ISO you install from.

Once that's done, you then need to actually install the operating system onto your VM. This is a slow process, and I imagine large cloud providers aren't secretly doing this in the background when you are spinning up instances.

Having to do all these steps manually makes deployment velocity slow, and increases Toil. It also limits my avenues of attack when looking at ways I can automate my work. If I have to manually click through 'next' several times before getting to a usable machine, there's not much point in me automating the rest of the stack... is there?

In this blog post, I take a look at different ways of automating the VM provisioning process, alongside eliminating a lot of the repetitive work involved with setting up a VM.

Part 1: Approaches

PXEBooting

PXE booting, commonly referred to as 'Pixie booting' is a method of sending a boot image over the network to a target, and booting an image from that. Dating back to 1985, PXEbooting is well established and just about every machine with network capabilities can do it; from Rasberry Pis to Mainframes.

While PXEbooting in and of itself is not used to provision virtual machines as all it really does is send a bootloader to a machine over the network; it can be used in combination with several different tools to make it an effective method to do Virtual (and physical) machine installs.

Some of those tools are:

• Foreman
• Cobbler
• MaaS Project
• FOG Project
• Kickstart files (for RPM based distros)
• Preseed files (for APT baed distros)

Kickstart and Preseed files are less PXE Solutions and more predefined 'defaults'; but they are used commonly with PXE to do unattended installs of operating systems.

Foreman, Cobbler, MaaS and FOG are PXE solutions that all offer a good experience all around; and I've given the former 3 a try before. Foreman is useful if you're interested in managing the entire lifecycle of a machine, as it also installs puppet onto the target machine.

While they all have their respective benefits; PXE Provisioning solutions all have a common downside. Time.

When you provision a virtual machine using PXE, it has to download the base operating system from a mirror, and install all the packages. Even on the fastest of internet connections, this will take time.

When I was using foreman, it took about 10 minutes from clicking 'done' to having a system ready for use.

How is it that Cloud services can have a machine ready for me in less than a minute? Uber fast mirrors? No. If you want a Cloud-like experience, you need a Cloud-like solution.

Cloud-init

Cloud-init was born out of the idea that instead of re-downloading and installing the operating system loads of different times; why not just have one 'base' image with the base OS installed and work from that?

What cloud-init does, is apply a set configuration changes you want to make to that base OS to get it to the state you want. You're always going to need to install the base operating system anyway, so why not start from when you actually make changes that make it different from any other system.

How does it work though?

On any given OS with cloud-init installed and enabled, it will check to see if an additional ISO is mounted to the Virtual Machine. In that ISO it will check for two files, user-data and meta-data.

#cloud-config
users:
  - name: okami
    gecos: okami
    lock-passwd: false
    sudo: ALL=(ALL) NOPASSWD:ALL
    ssh_authorized_keys:
      - ssh-rsa AAA...6eNKCKYJ8iXuHUpZJ7EOesnCR9 okami@pegasus
runcmd:
  - sed -i 's/SELINUX=enforcing/SELINUX=disabled/' /etc/selinux/config

timezone: Europe/London
locale: en_GB.UTF-8

write_files:
  - path: /etc/yum.repos.d/kubernetes.repo
    content: |
      [kubernetes]
      name=Kubernetes
      baseurl=https://packages.cloud.google.com/yum/repos/kubernetes-el7-x86_64
      enabled=1
      gpgcheck=1
      repo_gpgcheck=1
      gpgkey=https://packages.cloud.google.com/yum/doc/yum-key.gpg https://packages.cloud.google.com/yum/doc/rpm-package-key.gpg

  - path: /etc/sysctl.d/kubernetes.conf
    content: |
      net.bridge.bridge-nf-call-ip6tables = 1
      net.bridge.bridge-nf-call-iptables = 1
      net.ipv4.ip_forward = 1

  - path: /etc/modules-load.d/br_netfilter.conf
    content: |
      br_netfilter

power_state:
  mode: reboot
  message: Bye Bye
  timeout: 30
  condition: True

runcmd:
  - touch /etc/cloud/cloud-init.disabled  

user-data is the file which tells cloud-init what you want done to the operating system. Some examples of things you can put into a user-data file are as follows:

• Adding users (and authorized SSH keys)
• Setting the Locale
• Adding Repos
• Installing Packages
• Updating the System
• Running arbitrary commands

Feel free to check out the documentation on cloud-init as there is a lot more you can do with it, than with what I just listed.

The meta-data file is where you set the machine's hostname. There are other things you can put in meta-data but for now that's all you need to know.

This allows you to create customized images, and have them ready in only a few minutes. From what I understand, this is what cloud providers use to spin up instances, as it allows you to have a very short time-to-live.

Using Cloud-init ties in very well with the use of Cloud images from Distribution vendors. A Cloud image is the 'base' OS I mentioned earlier. Typically only a few hundred MB in size; and on startup resize themselves to cover the size of the disk you give it. These cloud-images have already got cloud-init installed and enabled so you're ready to go very quickly.

You might be thinking "that's great and all, but surely I would need to generate the cloud-init ISOs manually, right?"

Well, sorta? There's another weapon up my sleeve that handles all of that for you.

Part 2. Ansible

Ansible is a tool developed by Red Hat, that is essentially clever shell scripting on steroids. I use it heavily in my lab to automate processes I feel need automating, and since my adoption of Ansible; I cannot praise the tool enough.

I took an Ansible script I found on github a while back (and cannot find the source [sorry]); and modified it fairly heavily to suit my needs.

I'll give you a high level overview of what it does.

1. Copy the base disk image from my Rsync share to a local location, renaming it {FQDN}.qcow2
2. Generates the hostname from one I give it on the command line and chucks it into meta-data
3. Generates a cloud-init ISO using the machine specific hostname, and a base user-data file
4. Resizes the new disk image from 200MB to 10GB
5. Runs virt-install, importing the new disk image and attaching the cloud-init ISO
6. Ejects the cloud-init ISO
7. Deletes the cloud-init ISO and meta-data file (no longer needed)

This is all done from a single command.

ansible-playbook -i inventory.ini image_provisioner.yml --extra-vars="runon=blackbird host_name=shuttlefish"

After a minute or two, I can see the hostname shuttlefish.vm.oxide.one has been registered to my DNS server, and I can ssh into the machine.

This has dramatically increased the TTL for my virtual machines, and reduced the amount of Toil relating to VM provisioning to almost none.

This script can very easily be used in an automated setting, which I plan on doing soon (keep watch of this blog)

Downsides of Cloud-init + Ansible

There are downsides to taking the cloud-init approach. The biggest being that you can't use this method to provision physical machines.

If that's what you're after, you're better off going for a PXE approach. Forman is pretty damn good for this.

Conclusion

Ansible is awesome; and when used in addition to tools like cloud-init, I am able to create Cloud-like experiences at home.

I hope you enjoyed this exploration into how I do my deployments. I mostly wrote this so I can write more blog posts that link back to this, so I hope you'll forgive if it lacks substance.

Thanks,

Okami.