upstream hyperkube, rpm edition

I’ve written recently about running kubernetes in containers on an atomic host. There are a few different ways to do it, but the simplest method involves fetching and running the Debian-based container provided by the upstream kubernetes project.

Debian is awesome, but I’m team RPM — when I run containerized apps, I tend to base them on CentOS or Fedora. If I can run kubernetes itself from an image based on one of those distros, I can save myself some storage and network transfer up front, and set myself up better to understand what’s going on inside the kubernetes containers.

As it turns out, it was pretty easy to mod the Makefile and Dockerfile that generate the containers. I swapped Debian apt-get specific bits for yum ones, changed the default baseimage to centos:centos7, and removed the gcloud-specifc push command.

The script expects to get a freshly-built copy of the all-in-one hyperkube binary that wraps together all of the kubernetes components from your local system. I modded the Makefile to grab a pre-built (by the kubernetes project) copy of this binary if it doesn’t exist on your machine.

Here’s how to make your own CentOS or Fedora-based kubernetes container, which you can then run using the directions under the heading “Containers from Upstream” from this post:

$ git clone https://github.com/jasonbrooks/kubernetes.git

$ git checkout hyperkube-rpm

$ cd kubernetes/cluster/images/hyperkube

$ make VERSION=v1.3.6

That command would build a CentOS-based hyperkube container, targeting the 1.3.7 release. To build and push to your docker registry, you could use the command:

$ make push VERSION=v1.3.6 REGISTRY="YOUR-DOCKER-REGISTRY"

To build and push a Fedora-based container with the very latest kube beta container, you can bump the VERSION and add a BASEIMAGE argument:

$ make push VERSION=v1.4.0-beta.8 REGISTRY="YOUR-DOCKER-REGISTRY" BASEIMAGE=fedora:24

I tested out v1.3.7 and v1.4.0-beta.8on CentOS Atomic, but hit this issue with cAdvisor and cgroups. I clicked the version count back to v1.3.6, and that worked.

New CentOS Atomic Host with Package Layering Support

Last week, the CentOS Atomic SIG released an updated version of CentOS Atomic Host (tree version 7.20160818), featuring support for rpm-ostree package layering.

CentOS Atomic Host is available as a VirtualBox or libvirt-formatted Vagrant box, or as an installable ISO, qcow2 or Amazon Machine image. Check out the CentOS wiki for download links and installation instructions, or read on to learn more about what’s new in this release.

http://www.projectatomic.io/blog/2016/08/new-centos-atomic-host-with-package-layering-support/

Up and Running with oVirt 4 and Gluster Storage

In June, the oVirt Project shipped version 4.0 of its open source virtualization management system. With a new release comes an update to this howto for running oVirt together with Gluster storage using a trio of servers to provide for the system’s virtualization and storage needs, in a configuration that allows you to take one of the three hosts down at a time without disrupting your running VMs.

One of the biggest new elements in this version of the howto is the introduction of gdeploy, an Ansible based deployment tool that was initially written to install GlusterFS clusters, but that’s grown to take on a bunch of complementary tasks. For this process, it’ll save us a bunch of typing and speed things up significantly.

Read more on the oVirt blog

running kubernetes in containers on atomic

The atomic hosts from CentOS and Fedora earn their “atomic” namesake by providing for atomic, image-based system updates via rpm-ostree, and atomic, image-based application updates via docker containers.

This “system” vs “application” division isn’t set in stone, however. There’s room for system components to move across from the somewhat rigid world of ostree commits to the freer-flowing container side.

In particular, the key atomic host components involved in orchestrating containers across multiple hosts, such as flannel, etcd and kubernetes, could run instead in containers, making life simpler for those looking to test out newer or different versions of these components, or to swap them out for alternatives.

Suraj Deshmukh wrote a post recently about running kubernetes in containers. He wanted to test kubernetes 1.3, for which Fedora packages aren’t yet available, so he turned to the upstream kubernetes-on-docker.

Suraj ran into trouble with flannel and etcd, so he ran those from installed rpms. Flannel can be tricky to run as a docker container, because docker’s own configs must be modified to use flannel, so there’s a bit of a chicken-and-egg situation.

One solution is system containers for atomic, which can be run independently from the docker daemon. Giuseppe Scrivano has built example containers for flannel and for etcd, and in this post, I’m describing how to use these system containers alongside a containerized kubernetes on an atomic host.

setting up flannel and etcd

You need a very recent version of the atomic command. I used a pair of CentOS Atomic Hosts running the “continuous” stream.

The master host needs etcd and flannel:

# atomic pull gscrivano/etcd

# atomic pull gscrivano/flannel

# atomic install --system gscrivano/etcd

With etcd running, we can use it to configure flannel:

# export MASTER_IP=YOUR-MASTER-IP

# runc exec gscrivano-etcd etcdctl set /atomic.io/network/config '{"Network":"172.17.0.0/16"}'

# atomic install --name=flannel --set FLANNELD_ETCD_ENDPOINTS=http://$MASTER_IP:2379 --system gscrivano/flannel

The worker node needs flannel as well:

# export MASTER_IP=YOUR-MASTER-IP

# atomic pull gscrivano/flannel

# atomic install --name=flannel --set ETCD_ENDPOINTS=http://$MASTER_IP:2379 --system gscrivano/flannel

On both the master and the worker, we need to make docker use flannel:

# echo "/usr/libexec/flannel/mk-docker-opts.sh -k DOCKER_NETWORK_OPTIONS -d /run/flannel/docker" | runc exec flannel bash

Also on both hosts, we need this docker tweak (because of this):

# cp /usr/lib/systemd/system/docker.service /etc/systemd/system/

# sed -i s/MountFlags=slave/MountFlags=/g /etc/systemd/system/docker.service

# systemctl daemon-reload

# systemctl restart docker

On both hosts, some context tweaks to make SELinux happy:

# mkdir -p /var/lib/kubelet/

# chcon -R -t svirt_sandbox_file_t /var/lib/kubelet/

# chcon -R -t svirt_sandbox_file_t /var/lib/docker/

setting up kube

With flannel and etcd running in system containers, and with docker configured properly, we can start up kubernetes in containers. I’ve pulled the following docker run commands from the docker-multinode scripts in the kubernetes project’s kube-deploy repository.

On the master:

# docker run -d \
--net=host \
--pid=host \
--privileged \
--restart="unless-stopped" \
--name kube_kubelet_$(date | md5sum | cut -c-5) \
-v /sys:/sys:rw \
-v /var/run:/var/run:rw \
-v /run:/run:rw \
-v /var/lib/docker:/var/lib/docker:rw \
-v /var/lib/kubelet:/var/lib/kubelet:shared \
-v /var/log/containers:/var/log/containers:rw \
gcr.io/google_containers/hyperkube-amd64:$(curl -sSL "https://storage.googleapis.com/kubernetes-release/release/stable.txt") \
/hyperkube kubelet \
--allow-privileged \
--api-servers=http://localhost:8080 \
--config=/etc/kubernetes/manifests-multi \
--cluster-dns=10.0.0.10 \
--cluster-domain=cluster.local \
--hostname-override=${MASTER_IP} \
--v=2

On the worker:

# export WORKER_IP=YOUR-WORKER-IP

# docker run -d \
--net=host \
--pid=host \
--privileged \
--restart="unless-stopped" \
--name kube_kubelet_$(date | md5sum | cut -c-5) \
-v /sys:/sys:rw \
-v /var/run:/var/run:rw \
-v /run:/run:rw \
-v /var/lib/docker:/var/lib/docker:rw \
-v /var/lib/kubelet:/var/lib/kubelet:shared \
-v /var/log/containers:/var/log/containers:rw \
gcr.io/google_containers/hyperkube-amd64:$(curl -sSL "https://storage.googleapis.com/kubernetes-release/release/stable.txt") \
/hyperkube kubelet \
--allow-privileged \
--api-servers=http://${MASTER_IP}:8080 \
--cluster-dns=10.0.0.10 \
--cluster-domain=cluster.local \
--hostname-override=${WORKER_IP} \
--v=2

# docker run -d \
--net=host \
--privileged \
--name kube_proxy_$(date | md5sum | cut -c-5) \
--restart="unless-stopped" \
gcr.io/google_containers/hyperkube-amd64:$(curl -sSL "https://storage.googleapis.com/kubernetes-release/release/stable.txt") \
/hyperkube proxy \
--master=http://${MASTER_IP}:8080 \
--v=2

get current kubectl

I usually test things out from the master node, so I’ll download the newest stable kubectl binary to there:

# curl -sSL https://storage.googleapis.com/kubernetes-release/release/$(curl -sSL "https://storage.googleapis.com/kubernetes-release/release/stable.txt")/bin/linux/amd64/kubectl > /usr/local/bin/kubectl

# chmod +x /usr/local/bin/kubectl

test it

It takes a few minutes for all the containers to get up and running. Once they are, you can start running kubernetes apps. I typically test with the guestbookgo atomicapp:

# atomic run projectatomic/guestbookgo-atomicapp

Wait a few minutes, until kubectl get pods tells you that your guestbook and redis pods are running, and then:

# kubectl describe service guestbook | grep NodePort

Visiting the NodePort returned above at either my master or worker IP (these kube scripts configure both to serve as workers) gives me this:

Download and Get Involved with Fedora Atomic 24

This week, the Fedora Project released updated images for its Fedora 24-based Atomic Host. Fedora Atomic Host is a leading edge operating system designed around Kubernetes and Docker containers.

Fedora Atomic Host images are updated roughly every two weeks, rather than on the main six-month Fedora cadence. Because development is moving quickly, only the latest major Fedora release is supported.

Read more on the Project Atomic blog

fedora and docker storage

While (pretty much) everyone who’s using docker is running it on Linux, and while lots of people run docker on their laptops and desktops, most aren’t running it directly on Linux desktops and laptops. Instead, most individual docker users are relying on some sort of purpose-built Linux distribution running as a virtual machine on their Mac or Windows machine.

However, if you are (like me) running Linux on your desktop, you can run docker containers right on your bare metal, with no virtualization overhead in between. Yay, Desktop Linux!

But wait. If you are (like me) running Fedora Linux on your desktop, and if you (also like me) weren’t thinking about docker and its particular storage needs when you installed Fedora on your machine, you could be in for some perplexing issues or at least crap performance, because of the way that docker storage works on Fedora.

I’ve written about the general issue elsewhere:

…the AUFS backend that started out as Docker’s default storage option, but never made its way into the mainlain Linux kernel, posed a problem for Red Hat and our upstream first, no out-of-tree bits ways.

The settled-upon solution was device mapper thin provisioning, which takes a block storage device to create a pool of space that can be used to create other block devices for Docker containers and images. The device mapper backend can be configured to use direct LVM volumes or you can let Docker create a pair of loopback mounted sparse files to serve as the block devices.

from: Friends Don’t Let Friends Run Docker on Loopback in Production

When you install Fedora on your desktop or laptop, the installer divvies your entire disk up into a small boot partition and a big LVM partition, and then divides that LVM space up into a swap volume that varies in size based on how much RAM you have installed, a root volume of 50GB, and a home volume that takes over whatever’s left.

With no room left for the pool of space that the docker device mapper storage driver needs for containers and images, the storage driver will turn instead to crappily-performing loopback mounted files. Boo!

A Fix

You can cut back the size of the home volume in Fedora without too much trouble. I like to use system-storage-manager to work with my disks:

NOTE: I guess I should add that whenever you’re mucking with your disks, you should make sure you have backups, and so on, but I have resized my own laptop partitions in just this way on more than one occasion, and I’ve tested the steps written here with a VM as I wrote this, so, yeah.

$ sudo dnf install -y system-storage-manager

Next, reboot your machine, and when you get back to the login screen, hit CTRL-ALT-F2 to get to a virtual terminal, and then log in as root. We need to do this in order to unmount the home directory before we shrink it. As root, you can use system-storage-manager to shrink down the home volume. Below I’m shrinking the home volume to 20G, because I’m testing these instructions on a VM with a 100GB drive. Substitute a value that makes sense for your rig.

# umount /home
# ssm resize -s 20G /dev/fedora/home
# reboot

If you’ve already installed and run docker, you’ll need to delete /var/lib/docker, where all of your containers and volumes live, so be prepared to rebuild those.

$ sudo systemctl stop docker
$ sudo rm -rf /var/lib/docker
$ sudo systemctl start docker

When docker starts up again, a script that comes bundled with Fedora’s distribution of docker will check to see that there’s space available in your volume group and will set up your storage correctly. If you want to grow your home volume later, it’s easy to do and doesn’t require unmounting anything. You’ll run the same ssm resize command from above, and swap in your desired volume size.

NOTE: If you’re using docker-engine from docker.com, check out these docs for setting up devicemapper driver correctly by hand.

Starting out right

If you haven’t yet installed Fedora, you can configure your system to accommodate this and other LVM storage scenarios moving forward by making your home volume smaller and modifying your “fedora” volume group from its default “Automatic” size policy to the “As large as possible” policy. This way, all your spare disk space will be ready for new volumes (such as the docker thin pool) or for growing your home or root volumes if you decide that you need the space later on. This is probably how Fedora partitioning should be configured by default, anyway, but it isn’t.

Looking ahead

Finally, there’s another option on the horizon for docker storage on Fedora, an option that doesn’t require partition changes or planning: OverlayFS. I wrote about this in the post I linked above, too, but the TLDR is that OverlayFS and SELinux don’t work together yet, although that’s set to change. Stay tuned.

HTTPS Everywhere: Encryption for All WordPress.com Sites

I’ve been wishing for this exact thing to happen.

WordPress.com News

Today we are excited to announce free HTTPS for all custom domains hosted on WordPress.com. This brings the security and performance of modern encryption to every blog and website we host.

Best of all, the changes are automatic — you won’t need to do a thing.

As the EFF points out as part of their Encrypt the Web initiative, strong encryption protects our users in various ways, including defending against surveillance of content and communications, cookie theft, account hijacking, and other web security flaws.

WordPress.com has supported encryption for sites using WordPress.com subdomains (like https://barry.wordpress.com/) since 2014. Our latest efforts now expand encryption to the million-plus custom domains (like automattic.com) hosted on WordPress.com.

The Let’s Encrypt project gave us an efficient and automated way to provide SSL certificates for a large number of domains. We launched the first batch of certificates in January 2016 and immediately starting working with Let’s Encrypt…

View original post 180 more words

blogging, control and open source

I’ve been sort of trying to get blogging again on WordPress lately, and as part of that, I’ve been paying more attention to the blogs I’m following using the WordPress Reader function. Recently, on Om Malik’s blog, I saw this item, pointing to the blog of photographer Eric Kim:

Instagram is now like Facebook

I clicked because I’ve been thinking grumbling thoughts lately about Instagram and its Twitter preview hiding ways.

OK, yeah, algorithm hate — it seems that every social network eventually adds features for bubbling up, through the magic of computer science, potentially interesting items you may have missed. These features disrupt the straight chronological feed presentation that many users prize (for themselves or for their followers). I know about these sentiments. I don’t share them — I like the option of “smart” timeline presentation. But, to each their own.

Anyway, I was interested to read, lower down, his suggestion: Start your own blog, and use WordPress.

I agree with this. I too am using WordPress, in part because WordPress works well, and in part because WordPress is open source software, and I’m an open source software loving sort. I was intrigued, further down, to see that all of Kim’s site content was open source:

Awesome! I clicked this link, assuming I’d find a creative commons license, but found instead a pair of essays about Kim’s ideas around open source photography:

OK, that sounds good. But what about licensing? I couldn’t Ctrl-F my way to finding the word “license” anywhere, so I broke down and read a bit more closely. I found this:

I also wanted to announce that I have recently made all of my photos on Flickr available for free as full-resolution downloads.

The terms he described sounded like a creative commons non-commercial license, which I don’t really consider to be open source, but whatever. Interestingly, the license he’s actually using is more permissive than he describes. He’s chosen the Public Domain Mark 1.0 license, which states that, “You can copy, modify, distribute and perform the work, even for commercial purposes, all without asking permission.”

As for the content on his site, there’s no license, but this chunk of text suggests that he may be talking about something else when he says “open source”:

So, interestingly, he’s at once ceding more control (over his pictures) and less control (over his blog posts) than he seems to intend. Even among open source software enthusiasts, it’s sort of common to downplay/misunderstand/dismiss licensing, and to insist that what really matters is something else: community, process, governance, etc. All of that definitely matters, but we can’t forget about licensing — copyright laws enforce a no-sharing-allowed default setting, so when we set out to collaborate freely, we need to explicitly change that setting, and open source licensing is how that’s done.

Speaking of which — I hadn’t yet put any license information of my own on this blog, which made it ALL RIGHTS RESERVED. I fixed that with a visit to the creative commons license chooser page and some wordpress widget box-fiddling.

flannel-bodhi

testing flannel

I noticed today (maybe I’ve noticed before, but forgotten) that the version of flannel in Fedora 23 is older than what’s available in CentOS. It looks like this is because no one tested the more-recent version of flannel in Fedora’s Bodhi, a pretty awesome application for testing packages.

Why not? Maybe because it isn’t always obvious how to test a package like flannel, but I here’s how I tested it, and added karma to the package in Bodhi.

I use flannel when I cluster atomic hosts together with kubernetes. I typically use the release versions of centos or fedora atomic, but the fedora project also provides an ostree image built from fedora’s updates-testing repo, where packages await karma from testers. I prepare three atomic hosts with vagrant:

[my-laptop]$ git clone https://github.com/jasonbrooks/contrib.git

[my-laptop]$ cd contrib/ansible/vagrant

[my-laptop]$ export DISTRO_TYPE=fedora-atomic

[my-laptop]$ vagrant up --no-provision --provider=libvirt

Next, I rebase the trio of hosts to the testing tree:

[my-laptop]$ for i in {kube-node-1,kube-master,kube-node-2}; do vagrant ssh $i -c "sudo rpm-ostree rebase fedora-atomic:fedora-atomic/f23/x86_64/testing/docker-host"; done

[my-laptop]$ vagrant reload --no-provision && vagrant provision kube-master

Reloading the hosts switches them to the testing image, and runs the ansible provisioning scripts that configure the kubernetes cluster. Now to ssh to one of the boxes, confirm that I’m running an image with the newer flannel, and then run a test app on the cluster to make sure that everything is in order:

[my-laptop]$ vagrant ssh kube-master

[kube-master]$ rpm -q flannel
flannel-0.5.4-1.fc23.x86_64

[kube-master]$ sudo atomic host status
  TIMESTAMP (UTC)         VERSION   ID             OSNAME            REFSPEC                                                        
* 2016-02-03 22:47:33     23.63     65cc265ae1     fedora-atomic     fedora-atomic:fedora-atomic/f23/x86_64/testing/docker-host     
  2016-01-26 18:16:33     23.53     22f0b303da     fedora-atomic     fedora-atomic:fedora-atomic/f23/x86_64/docker-host

[kube-master]$ sudo atomic run projectatomic/guestbookgo-atomicapp

That last command pulls down an atomicapp container that deploys a guestbook example app from the kubernetes project. The app includes two redis slaves, a redis master, and a trio of frontend apps that talk to those backend pieces. The bits of the app are spread between my two kubelet nodes, with flannel handling the networking in-between. If this app is working, then I’m confident that
flannel is working.

[kube-master]$ kubectl get svc guestbook
NAME        CLUSTER_IP       EXTERNAL_IP   PORT(S)    SELECTOR        AGE
guestbook   10.254.233.237                 3000/TCP   app=guestbook   55m

[kube-master]$ exit

[my-laptop]$ vagrant ssh kube-node-1

[kube-node-1]$ curl http://10.254.233.237:3000/info
# Server
redis_version:2.8.19
redis_git_sha1:00000000
redis_git_dirty:0
redis_build_id:c0359e7aa3798aa2
....

The app is working, flannel appears to be doing its job, so I marched off to bodhi to offer up my karma:

instant karma