I’ve been using Kubespray to deploy Kubernetes since Kube version 1.9. It’s by far the most customizable and flexible deployment tool for Kubernetes on the open-source market. So I think it’s worth a post for it.

To begin with, let’s talk about the dark and stone-age time when Kubespray just came out as “Kargo”, it used to be so confusing and felt like today’s Kubeadm which is using command-line option to interact and deploy, but now, it’s well maintained and fully Ansible based, which means all variables and parameters are configured inside script, not more command-line options. I’d say well done kubespray, you guys have done such a great job, making this tool so much easier to use.

I don’t want to talk too much about how to use it to deploy kubernetes, since there are already so many posts on the Internet for this kind of init build guide. Rather I’d like to leave some hints about how to maintain its deployment, upgrade, and customize.

Autoscaling with Openstack Nova

Kubespray itself provides fucntion to add/delete hosts. To add new nodes ansible-playbook -i inventory/mycluster/hosts.ini scale.yml To remove nodes ansible-playbook -i inventory/mycluster/hosts.ini remove-node.yml Above are generic ways to add/remove nodes when dealing with bare-metal servers, you need to manually edit hosts.ini with new nodes information every time you add a new server. However, if we host K8s clusters on clouds, we can actually make this more intelligent and automated. Ansible itself provides a well-scripted python file to list Openstack vms(I also do this myself creating bunch of scripts to show vm, network, images and etc on Openstack, but I found Ansible has one for this purpose, then why not use it? )

wget https://raw.githubusercontent.com/ansible/ansible/devel/contrib/inventory/openstack_inventory.py
chmod +x openstack_inventory.py

To try out if it works in your env, source your Openstack RC file, and type ‘openstack_inventory.py –list’, it will call openstack sdk and list Json style info of all vms in that tenant. So if we put all our K8s slave nodes on a single tenant ‘slave’, while keep our master nodes in another tenant ‘master’ and not touching them at all, we then source ‘slave’ RC and do

ansible-playbook -i openstack_inventory.py scale.yml 


ansible-playbook -i openstack_inventory.py remove-node.yml

and run this commands every 10mins in a turn on some jump box that has access to all slave nodes, then it becomes dynamic deployable cluster.

K8s network isolation with local docker network

This actually has been posted on another post of mine. This is not a bug, but rather a security design and concern, some users may not want their local docker container to be able to directly access their pods in k8s which live on the same hosts. So what they do is to disable Iptable’s auto config for docker process, then local containers can’t go outside because they’re all blocked by default Iptable rules.

  1. When use k8s CNI, it changes host Iptable to forbiden local non k8s docker container to go outside. To bring this feature back, we can add these on Iptables( is the default docker network IP range):

    iptables -t nat -N DOCKER
    iptables -t nat -A DOCKER -i docker0 -j RETURN
    iptables -t nat -A POSTROUTING -s ! -o docker0 -j MASQUERADE
    iptables -t nat -A OUTPUT ! -d -m addrtype --dst-type LOCAL -j DOCKER
    iptables -A FORWARD -i docker0 -o ens3 -j ACCEPT
    iptables -A FORWARD -i ens3 -o docker0 -j ACCEPT


  2. Another option is to enable IPTABLES binding with local docker. In kubespray, it’s hardcoded to disable iptalbes on non k8s container by removing docker0 related iptable rules. To enable it, change --iptables=true in roles/docker/templates/docker-options.conf.j2 if deploy using kubespray, it will eventually be pushed to /etc/systemd/system/docker.service.d/docker-options.conf. This is a better fix, as 1st fix would be lost once system rebooted. Note that this j2 file content and location may chagne due to code updates on kubespray