Debug

Add codes in ansible script before error script to troubleshoot with hostvars issue, it will list all var resources

- debug:
msg: "{{ hostvars[inventory_hostname] }}"

List of Hosts

10.240.169.3: “MAAS, docker, openstack-kolla, kolla” all installed on here

Local Docker Registry

To install multi-node docker openstack, we need to have local registry service, Nexus3 is a GUI visible easy to use registry server. install it via docker:
create ./nexus3/data/docker-compose.yml

nexus:
image: sonatype/nexus3:latest
ports:
- "8081:8081"
- "5000:5000"
volumes:
- ./data:/nexus-data

and then docker-compose up -d to create docker container. May need to pip install docker-compose.

Launch web browser to 10.240.169.3(docker host):8081, default account:admin/admin123, then create a new repo type hosted/docker, use port 5000 and enable docker v1.

verify on docker hosts they can login this private registry: docker login -p admin123 -u admin 10.240.169.3:5000

To pull images from internet repo to local registry

on 10.240.169.3

pip install kolla kolla-build --base ubuntu --type source --registry 10.240.169.3:5000 --push

This will pull all available docker images from internet, and stored at local.

Prepare hosts for ceph osd

part disk label on each host:

parted /dev/sdb -s -- mklabel gpt mkpart KOLLA_CEPH_OSD_BOOTSTRAP 1 -1
parted /dev/sdc -s -- mklabel gpt mkpart KOLLA_CEPH_OSD_BOOTSTRAP 1 -1
parted /dev/sdd -s -- mklabel gpt mkpart KOLLA_CEPH_OSD_BOOTSTRAP 1 -1
parted /dev/sde -s -- mklabel gpt mkpart KOLLA_CEPH_OSD_BOOTSTRAP 1 -1
parted /dev/sdf -s -- mklabel gpt mkpart KOLLA_CEPH_OSD_BOOTSTRAP 1 -1
parted /dev/sdg -s -- mklabel gpt mkpart KOLLA_CEPH_OSD_BOOTSTRAP 1 -1
parted /dev/sdh -s -- mklabel gpt mkpart KOLLA_CEPH_OSD_BOOTSTRAP_1 1 -1
parted /dev/sdi -s -- mklabel gpt mkpart KOLLA_CEPH_OSD_BOOTSTRAP_2 1 -1
parted /dev/sdj -s -- mklabel gpt mkpart KOLLA_CEPH_OSD_BOOTSTRAP_1_J 1 -1
parted /dev/sdk -s -- mklabel gpt mkpart KOLLA_CEPH_OSD_BOOTSTRAP_2_J 1 -1

each host needs to install following:

apt install python-pip -y
pip install -U docker-py

apt-get install bridge-utils debootstrap ifenslave ifenslave-2.6 lsof lvm2 ntp ntpdate openssh-server sudo tcpdump python-dev vlan -y

no need to install docker.io manually, as there’s a bootstrap cmd doing this job under kolla-ansible: kolla-ansible -i multinode bootstrap-servers

if any deployment failure, copy /usr/local/share/kolla-ansible/tools/cleanup-containers to each host and run it to clean up containers and redo deploy again.

kolla-ansible -i multinode destroy can remove all deployed containor on all nodes, but ceph partitions will be kept. so to erase partitioned disks, run following on each host:

umount /dev/sdb1
umount /dev/sdc1
umount /dev/sdd1
umount /dev/sde1
umount /dev/sdf1
umount /dev/sdg1
umount /dev/sdh1
umount /dev/sdi1
dd if=/dev/zero of=/dev/sdb bs=512 count=1
dd if=/dev/zero of=/dev/sdc bs=512 count=1
dd if=/dev/zero of=/dev/sdd bs=512 count=1
dd if=/dev/zero of=/dev/sde bs=512 count=1
dd if=/dev/zero of=/dev/sdf bs=512 count=1
dd if=/dev/zero of=/dev/sdg bs=512 count=1
dd if=/dev/zero of=/dev/sdh bs=512 count=1
dd if=/dev/zero of=/dev/sdi bs=512 count=1
dd if=/dev/zero of=/dev/sdj bs=512 count=1
dd if=/dev/zero of=/dev/sdk bs=512 count=1

Ceph has disk usage ratio for its distributed disk arries, it may cause a problem as some of the disk may use 100% space and stop accepting any new data and make the entire storage stop working. To fix this, someone needs to watch disk usage frequently and do ceph osd reweight-by-utilization to forcely make disk arry recalculate and redo distribution to all disks, so that all disks will start accepting new data again.

Swift

Here’s a guide to calculate what number should be used for “swift-ring-builder create “. The first step is to determine the number of partitions that will be in the ring. We recommend that there be a minimum of 100 partitions per drive to insure even distribution across the drives. A good starting point might be to figure out the maximum number of drives the cluster will contain, and then multiply by 100, and then round up to the nearest power of two.

For example, imagine we are building a cluster that will have no more than 5,000 drives. That would mean that we would have a total number of 500,000 partitions, which is pretty close to 2^19, rounded up.

It is also a good idea to keep the number of partitions small (relatively). The more partitions there are, the more work that has to be done by the replicators and other backend jobs and the more memory the rings consume in process. The goal is to find a good balance between small rings and maximum cluster size.

The next step is to determine the number of replicas to store of the data. Currently it is recommended to use 3 (as this is the only value that has been tested). The higher the number, the more storage that is used but the less likely you are to lose data.

It is also important to determine how many zones the cluster should have. It is recommended to start with a minimum of 5 zones. You can start with fewer, but our testing has shown that having at least five zones is optimal when failures occur. We also recommend trying to configure the zones at as high a level as possible to create as much isolation as possible. Some example things to take into consideration can include physical location, power availability, and network connectivity. For example, in a small cluster you might decide to split the zones up by cabinet, with each cabinet having its own power and network connectivity. The zone concept is very abstract, so feel free to use it in whatever way best isolates your data from failure. Each zone exists in a region.

A region is also an abstract concept that may be used to distinguish between geographically separated areas as well as can be used within same datacenter. Regions and zones are referenced by a positive integer.

Run following script on any random host first to create swift templates for kolla to use.

export KOLLA_INTERNAL_ADDRESS=10.240.101.10 #don't really need for multinodes
export KOLLA_SWIFT_BASE_IMAGE="10.240.100.4:5000/kolla/ubuntu-source-swift-base:4.0.1"

mkdir -p /etc/kolla/config/swift

Object ring

docker run \
--rm \
-v /etc/kolla/config/swift/:/etc/kolla/config/swift/ \
$KOLLA_SWIFT_BASE_IMAGE \
swift-ring-builder \
/etc/kolla/config/swift/object.builder create 8 3 1

for i in {1..8}; do
docker run \
--rm \
-v /etc/kolla/config/swift/:/etc/kolla/config/swift/ \
$KOLLA_SWIFT_BASE_IMAGE \
swift-ring-builder \
/etc/kolla/config/swift/object.builder add r1z1-10.240.103.1${i}:6000/d0 1;
done

# Account ring
docker run \
--rm \
-v /etc/kolla/config/swift/:/etc/kolla/config/swift/ \
$KOLLA_SWIFT_BASE_IMAGE \
swift-ring-builder \
/etc/kolla/config/swift/account.builder create 8 3 1

for i in {1..8}; do
docker run \
--rm \
-v /etc/kolla/config/swift/:/etc/kolla/config/swift/ \
$KOLLA_SWIFT_BASE_IMAGE \
swift-ring-builder \
/etc/kolla/config/swift/account.builder add r1z1-10.240.103.1${i}:6001/d0 1;
done

# Container ring
docker run \
--rm \
-v /etc/kolla/config/swift/:/etc/kolla/config/swift/ \
$KOLLA_SWIFT_BASE_IMAGE \
swift-ring-builder \
/etc/kolla/config/swift/container.builder create 8 3 1

for i in {1..8}; do
docker run \
--rm \
-v /etc/kolla/config/swift/:/etc/kolla/config/swift/ \
$KOLLA_SWIFT_BASE_IMAGE \
swift-ring-builder \
/etc/kolla/config/swift/container.builder add r1z1-10.240.103.1${i}:6002/d0 1;
done

for ring in object account container; do
docker run \
--rm \
-v /etc/kolla/config/swift/:/etc/kolla/config/swift/ \
$KOLLA_SWIFT_BASE_IMAGE \
swift-ring-builder \
/etc/kolla/config/swift/${ring}.builder rebalance;
done

Then copy all what’s been generated by this script into kolla-deployer host’s /etc/kolla/config/swift.

Neutron

By default, kolla uses flat network and only enable vlan provider network when ironic is enabled. so you’ll see this in the ml2_config.ini

[ml2]
type_drivers = flat,vlan,vxlan
tenant_network_types = vxlan
mechanism_drivers = linuxbridge,l2population
extension_drivers = qos,port_security,dns

[ml2_type_vlan]
network_vlan_ranges =

[ml2_type_flat]
flat_networks = physnet1

[ml2_type_vxlan]
vni_ranges = 1:1000
vxlan_group = 239.1.1.1

[securitygroup]
firewall_driver = neutron.agent.linux.iptables_firewall.IptablesFirewallDriver

[linux_bridge]
physical_interface_mappings = physnet1:br_vlan

[vxlan]
l2_population = true
local_ip = 10.240.102.14

Move physnet1 from flat to network_vlan_ranges will enable vlan provider feature.

Ironic

Ironic is Bare Metal Service on openstack. It needs few parts to be installed before deployed by kolla.

  1. apt-get install qemu-ultis
  2. sudo pip install -U "diskimage-builder>=1.1.2"
  3. disk-image-create ironic-agent ubuntu -o ironic-agent (this cannot be done under lxc)
  4. copy generated ironic-agent.kernel and ironic-agent.initramfs to kolla-ansible host /etc/kolla/config/ironic.
  5. enable ovs, and then kolla-ansible deploy.

When deploying ironic, iscsid will be required and it may have error iscsid container: mkdir /sys/kernel/config: operation not permitted, the fix is to run modprobe configfs on each host.

iscsid may fail to start, remove open-iscsi on all hosts will fix this.

Manila

The share function component of openstack. Simple and easy to use, just few default parameters may not be showing in your default kolla generated conf file.

Take a look at manila.conf, you’ll find line as:

[DEFAULT]
enabled_share_backends = generic
default_share_type = default_share_type

[generic]
share_driver = manila.share.drivers.generic.GenericShareDriver
interface_driver = manila.network.linux.interface.OVSInterfaceDriver
driver_handles_share_servers = True
service_instance_password = ####
service_instance_user = manila
service_image_name = manila-service-image
share_backend_name = GENERIC

It’s good enough to start, but few default parameters not showing and may block your way to use it.

  1. Generic, it means using cinder as backend storage. and it will be by default using a share server(driver_handles_share_servers = True) which should be pre-provisioned by you prior to any of share instances creation.
  2. The default share type has been setup as default_share_type, so to keep it simple, just create a new share type and name it default_share_type.
  3. New image manila-service-image is needed. manila project has pre-build images, download it from http://tarballs.openstack.org/manila-image-elements/images/manila-service-image-master.qcow2, and then upload to glance..
  4. There’s a Flavor ID 100 missing under generic tab, it’s the default value if we don’t modify source script. Create image accordingly “openstack flavor create manila-service-flavor –id 100 –ram 256 –disk 0 –vcpus 1”

After all setup well, you should see no more error 501 on share tab under project.

Magnum

pip install python-magnumclient, version 2.6.0.

source user who need to use magnum need to have role in heat

make sure have following value in magnum.conf, otherwise barbican will complain for not being able to create certs.

[certificates]
cert_manager_type = barbican
cert_manager_type = x509keypair

Current COE and their supported distro, it has to match this table, otherwise it will complain vm type not supported.

Zun

Zun is another container component on Openstack. Not like Magnum, it doesn’t focus on HA or SWARM, instead it manages containers by using other components(neutron, glance, etc) on Openstack. So it’s easier and better managed than using docker machine, because that will make docker user to manage openstack too much, and each time when docker user creates a docker container, openstack admin needs to tell them about net-id, flavor-id, image-id.

But it has its own problem.

  1. The container will be deployed on compute node, instead of vms, so if it’s running on Kolla, then it will be running along with your other Kolla docker containers.
  2. Everytime, it will look for image locally, and will fail if it could not find it.

So the better way to do it is to use dedicated ZUN computes, which can be done by indicating nova availability zones.

Configuration should follow this guide: https://docs.openstack.org/kolla-ansible/latest/reference/zun-guide.html

Attention to docker setup on each compute node, for ubuntu, /etc/systemd/system/docker.service.d/kolla.conf  needs to be changed,

[Service]
MountFlags=shared
ExecStart=
ExecStart=/usr/bin/docker daemon -H tcp://:2375 -H unix:///var/run/docker.sock --cluster-store=etcd://:2379 --cluster-advertise=:2375 --insecure-registry 10.240.100.4:5000

The proper way to merge zun settings with kolla’s local insecure registry setup is above.

after this, reload daemon: systemctl daemon-reload, and restart: systemctl restart docker.service, then you should see docker listens on those ports.

Collectd Influxdb and Grafana

these combination can be a really nice tool for monitoring openstack activities.

few things need to changed from default kolla deployment config:

collectd:

FQDNLookup false
LoadPlugin network
LoadPlugin syslog
LoadPlugin cpu
LoadPlugin interface
LoadPlugin load
LoadPlugin memory
Server "10.240.101.11" "25826"

influxdb:

[[collectd]]
enabled = true
bind-address = "10.240.101.11:25826"
database = "collectd"
typesdb = "/usr/share/collectd/types.db"

Caution: it needs [[]] for collectd on influxdb. And also this types.db won’t be created automatically! Even though you can see on influxdb some udp traffic received from collectd, but it’s not stored in types.db until you manually copy it from collectd host/folder. This is critical!!!

Then Grafana is much simpler. You just need to add influxdb as datasource, and make up graphics in the dashboard. if you want to show interface traffic in bit/s, just use derivative and if_octets.

Rally with Tempest testing benchmark

Create tempest verifier, this will automatically download from github repo.

rally verify create-verifier –type tempest –name tempest-verifier

set this tempest verifier for current deployment with modified part in options.conf

rally verify configure-verifier --extend extra_options.conf

cat options.conf

[compute]
image_ref = acc51ecc-ee27-4b3a-ae2a-f0b1c1196918
image_ref_alt = acc51ecc-ee27-4b3a-ae2a-f0b1c1196918
flavor_ref = 7a8394f1-056b-41b3-b422-b5195d5a379f
flavor_ref_alt = 7a8394f1-056b-41b3-b422-b5195d5a379f
fixed_network_name = External

then just run test, “rally verify start –pattern set=compute” for specific parts of openstack.

Designate DNS

Steps to setup Designate:

  1. Config:

    enable_designate: "yes"
    dns_interface: "br_host"
    designate_backend: "bind9"
    designate_ns_record: "sample.openstack.org
    
  2. Run kolla-ansible deploy --tags designate first to make kolla install Designate onto all controllers, then kolla-ansible reconfigure --tags haporxy neutron horizon to refresh haproxy horizon and neutron for newly added designate service.

    openstack zone create --email [email protected] openstack.tdlab.ca.
    
  3. create a new domain zone.

    mkdir -p /etc/kolla/config/designate/designate-sink/
    
  4. create kolla customized config folder for sink.

    nano /etc/kolla/config/designate/designate-sink.conf
    
    [handler:nova_fixed]
    zone_id = 4a44b0c9-bd07-4f5c-8908-523f453f269d
    [handler:neutron_floatingip]
    zone_id = 4a44b0c9-bd07-4f5c-8908-523f453f269d
    
  5. Rerun deploy

    kolla-ansible reconfigure -i multinode --tags designate
    

6.refresh settings for designate. bash neutron net-list 7. check network id bash neutron net-update 3b56c605-5a01-45be-9ed6-e4c3285e4366 --dns_domain openstack.tdlab.ca. 8. update network with designate zone dns

then you should see dns record on this network, and all newly create instance using this network with has DNS name automatically associated with itself.

Mount cdrom along with disk drive

Some time we’d like to have cdrom mounted with bootable disk to install OS instead of boot from images. In such a case, we need to tell openstack volume a will be a bootable cdrom, volome b will be secondary and be kept as vdb disk. After OS installed, we can then kill whole VM and recreate it again with volume b only and assign it as bootable vda, then it will be working as regular vm.

Here’s how to create VM with cdrom

nova boot --flavor m1.small --nic net-id=e3fa6e8f-5ae9-4da6-84ba-e52d85a272bb --block-device id=e513a39b-36a1-49df-a528-0ccdb0f8515b,source=volume,dest=volume,bus=ide,device=/dev/vdb,type=cdrom,bootindex=1 --block-device source=volume,id=87ae535a-984d-4ceb-87e9-e48fa109c81a,dest=volume,device=/dev/vda,bootindex=0 --key-name fuel fuel

Create PXE boot image

Openstack doesn’t support instance PXE boot. To make it work, we need to create our own PXE bootable image.

Here’s how(only works on non-container):

https://kimizhang.wordpress.com/2013/08/26/create-pxe-boot-image-for-openstack/

  1. Create a small empty disk file, create dos filesystem.

    dd if=/dev/zero of=pxeboot.img bs=1M count=4
    fdisk pxeboot.img(create partition and flag it bootable)
    mkdosfs pxeboot.img
    
  2. Make it bootable by syslinux

losetup /dev/loop0 pxeboot.img
mount /dev/loop0 /mnt
syslinux --install /dev/loop0
  1. Install iPXE kernel and make sysliux.cfg to load it at bootup
wget http://boot.ipxe.org/ipxe.iso
mount -o loop ipxe.iso /media
cp /media/ipxe.krn /mnt
cat > /mnt/syslinux.cfg <<EOF
DEFAULT ipxe
LABEL ipxe
(2 space here)KERNEL ipxe.krn
EOF
umount /media/
umount /mnt

And then we need to figure out how to bypass neutron’s anti-spoofing. there are 2 ways to do it, either create flat network so it will not use neutron, or use dhcp opt to redirect dhcp/pxe traffic. In order to avoid future mass, I’d use vxlan network and keep using neutron’s default config.

  1. Create new vm using pxeboot.img on PXE subnet with fixed IP, 192.168.2.4.

  2. neutron port-list find out where the port is, and neutron port-update to change dhcp opt to redirect traffic from neutron’s dhcp to PXE server’s dhcp.

    neutron port-update 9dd25815-753b-4138-99ed-e2ba30048c3e --extra-dhcp-opt opt_value=pxelinux.0,opt_name=bootfile-name
    neutron port-update f9a416cd-02b0-4397-b0cc-cac6fc2556e9 --extra-dhcp-opt opt_value=192.168.2.11,opt_name=tftp-server
    neutron port-update f9a416cd-02b0-4397-b0cc-cac6fc2556e9 --extra-dhcp-opt opt_value=192.168.2.11,opt_name=server-ip-address
    
3. you may also need to change PXE dhcp lease record to mark new vm be assigned with fixed ip `192.168.2.4`, because we didn't turn off antispoofing, if vm gets assigned with different IP than what neutron dhcp would like to give, it will drop all traffic from this new vm.
to add static mapping for dnsmasq, modify `/etc/dnsmasq.d/default.conf` with `dhcp-host=AB:CD:EF:11:22:33,192.168.1.10,24h`

add whole subnet for a port to bypass antispoofing
```bash
neutron port-update b7d1d8bd-6ca7-4c35-9855-ba0dc2573fdc --allowed_address_pairs list=true type=dict ip_address=10.101.11.0/24

Enable Root Access

Kolla image disable root login by default. To enable it, we need to manually add sudoer inside container.

add following inside json under /etc/kolla/config/ceph/, take ceph as an example(file can’t have . in its name, otherwise system won’t read it):

{
"source": "{{ container_config_directory }}/cephsudo",
"dest": "/etc/sudoers.d/cephsudo",
"owner": "root",
"perm": "0600"
}

and then create ceph.sudo under same folder:

ceph ALL=(ALL) NOPASSWD: ALL

Enable usb hot plug for kolla nova kvm

Normally we enable usb hot plug by “virsh attach-device”, and if anything we need to change like controller settings for usb2.0 we use “virsh edit”, but in openstack, nova is monitoring and controlling the whole process of running a kvm, which means it will remove anything added after kvm created by it self, so we need to find a way to bypass its detection.

  1. Install lsusb to list all usb devices seen on nova compute.

    host# lsusb
    Bus 002 Device 004: ID 0781:5530 SanDisk Corp. Cruzer
    Bus 002 Device 005: ID 0781:5530 SanDisk Corp. Cruzer
    
  2. Edit usb.xml to prepare for adding hot plug usb, and virsh dumpxml instance-000000xx to dump and save existing instance target.

    address bus = lsusb info
    address type bus = controller index info
    
  3. Start instance in openstack and then virsh destroy instance-000000xx from nova compute.

  4. virsh undefine instance-000000xx to remove it from database, and edit and add usb2.0 controller within dumpxml file, then virsh define instance-000000xx to recreated it.

  5. Finally virsh start instance-000000xx to boot it. now it should have the new usb2.0 controller mounted and won’t be removed by nova.

virsh attach-device instance-000000xx usb.xml to add usb hot plug device.

Tips to LXC

Add usb device onto lxc container is much easier:

  1. Make sure USB connected on host, mount it onto a folder:

    mount /dev/sdc1 /media/usb1
    
  2. Bind folder with folder inside lxc container:

    lxc config device add container1 disk source=/share/c1 path=opt
    lxc config device add openstacktool usb1 disk source=/media/usb1 path=/media/usb1
    

MariaDB

To fix a MariaDB failure: This is our process (which is basically what the kolla bootstrap mariadb function does):

  • Check the following file on every controller node
[email protected]:/var/lib/docker/volumes/mariadb/_data# cat grastate.dat && echo
# GALERA saved state
version: 2.1
uuid: fbf02533-f205-11e7-ad67-ee3d1299a58d
seqno: -1
safe_to_bootstrap: 0
  • Choose the controller with the highest seqno value.
  • Change save_to_bootstrap: 0 to save_to_bootstrap: 1 on this node
  • On this node, run
docker run --net host --name mariadbbootstrap -v /etc/localtime:/etc/localtime:ro -v kolla_logs:/var/log/kolla/ -v mariadb:/var/lib/mysql -v /etc/kolla/mariadb/:/var/lib/kolla/config_files/:ro --restart on-failure:10 --env KOLLA_CONFIG_STRATEGY=COPY_ALWAYS --env BOOTSTRAP_ARGS='--wsrep-new-cluster' 172.28.0.1:5000/kolla/ubuntu-binary-mariadb:ocata

(You will have to modify the IP add and perhaps the label of the container depending on your situation). Wait until it looks like the container come up in a healthy state - this can take about a minute usually.

  • Restart the MariaDB Docker containers on the two other controller nodes - do it one by one. They should come up now. Look at the logs to see if they sync with the other mariadb instances in the cluster.
  • Stop and remove the bootstrap container -> docker rm -f mariadbbootstrap
  • Restart mariadb container on the node on which the rescue container was run. -> docker restart mariadb.