Ovirt Hosted Engine – HA Gluster

oVirt-logoantmascot

Being conscious of wasted watts from a growing number of machines in my POC OpenStack cluster, I decided to downsize and try nested KVM. An added attraction was the opportunity to play with the hosted engine, enhanced glusterfs, and OpenStack Neutron integration; all new with oVirt 3.4. As a onetime fan of DRBD, I thought I’d take a fresh look at the latest there too.

Very impressed with what I’ve tasted so far, but for my part, it was quite an adventure to realise a stable HA platform.
Another limitation in my mind, is the direct support of storage formats.
oVirt 3.4 also improved integration with the OpenStack glance image store.
Despite QEMU & OpenStack support for VMDK & other image formats, for now we are still restricted to RAW & QCOW2 based images when integrating Glance image stores with oVirt.
Import of existing virtual guests and P2V of physical hosts is on the drawing board, but still a good few lines of code away.

Notes and resources

Test System Setup

    Supermicro X8DT3
    Dual Xeon E5620 Quad core CPU @ 2.40GHz
    3Ware 9690SA-8I Raid Controller
    2 Disk Raid 1 => Minimal Centos 6.5 host OS
    6 Disk Raid 5 (5 +sp) => Gluster, DRBD, Glance
    4 x GB Ethernet

The raid 5 was carved up into aligned partitions using parted.
Apps like sfdisk and cfdisk have been great tools over the years, but can’t handle guid partition tables.
Tip: partition alignment can have an impact on IO performance.
A simple means of keeping the partitions aligned is to use “G” or percentage.

E.g.
mkpart primary 0% 20%
mkpart primary 2000G 4000G
While mkpart primary 0 1000G results in…
Warning: The resulting partition is not properly aligned for best performance.

Pitfalls encountered

  • I missed adding a firewall exception for the VDSMD service on tcp 54321.
    This produced downstream problems that were less obvious to the uninitiated.
    Firstly the initial host node set-up would succeed, but the second would fail.
    The hosts would list as non responsive within the engine gui.
    The engine log would complain no route to path.
  • Installing additional nodes from the engine GUI automatically configures iptables, unless manually unchecked. The firewall rules automatically configured by engine, break the HA cluster.

The logs for vdsmd, engine, glusterfs & ovirt-hosted-engine-ha, were key to navigating the dark woods.

Useful Guides
oVirt 3.4, Glusterized by Jason Brooks

More notes & specifics posted soon…