OLVM + BMS Environment Setup

Peter Goldthorp, Dito. March 2022

OLVM environments configured on BMS need DNS and NTP access. They also need firewall rules to control access to the KVM hosts.

DNS configuration

Setup Dnsmasq VMs in each region to perform DNS resolution for the BMS VMs

Setup Instructions

  1. Provision OL7 VMs using the “Custom” instance type and 1024G of RAM

  2. Edit the /etc/sysconfig/network-scripts/ifcfg-eth0 file setting the ip address and DNS sequence. Example:

     TYPE="Ethernet"
 PROXY_METHOD="none"
 BROWSER_ONLY="no"
 BOOTPROTO="none"
 DEFROUTE="yes"
 IPV4_FAILURE_FATAL="no"
 IPV6INIT="yes"
 IPV6_AUTOCONF="yes"
 IPV6_DEFROUTE="yes"
 IPV6_FAILURE_FATAL="no"
 IPV6_ADDR_GEN_MODE="stable-privacy"
 NAME="eth0"
 UUID="e235398c-c24c-44a5-9dbe-2d757335b5fc"
 DEVICE="eth0"
 ONBOOT="yes"
 IPADDR="10.216.218.171"
 PREFIX="26"
 GATEWAY="10.216.218.190"
 DNS1="127.0.0.1"
 IPV6_PRIVACY="no"

  3. Edit the /etc/hostname files.

  4. Restart the network using systemctl restart network.service

  5. Install dnsmasq and bind-utils. Tip: you may need to add a proxy entry to /etc/yum.conf - see Squid documentation

     yum install dnsmasq
 yum install bind-utils

  6. Open firewall port 53

     firewall-cmd --get-active-zones
 firewall-cmd --zone=public --permanent  --add-port=53/tcp
 firewall-cmd --zone=public --permanent  --add-port=53/udp
 firewall-cmd --reload

dnsmasq configuration

  1. Edit the dnsmasq.config file. Note 10.216.218.171 in listen-address=10.216.218.171 is the ip address of the VM

     mv /etc/dnsmasq.conf /etc/dnsmasq.conf.safe
 vi /etc/dnsmasq.conf

 port=53
 domain-needed
 bogus-priv
 expand-hosts
 domain=olvm.my.domain.com
 listen-address=10.216.218.171
 listen-address=127.0.0.1
 bind-interfaces
 no-dhcp-interface=eth0
 auth-server=dns.olvm.my.domain.com,eth0
 auth-zone=olvm.my.domain.com

 # Setup Google private API access for use with cloud storage
 # https://cloud.google.com/vpc/docs/configure-private-google-access-hybrid

 domain=googleapis.com,199.36.153.8/30
 auth-zone=googleapis.com,199.36.153.8/30,eth0
 host-record=private.googleapis.com,199.36.153.8
 host-record=private.googleapis.com,199.36.153.9
 host-record=private.googleapis.com,199.36.153.10
 host-record=private.googleapis.com,199.36.153.11

 # Decode each domain name used to make a cloud storage request
 # because dnsmasq does not support cname wildcards

 cname=www.googleapis.com,private.googleapis.com
 cname=cloudresourcemanager.googleapis.com,private.googleapis.com
 cname=dl.google.com,private.googleapis.com
 cname=accounts.google.com,private.googleapis.com
 cname=oauth2.googleapis.com,private.googleapis.com
 cname=storage.googleapis.com,private.googleapis.com

 #Uncomment log-queries to write debug entries in /var/log/messages
 #log-queries

    BMS environments need access to private Google APIs. This is achieved by using a combination of Cloud Router and DNS settings. The DNS settings are contained in /etc/dnsmasq.conf. It sets a host record for each IP address associated with the private.googleapis.com domain. It also sets multiple CNAME entries. Multiple CNAME entries are required due to dnsmasq limitations. dnsmasq does not support wildcard characters in cname entries. The entry cname=*.googleapis.com,private.googleapis.com resolves a single canonical name *.googleapis.com instead of being expanded to match any googleapis.com subdomain. The list of required entries was determined by reviewing entries in /var/log/messages while attempting to access Cloud Storage from a BMS VM

Identify VM IP addresses

  1. Edit the /etc/hosts file

  2. Restart dnsmasq and test

      systemctl restart dnsmasq

  nslookup <value-from-hosts-file>

  3. Enable dnsmasq service

      systemctl enable dnsmasq

Update

  1. Edit /etc/hosts in both VMs to add or remove a record
  2. Restart dnsmasq: systemctl restart dnsmasq

Network Time Protocol Setup

GCP Compute Engine VMs use an internal NTP server at metadata.google.internal. This server does not respond to NTP requests from BMS VMs. As an alternative Compute Engine VMs are configured as NTP servers using chrony.

NTP Server

  1. Use the ntp-server terraform code to provision 2 or more NTP server VMs and an internal load balancer
  2. Update the /etc/hosts file on the 2 DNS servers with an entry that resolves ntp.my.domain.com to the load balancer IP address.

NTP Client

  1. Install chrony sudo yum install chrony

  2. Login as root and edit the /etc/chrony.conf file on each VM. Replace the server entries

     # Use public servers from the pool.ntp.org project.
 # Please consider joining the pool (http://www.pool.ntp.org/join.html).
 server 0.rhel.pool.ntp.org iburst
 server 1.rhel.pool.ntp.org iburst
 server 2.rhel.pool.ntp.org iburst
 server 3.rhel.pool.ntp.org iburst

    with

     server ntp.my.domain.com iburst

  3. Restart the chrony daemon

     systemctl restart chronyd
 systemctl status chronyd

  4. Verify result by running chronyc sources and timedatectl

     chronyc sources -v
 210 Number of sources = 1

 .-- Source mode  '^' = server, '=' = peer, '#' = local clock.
 / .- Source state '*' = current synced, '+' = combined , '-' = not combined,
 | /   '?' = unreachable, 'x' = time may be in error, '~' = time too variable.
 ||                                                 .- xxxx [ yyyy ] +/- zzzz
 ||      Reachability register (octal) -.           |  xxxx = adjusted offset,
 ||      Log2(Polling interval) --.      |          |  yyyy = measured offset,
 ||                                \     |          |  zzzz = estimated error.
 ||                                 |    |           \
 MS Name/IP address         Stratum Poll Reach LastRx Last sample
 ===============================================================================
 ^* ntp.my.domain.com        3   6    17     9   -286us[ -316us] +/-  847us

    Verify the Reach value is not 0 and the System clock synchronized value from timedatectl = yes.

     timedatectl
             Local time: Wed 2022-03-23 15:42:33 UTC
         Universal time: Wed 2022-03-23 15:42:33 UTC
                 RTC time: Wed 2022-03-23 15:42:34
                 Time zone: Etc/UTC (UTC, +0000)
 System clock synchronized: yes
             NTP service: inactive
         RTC in local TZ: no

Firewall Rules

Configure firewalld rules to control access to the KVM hosts.

KVM Host

The KVM host firewall requirements see Oracle docs are implemented using firewalld. Firewall rules are created in the default firewalld zone when a KVM host is configured. The default zone is likely to be public. This exposes the KVM hosts to traffic originating from VMs provisioned on them. To secure the environment additional zones and scopes need to be configured to prevent unauthorized access to the KVM host server.

Isolation can be achieved through firewalld and service definitions. These definitions are stored in XML files which can be modified.

  1. Modify the XML definition file for the public zone to remove OLVM and ssh services and add an unfiltered source address.
  2. Create a new olvm-mgmt zone identifying the OLVM engine, BMS browser and jump hosts as source addresses.

Zones

Create the following files (in folder /etc/firewalld/zones/) then run firewall-cmd --reload to apply them:

public: /etc/firewalld/zones/public.xml

<?xml version="1.0" encoding="utf-8"?>
<zone>
  <short>Public</short>
  <description>For use in public areas. You do not trust the other computers on networks
  to not harm your computer. Only selected incoming connections are accepted.</description>
  <service name="dhcpv6-client"/>
  <source address="0.0.0.0/0"/>
</zone>

olvm-mgmt: /etc/firewalld/zones/olvm-mgmt.xml

<?xml version="1.0" encoding="utf-8"?>
<zone>
  <short>OLVM Management</short>
  <description>Allow OLVM management traffic</description>
  <service name="ssh"/>
  <service name="cockpit"/>
  <service name="libvirt-tls"/>
  <service name="snmp"/>
  <service name="vdsm"/>
  <service name="ovirt-imageio"/>
  <service name="ovirt-vmconsole"/>
  <source address="10.216.218.132"/>
  <source address="10.216.218.196"/>
  <source address="10.216.106.6"/>
  <source address="10.216.104.31"/>
  <source address="10.216.106.14"/>
</zone>

Service Definitions

Use firewalld predefined services (in folder /usr/lib/firewalld/):

ssh: /usr/lib/firewalld/ssh.xml

<?xml version="1.0" encoding="utf-8"?>
<service>
  <short>SSH</short>
  <description>Secure Shell (SSH) is a protocol for logging into and executing commands on remote machines.
   It provides secure encrypted communications. If you plan on accessing your machine remotely via SSH over
   a firewalled interface, enable this option. You need the openssh-server package installed for this option
   to be useful.</description>
  <port protocol="tcp" port="22"/>
</service>

dhcpv6-client: /usr/lib/firewalld/dhcpv6-client.xml

<?xml version="1.0" encoding="utf-8"?>
<service>
  <short>DHCPv6 Client</short>
  <description>This option allows a DHCP for IPv6 (DHCPv6) client to obtain addresses and other IPv6 settings
   from DHCPv6 server.</description>
  <port protocol="udp" port="546"/>
  <destination ipv6="fe80::/64"/>
</service>

cockpit: /usr/lib/firewalld/cockpit.xml

<?xml version="1.0" encoding="utf-8"?>
<service>
  <!-- This is a firewalld service definition for Cockpit -->
  <short>Cockpit</short>
  <description>Cockpit lets you access and configure your server remotely.</description>
  <port protocol="tcp" port="9090"/>
</service>

libvirt-tls: /usr/lib/firewalld/libvirt-tls.xml

<?xml version="1.0" encoding="utf-8"?>
<service>
  <short>Virtual Machine Management (TLS)</short>
  <description>Enable this option if you want to allow remote virtual machine management with TLS encryption,
   x509 certificates and optional SASL authentication. The libvirtd service is needed for this option to be
   useful.</description>
  <port protocol="tcp" port="16514"/>
</service>

snmp: /usr/lib/firewalld/snmp.xml

<?xml version="1.0" encoding="utf-8"?>
<service>
  <short>SNMP</short>
  <description>Simple Network Management Protocol is an "Internet-standard protocol for managing devices on
  IP networks". Enable this service if you run SNMP agent (server).</description>
  <port protocol="tcp" port="161"/>
  <port protocol="udp" port="161"/>
</service>

vdsm: /usr/lib/firewalld/vdsm.xml

<?xml version="1.0" encoding="utf-8"?>
<service>
  <short>oVirt's Virtual Desktop and Server Manager</short>
  <description>The VDSM service is required by a Virtualization Manager to manage the Linux hosts. VDSM manages
  and monitors the host's storage, memory and networks as well as virtual machine creation, other host
  administration tasks, statistics gathering, and log collection.</description>
  <port protocol="tcp" port="54321"/> <!-- vdsmd -->
  <port protocol="tcp" port="5900-6923"/> <!-- guest consoles -->
  <port protocol="tcp" port="49152-49216"/> <!-- migration -->
</service>

ovirt-imageio: /usr/lib/firewalld/ovirt-imageio.xml

<?xml version="1.0" encoding="utf-8"?>
<service>
  <short>oVirt Image I/O</short>
  <description>oVirt Image I/O simplifies the workflow of introducing new oVirt images into the
  oVirt environment.</description>
  <port protocol="tcp" port="54322"/>
</service>

ovirt-vmconsole: /usr/lib/firewalld/ovirt-vmconsole.xml

<?xml version="1.0" encoding="utf-8"?>
<service>
  <short>oVirt VM Console</short>
  <description>oVirt VM Consoles enables secure access to virtual machine serial console.</description>
  <port protocol="tcp" port="2223"/>
</service>

Copyright © Dito LLC, 2023