I rent a physical server at OVH’s daughter company SoYouStart, and use it to run VMware vSphere/ESXi 6.0 on it, with a number of OpenBSD Virtual Machines (one of which hosts this blog). It took me a while to figure out the correct way of configuring the network settings on my OpenBSD VMs, as their network setup is somewhat unorthodox.

This post is based on an answer I’ve provided on ServerFault OpenBSD: Gateway outside subnet (works in Linux).

The Problem: Network Setup

When ordering a server from OVH (or their daughter companies), or I’m sure many other providers of the kind, the customer will be assigned an IP address to go with that server, with an option for additional IPs. My intention was to install ESXi and then spin up a few VMs, therefore I needed to use those extra IPs.

The additional IPs will most likely belong to a different pool than the main IP address of the server, and in any case all those IPs will need to be configured on a network interface with a 32 bit netmask (255.255.255.255), making them each belong to a different subnet.

What that means is that the default gateway will always be outside of the network configured on the network interface.

If I read correctly into that design, it is intended to eliminate the Layer 2 broadcast traffic, of which there could be quite a significant amount if all the hosts were in a large shared subnet. Also, this design allows them to assign IPs from different pools, each promising to be recognised as located in a different region of the world – a nice bonus, if one wants to provide or access geographically restricted services.

OVH have a documentation providing examples how to configure this on a number of operating systems (currently Debian derivatives, RedHat derivatives, FreeBSD and Windows) but nowhere I could find the necessary information for OpenBSD.

The Solution

As mentioned above, I’ve requested extra IP addresses from OVH and they come from a completely different range. For the discussion here, let’s assume these are my settings:

• my main IP address (which the ESXi Host is using): 213.0.113.78/32
• the extra IP address range for VM guests: 192.0.2.64/30
• the default gateway for ALL of the above: 213.0.113.254
• please note – all hosts need to use a host netmask (255.255.255.255) due to the way the OVH network is configured

The configuration for OpenBSD varies slightly depending on which version you’re running. I’ll provide details for OpenBSD 5.8 and OpenBSD 5.9.

OpenBSD 5.8

To configure the routing on a VM running OpenBSD 5.8, I’ve had to run the following commands:

ifconfig vmx0 inet 192.0.2.64 255.255.255.255 
route add -inet 213.0.113.254 -llinfo -link -static -iface vmx0 
route add -inet default 213.0.113.254

To make these persistent across the system’s reboot, I ignore the /etc/mygate file and put the following into the /etc/hostname.vmx0 file instead:

inet 192.0.2.64 255.255.255.255
!route add -inet 213.0.113.254 -llinfo -link -static -iface vmx0
!route add -inet default 213.0.113.254

As I understand it, this works using the following trick:

• we configure the IP address on the interface
• with the first route command we translate the IP address of the gateway (213.0.113.254) to a link address (MAC address); this is done by the -llinfo option;
• in the same route command, using the -link option, we install the link address to the routing table and using the -iface vmx0 we tell the kernel which network interface that link address is reachable via; the -static switch marks it as a manually inserted entry into the routing table
• the second route command can now succeed, as the route to the default gateway is now known

OpenBSD 5.9

After upgrading my OpenBSD VMs from 5.8 to 5.9, I’ve noticed that the IPv4 connectivity to the VMs would drop after a couple of days (for some VMs it would happen more often than for others, but eventually it happened on each and every one of them). When I investigated, I found out that the route to 213.0.113.254 via vmx0 was dropped, and therefore the default route became invalid. This apparently has been caused by a change in arptfree() between 5.8 and 5.9, as per this post.

To fix that issue, the configuration for 5.9 has to be slightly different. In your /etc/hostname.vmx0 file put the following lines:

inet 192.0.2.64 255.255.255.255
!sleep 2
!route add -inet -net 213.0.113.254 -netmask 255.255.255.255 -link -cloning -iface vmx0
!route add -inet default -ifa 192.0.2.64 213.0.113.254

You will notice the sleep command – for some reason this is required on OpenBSD 5.9 but wasn’t before. Without the sleep, the first of the two route commands will not be executed and therefore your routing will not be configured correctly.

Also, the above configuration uses -cloning option for route, which indicates that the link entry for the route should be generated, as opposed to using the -link option which creates a manual entry, which will be removed when the ARP timer for that entry expires.

OpenBSD 6.0

The information for this section has been provided by one of the readers, Jean Michel Rizzardi. Thanks Jean Michel!

Once again, the configuration of the /etc/hostname.vmx0 file needs to be changed due to changes in the networking code in the kernel – the good news is that the configuration is now a bit simpler and makes more sense. The working config for 6.0 is as follows:

inet 192.0.2.64 255.255.255.255
!sleep 2
!route add -inet 213.0.113.254/32 -link -iface vmx0
!route add -inet default 213.0.113.254