First sysadmin/devops impression on RHEL 8 (article 1 -- initial impressions and installation overview )

If you are a Linux techie and fan of the RedHat ecosystem, you might have received word that the beta version of RHEL 8 is out. Years ago, I did a popular cover story for RHEL 7. It seems natural that I should continue the tradition and do the same with RHEL 8, even as it is still being polished. Chances are that by the time the final GA/production release is out, certain performance and versioning bits might be slightly different, so you are warned that this blog post will change, to reflect the expected changes.

Let's start with a visual which is really the first thing you are going to see if you start the graphical target, which RedHat now calls the 'Workstation' environment group (more on that later, when I describe the installation bits). I bet it will look familiar to you (excluding the wallpaper) especially if you are a Fedora 28/29 user.

Yes, it is GNOME 3.28, in particular version 3.28.2, the same as Fedora 28. No surprises there as the Fedora project is used as the testbed for things that will eventually end up in the RHEL release. Wayland is at play by default here, although breath easily, as you can keep X.Org with your binary NVIDIA drivers and your multi-GPU setup (that will not work with Wayland, this is not a RHEL 8 thing).

Other important component versions that mark the RHEL 8 beta release are:

• the Linux 4.18 kernel, 4.18.0-32.el8 in particular. This is a big and welcome step considering that RHEL 7 is based around the 3.10 kernel, which is really outdated in many respects (the latest at the time of writing was3.10.0-957.1.3.el7). As I write this, both active Fedora versions (28 and 29) have moved to the 4.19 kernel, but it seems that RHEL 8 has touch base with the 4.18 version and is likely to remain with that kernel. System stability and a more conservative environment when it comes to the backporting features and fixes (such as the Spectre and Meltdown patches that have substantial negative performance impact on the 4.20 kernel).
• The default gcc version is now 8.2.1 20180905, in line with the active Fedora distros. Compare that to RHEL 7's 4.8.5 20150623 also showing its date. Just so that I am not misunderstood, if you run RHEL 7, you could install more modern compilers by using the Redhat's software collection repos (rhel-server-rhscl-7-rpms, the devtoolset-6* and devtoolset-7* yum packages). I emphasize the word *default* here, which means what comes with the basic installation and the simplest of entitlements. 4.8.5 is really out of date, it would make sense if Redhat makes an effort to set the default one to 4.9.4 for RHEL 7.
• Pythonistas should feel right at home, but they should note that only Python3 is installed by default, version 3.6.6 in particular. Python developers need to explicitly install the available python2 packages. Python 2.7.15-15 is there, but with limited support. Again, that's not Redhat's decision as Python 2 is reaching EOL by the end of 2019. The sooner you migrate your apps to Python3 the better, with or without RHEL 8.

• Perl fans should find a system wide version of 5.26.2 on RHEL 8. In comparison, RHEL 7 has Perl version 5.16.3. IMHO, if you run something production grade with Perl, you should at least be on 5.24.x these days to get the best performance and functionality. 

• What you used to do with yum can now be done with dnf. That should not be news to you, especially if you have been following the Fedora releases. The introduction of the dnf tool has to do with important changes in the way software packages are tagged, installed and used (keep reading).

A few words about installing RHEL 8 now, as there are some notable changes there. RHEL 8 seems to organize software content by means of using two software repositories:

• The BaseOS repo: This includes RPM based packages for the core functionality of the operating system that can be searched, installed/deployed with dnf in pretty much the same way one used to do it with yum in RHEL 7. 
• The Appstream repo: This includes utilities to run real world workloads (for example databases, web servers, runtime environments) that can be organized either as RPM packages (like in the BaseOS repo) OR as multi-versioned collections (called streams) organized in modules. Modules are RPM extensions and their streams should allow you to choose among different versions of the package.

The concept of Application Streaming should give you the ability to have a module (say X) that offers you the Y and Z versions (streams) of a webserver. If Y is the production and Z the development version of that webserver, the Appstream repo should give you the ability to install X:Y on production systems and X:Z on your development cluster, all from one repo with a single command. You cannot install both versions in parallel on a system (unless you run your webservers in containers), but you should be able to install and run a specific version at a time.

If you are thinking that someone is trying to re-invent the wheel, you are probably right. You could previously achieve the same functionality on RHEL 7 and other platforms with the Software Collections and you could also deploy things like Environment Modules to achieve the same result, albeit at a slightly higher complexity. The idea is to perform everything here from specific repos and via your package manager. Software collections require more repos and they modify your Shell environment in ways that can create complex issues. Well, I am not trying to convince you to use one or the other here. You will be the judge of what works best for you.

There will be an additional article exploring the issue of Application Streaming. For now, this article will conclude with an overview of the RHEL 8 installation. I am going to outline the steps of installing a Virtual Machine hosted guest instance. My host operating system is Fedora 28 with its stock KVM/QEMU components. I dedicated 4 vCPUs, 4 Gigs of RAM, a functioning NAT enabled virtual NIC (to ensure that I can reach Redhat's subscription management infrastructure) and about 20 Gigs of a VirtIO disk for my qcow2 image.

There are many ways to install a RHEL 8 instance and should start with Redhat's Customer portal. The one I describe here is the Anaconda graphical installer from the Binart DVD images. You will need an account and an active subscription (that you can obtain by request if you have a portal account). This will enable you to download the beta test distro in a number of ways, as shown below.

I chose to download the 8.0 Beta Binary DVD, although the KVM Guest Image would have worked equally well (I wanted a complete set on a DVD image).

After verifying the SHA-256 checksum, I immediately proceeded to install my guest image and was greeted by the first installation screen, choosing the installation language.

The main 'installation summary' screen feels very familiar to those of you that have recently installed a Fedora distro, although a couple options ('SECURITY POLICY' and 'System Purpose') seem new.

The next step was to chose and test my keyboard layouts. I chose a Nordic (Norwegian), English and Greek keyboards and they seem to work OK.

I *would* suggest that you choose to set your 'Time & Date' settings next, but this is not a good idea. This is additional feedback I would like to pass on the Redhat team. You see, if you go to the 'Time & Date' settings, you choose your time zone and attempt to turn on the Network Time Protocol (NTP) by clicking on the ON/OFF 'Network Time' button, the button will refuse to stay on the 'ON' state.

The seasoned sysadmin/developer might figure out that this is due to the fact that the NTP server was not reachable: Although I had a perfectly ready virtual NIC standing by, this was not enabled by default. The correct order is thus to jump first to the 'Network & Host Name' settings, enable the NIC and ensure you are online.

I can now navigate back to the 'Time & Date' settings and verify that NTP is on ('Network Time' button is set to 'ON'). Timing is important. I feel that turning the configured NIC on by default OR alternatively displaying some kind of error message (like 'Cannot turn Network Time on because your NIC is inactive')when the NIC is turned off would result in a smoother user experience for an Enterprise Operating System.

Moving on to the next item of interest, the 'Software Selection' settings allow you to customize what will be installed (you can always modify this post installation). The distinction between 'Server' and 'Workstation' on the Base Environment is not new. If you want something customized to combine aspects of both, your mileage may vary. I would choose either 'Server' if you do not want a graphical environment or the 'Workstation' option (this was my choice for the demo I describe here) for a GNOME graphical environment. As explained, you can always add/remove stuff after the initial installation.

The 'Installation Destination' setting offers no surprise. Here, you can choose your installation drive and possibly encrypt your partitions. Nothing new here.

What's new in RHEL 8 are the following couple of screen settings. In particular, the 'SECURITY POLICY' setting, one can choose to customize the system between two policies. These policies ensure that certain components that have to do with firewalls, audit data and other OS settings are configured in a way that adheres to strict standard rules, to maximize your security. You should always check with your resident Information Security Officer, but as a rule of thumb, if you run the system in a bank or your system is involved in processing Credit Card data, the PCI-DSS v3 Baseline policy is a good one to choose. Alternatively, you can select the OSPP protection profile for general purpose OSes.

Finally, the 'System Purpose' screen lets you categorize the Role, SLA and usage of the system. I am not clear as to how Redhat will uses these settings as part of their Support and system inventory processes, suffice to say that collecting these data can help them dedicate their resources more efficiently in a support case.

Hit the 'Begin Installation' button of the installation summary screen and while the installer is progressing, you can set the root account password and an account. Eventually, when you reboot, you should be able to see the login screen of the graphical target.

We are not done yet. The system has installed, but it has not been registered with a subscription. To do that, you will need to obtain root, ensure you have Internet access and then just type the following two commands on the shell :

subscription-manager register --username YOUR_USERNAME --password YOUR_PASSWORD

subscription-manager attach --auto

The first command will register the system to the Red Hat Subscription Management platform (you obviously need to replace YOUR_USERNAME and YOUR_PASSWORD with your own account credentials). The second command will ensure that your system will attach to the beta entitlement. When you are done, here's how it should look on the Subscription Management Portal (uuid, username and Serial Number removed):

That's it, the system is now ready for use. Stay tuned for more RHEL 8 tests and analysis!

Security of Linux systems and privacy today

In this article, I am not arguing on whether Linux (or Android, or iOS and any other operating system) is the most suitable platform to entrust your data and personal communications today. Look for such dogmatic views elsewhere. I like, use and develop Linux systems, but I have been in business long enough to realize that security (and privacy as part of it) is a lot more complex business than choosing carefully your OS platform. Instead, I shall attempt to walk through some real world examples of certain sysadmin practices and security assumptions that have failed the Linux platform.

During the first week of June 2013 we found that Hetzner, one of the largest web hosting providers in Europe, has had its server security compromised. As a result, server password hashes and other sensitive data went into the hands of third party individuals, who obviously designed the malware vector and stroke gold. The extent of the information breach at Hetzner is still unknown as I write this. Obviously, being compromised is not good, but not knowing the extent of the compromise is worse. While I do not have data for other ISPs and web hosting providers, I strongly suspect that there will be other companies in the same line of business that were affected by this attack. The use of the Plex panel, PHP/Apache/SSH stacks is widespread

The attack vector that affected the Hetzner folks targeted Apache and openssh servers and quoting Martin Hetzner 'the "backdoor" exclusively infects the RAM...the infection neither modifies the binaries of the service which has been compromised, nor does it restart the service which has been affected. The standard techniques used for analysis such as the examination of checksum or tools such as "rkhunter" are therefore not able to track down the malicious code'. If this proves to be true (they have yet to conclude with their investigation at the time of writing), the real eye opener here is not that Hetzner got hacked. The real eye opener is that we have an effective RAM based Linux rootkit that affects essential services (such as http, https and ssh).

Six months ago, Steelcyber Scientific, a company that I consult for was one of the first to detect and mitigate for another Apache based exploit that installed rogue SSH servers and fished for customer usernames and passwords. Admittedly although less sophisticated, the older exploit was still successful in targeting the same services (sshd and httpd). I made various statements then in customers and IT journalists that people need to start taking Linux server security seriously. The success of Linux in the server and web hosting business has been providing an ideal target for malware writers for a number of years now, contrary to the popular (?) belief that exploits are mainly written for Windows (Android and iOS/OSX systems recently).

I am sure that this trend is going to continue, so I shall need to justify now what I mean by taking Linux security seriously.

One of the most frequently encountered pitfalls that make Linux systems vulnerable to various types of attacks is that most sysadmins still turn SELinux off. Yes, even experienced ones, that do keep their software updated. Yes, even after 10 years since its introduction to the mainstream Linux kernels. Many hosting providers allow customers to do that. Many of the attacks (and certainly the two previously described ones) could have been repelled if proper SELinux policies were in place. An SSH rogue server could not run in /tmp if SELinux is on. An i-frame injection could compromise Apache but could not install/execute files in any system directory.  Yes, it is annoying to install/troubleshoot new or upgraded software with SELinux. The Permissive SELinux mode is your friend to find and correct those issues keeping things contained. When you are done, make sure that the Enforcing mode is back on. Don't just turn it off and rely on traditional Unix and filesystem ACLs to contain things. This does not work.

I am not sure what your view is about logging/auditing systems in Unix and Linux, but I consider that the current utilities are not suitable for providing accountability on which process/user account does what in the system. In addition, today's common log/audit utilities do not allow sysadmins, devops and security experts to construct a forensic post-intrusion/breach picture, so that the extent of information leaks/intrusion is better understood and contained fast. I argue in favour of a better logging/auditing approach in this science paper. The end product, LUARM, is a different philosophy of what to log and how you can search it. We have employed LUARM successfully in monitoring and forensically examining a number of systems, and frankly, we have had great success in realizing fast what happened. If you compare that to traditional syslog approaches, well, good luck, you will have a lot more to examine and filter manually and one day you might find it what actually happened in your systems.

The final point I am trying to raise relates to authentication. Most production Linux systems today employ some sort of PAM complying approach to authenticate users. This is likely not a Single Sign On (SSO) solution. It is also likely that it only uses a username and password combination or an (D)/(R)SA key tied to a client machine. Whether you have a local/LDAP/(NIS, NIS+ if you are really oldie)/AD backend implementation is not relevant. The relevant bit concerns two facts:

i)The use of only a username and password or username/client/cryptographic key

ii)The widespread exposure of the front end daemon (Apache, SSH) in the Internet

(Please understand that I am referring to what is common practice, if you are a cautious sysadmin of a large installation, you probably do things differently.) 

A person once told me: "I use a secure login because I always SSH to the system". I replied rather cynically to that statement. The same person (OK, you know who you are :-) ) uses Skype for communicating with his family because the encryption is adequate. On the other hand, the same person is absolutely disgusted that Verizon turned his/her phone records to NSA. So, I cannot stop being cynical here.

I am not going to go through the theory of why a cryptographically enabled endpoint pair is more secure than a plain text endpoint pair, but still is not secure. Encryption aids security. It is not by itself security and if people do not get that point, they should look at what happened to Hetzner (and other providers that entrusted their security solely on an SSH frontend). The point here is that an entire industry lays all their eggs and/or applies the rule 'one size fits all' when it comes to matching the sensitivity of data, the exposure of the authentication front-end and the criteria of authentication. If you go to share hosting/cloud provider today, the main question is not what data you are going to store there, but what size of CPU/RAM and storage you need. The end result of this is that on cloud provider hard drives today, you can find anything: From personal information (pictures, videos), valuable Intellectual Property, to - I suspect - nuclear missile location details. Yet, nobody has checked the strength of the authentication procedure. You want two factor authentication? Good, build it yourself.

To sum up, proper containment policies. forensically enabled auditing and monitoring, as well as an authentication scheme that is suitable to the sensitivity of the data/services deployed in Linux infrastructures is missing today. If these issues are not addressed, perfectly adequate penguins can leak your information to skilled individuals. Make sure you address these issues with your IT team(s). Stay safe!

KVM hosted virtual servers using bridging: theory and practice

If you are a systems or networks administrator that:

• works in enterprise data centers or 
• someone that wants to deploy virtual servers in a newly acquired multi-core server using RHEL 6 and nothing more than the Linux KVM and RedHat's basic virt-manager application and/or 
• you wish to gain an understanding of KVM's virtual networking architecture

then this article/technical walkthrough is for you. Most of these techniques will work on other Linux distributions besides RHEL 6. Admittedly, there are more user friendly, free and commercial tools that allow you to deploy virtual machines. The usual suspects include VMware, RedHat, Oracle, Parallels that provide industrial strength solutions with intuitive point-and-click interfaces that make the setup of virtual machines an easy task.

However, I like to keep my production server software stack as simple as possible. Those of you that had to troubleshoot VM performance or other problems and faced the 'ping-pong' between the virtualization and the OS vendors will know what I mean. Thus, I use KVM/qemu and virt-manager to cater for my VM needs. The downside is that these tools are less intuitive to use for the newcomer, but with a little bit of good documentation and practice, they can be effective. I draw this conclusion after looking around in various technical support threads and after browsing RedHat's documentation on the subject. The threads seem to confuse the various virtual switching modes and techniques when things could be done more easily with interface bridging. The same can be said for Redhat's Virtualization Administration Guide, which does a fairly good job detailing the Routed, NAT and isolated virtual networking modes (Chapter 18), however it fails to mention how bridging could be used for hosting virtual servers. I am going to spend the rest of the article to explain this in detail.

The Theory

Let's be more specific now and explain what I mean when I say I need to deploy a fully networked virtual server. When you use the virt-manager application, it's easy to deploy a network enabled guest OS by means of using Network Address Translation (NAT). In fact, NAT (IP Masquerading, a specific mode of NAT) is the default guest OS virtual networking mode, using the IP address of the physical host server.  

 Figure 1

The figure above displays the networking data path traversal from the VM guests, all the way to the physical network/VLAN, when using the default virtual networking mode (NAT). Starting at the bottom of the figure, each guest has been assigned to a virtual network interface (vnetx). This is essentially a software implementation of an interface which is part of a virtual switch. At the other end of the virtual switch, a virtual bridge interface (virbr0) merges the traffic from the VMs and interfaces to the IPTABLES module which performs the actual NAT. At the end, you have the eth0 physical interface which carries the packets to the actual wire.  

In this scenario, your guest OS will have outbound network connectivity. Should you wish to enable inbound network connectivity, you will fail. It is possible to perform other tricks and enable port forwarding/SNAT/DNAT to enable inbound connections. However, this is cumbersome. As a result, my definition of deploying a proper virtual server resembles the following aspects of a true physical server:

• You have a physical MAC address tied to a network/VLAN broadcast domain
• You can deal with that MAC address in any way you would deal with a true physical NIC: ARP, assign a static IP, (static) DHCP, etc.
• You can have unrestricted outbound and inbound network access within that network/VLAN broadcast domain, a must requirement for a server system.

In order to achieve this, we need to employ the technique of interface bridging. For references on bridges, you can consult a variety of sources such as:
i)The IEEE 802.1D standard
ii)The older (out of date but still useful) Ethernet Bridge + netfilter HOW TO from TDLP.
iii)A copy of A. S. Tanenbaum's  Computer Networks classic textbook.
However, prior explaining how this works, let's throw in a realistic production environment scenario.

Figure 2

Figure 2 displays the network topology of a production VM server scenario.  There are two networks. One Class C internal (192.168.14.24), where hosts may or may not have outbound connectivity. Inbound connectivity to this network is prohibited by the top server which offers FTP, DMZ, FIREWALL, DHCP, and DNS services on the INTERNAL net. The other network is a world routable Class B (129.230/16).  

The VM host server needs to serve a number of virtual servers that have different network access criteria:

• Guest_01: Linux server to run an LAMP stack, exposed on the internal network.
• Guest_02: Development Windows 7 box, which needs to be accessible via non standard port ranges on the internal network, but also needs Internet access.
• Guest_03: Legacy SCADA Windows XP based system which needs to be accessible only via the internal network.

Clearly, Guest_01 is the least restricted system, so it makes sense to place it on the INTERNET/EXTERNAL Class B net. Guest_02 needs some protection so the outside folks cannot reach it, only it should reach the outside world by means of IP Masquerading, by using the Public routable IP of the FTP/DMZ/FIREWALL/DHCP/DNS server (129.230.135.131). Thus, it's a candidate for the INTERNAL Class C net. The same goes for Guest_03, which is the most isolated environment we need to protect, accessible only by INTERNAL network hosts.

At this point, it is useful to modify Figure 1 to illustrate the virtual network data path of our new scenario.  

  Figure 3

Figure 3 above illustrates the virtual network data path of our production scenario (Figure 2). In this case, instead of the virbr0 we have bridging modules bound to physical interfaces. Each physical interface is connected to the proper network/VLAN and has a bridge bound to it (we will illustrate how this is done). The role of the bridge is to create a data channel and forward traffic between the vnetx interfaces of the virtual switch and the physical interfaces. The objective is to enable the MAC address of the Guest_X machines to connect to the actual physical network/VLAN, as stated earlier. As a result, via bridge br3, we enable the virtual  servers Guest_02 and Guest_03 for the internal network and via br4, we connect Guest_01 to the external world. 

The practice

The previous section presented the theory. It's time now for the hands-on practical part. First of all, if you are dealing with a fresh installation, make sure you yum install the following groups, in order to have the full range of virtualization utilities and install your guests.

yum groupinstall Virtualization "Virtualization Client" "Virtualization Platform" "Virtualization Tools"

You should also install the bridge utilities, as they are needed:

yum install bridge-utils

The next thing you should ensure is that you have enough physical network interfaces on your VM host server. In order to implement our production scenario, Figure 2 indicates clearly that we need four Ethernet NIC ports: Two of them (eth2, eth3) are used to enable the server to have IP connectivity and routing on both networks. In contrast, eth4 and eth5 will be dedicated to carry the virtual server traffic.

We will not need IP addresses for interfaces eth4 and eth5. They will be brought up only to carry the bridged VM traffic. Make sure you identify the NIC ports properly and connect them to the proper network/VLAN Ethernet switch ports. To do that, you can remove their network cables and use the ethtool command to blink the NIC lights on the server side by doing a:
ethtool -p eth4

and
ethtool -p eth5 

to respectively identify the proper NIC ports. The next step is to connect them to the proper switch ports. In principle, once you identify the NIC port side with ethtool you should be OK. In practice, it is easy to make mistakes in messy/unlabelled network panels. Thus, after connecting the cables to the switch ports, one easy check is to bring the interface to promiscuous mode and watch for traffic indicating you are indeed on the right network/VLAN, by doing things like:
tcpdump -i eth4

and amongst the rest of the traffic, you would get something like the ARP or UDP broadcasts below confirming that eth4 is indeed on the internal network (Figures 2 and 3):

tcpdump: WARNING: eth4: no IPv4 address assigned
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on eth4, link-type EN10MB (Ethernet), capture size 65535 bytes
16:51:47.089529 ARP, Request who-has intfn1.internal.net tell esxfarm.internal.net, length 46
16:51:47.407363 STP 802.1d, Config, Flags [none], bridge-id 8005.00:1e:14:e6:48:80.800a, length 43
16:51:49.936209 IP winsys01.internal.net.17500 > 255.255.255.255.17500: UDP, length 119
16:51:49.936588 IP winsys02.internal.net.17500 > 192.168.14.255.17500: UDP, length 119

Now that the cables are connected properly we can start configuring the Ethernet bridges. A bridge is just another interface and the best way to configure this on a RHEL 6 system is by getting your hands dirty. Go right under the /etc/sysconfig/network-scripts directory and use your favourite text editor (vim, nano, Emacs) to make two files, one for each bridge interface device

ifcfg-br3 with the following contents:
DEVICE=br3
BOOTPROTO=none
TYPE=Bridge
ONBOOT=yes
DELAY=0

ifcfg-br4 with the following contents:
DEVICE=br4
BOOTPROTO=none
TYPE=Bridge
ONBOOT=yes
DELAY=0


This takes care of the bridge interface declaration. What's left is to associate the newly defined bridges with the right physical interface. Thus, under the same directory (/etc/sysconfig/network-scripts), we create two more files:

ifcfg-eth4 with the following contents:
DEVICE=eth4
HWADDR=00:10:18:31:5A:5B
NM_CONTROLLED=no
ONBOOT=yes
BRIDGE=br3

ifcfg-eth5 with the following contents:
DEVICE=eth5
HWADDR=00:10:18:19:4F:5C
NM_CONTROLLED=no
ONBOOT=yes
BRIDGE=br4

In short, with these four files we ensure that we have a persistent config where all interfaces (bridges and physical ones) are up on boot and we associate br3 to eth4 and br4 to eth5 (Figure 3). Fans of the brctl utility could also achieve the same result by doing a:

brctl addbr br3
brctl addif br3 eth4
brctl addbr br4
brctl addif br4 eth5

At that point, it is good to issue a:

service network stop; service network start

and check that the bridges and physical interfaces are up and available by issuing an ifconfig command. If all is well, you should see output like the one below (I have excluded some of the non relevant output for length reduction purposes):

br3       Link encap:Ethernet  HWaddr 00:10:18:31:5A:5B 
          inet6 addr: fe80::210:18ff:fe31:5a4b/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:386265 errors:0 dropped:0 overruns:0 frame:0
          TX packets:7 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:46672357 (44.5 MiB)  TX bytes:578 (578.0 b)

br4       Link encap:Ethernet  HWaddr 00:10:18:19:4F:5C         
          inet6 addr: fe80::210:18ff:fe19:4f33/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:616409 errors:0 dropped:0 overruns:0 frame:0
          TX packets:7 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:58946648 (56.2 MiB)  TX bytes:578 (578.0 b)
...

eth4      Link encap:Ethernet  HWaddr 00:10:18:31:5A:5B 
          inet6 addr: fe80::210:18ff:fe31:5a4b/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:600933 errors:0 dropped:0 overruns:0 frame:0
          TX packets:128158 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:270119283 (257.6 MiB)  TX bytes:10497306 (10.0 MiB)
          Interrupt:16

eth5      Link encap:Ethernet  HWaddr 00:10:18:19:4F:5C 
          inet6 addr: fe80::210:18ff:fe19:4f33/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:708614 errors:0 dropped:0 overruns:0 frame:0
          TX packets:9547 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:96954226 (92.4 MiB)  TX bytes:986694 (963.5 KiB)
          Interrupt:16

...

Note that all relevant interfaces are up and do not have an IP address . The second thing you should note is that the each bridge interface has the same MAC address as the physical interface it is associated with.

If you have reached that point, you are almost done. What you need to do now is to build your virtual machines. I assume you are familiar with how to build VMs on virt-manager. If not, I have written a quick summary of the procedures. Alternatively, if you have already existing VMs, you could reconfigure their networking to use the bridge interfaces.

Figure 4

Figure 4 above illustrates the network config for Guest_02. Make sure that the 'Source device' is one the available vnet interfaces that connects to br3 and apply the changes. You can do the same for the rest of the virtual server VMs. When you are done, you can now check with the brctl utility the final configuration by doing a:

brctl show

and you should get output similar to the one below:

Figure 5

Note the interfaces column which should correctly list all the physical and vnet interfaces associated to each bridge.  When you fire up any of the virtual servers, you should be able to see it with its vnet's interface MAC address on the virtual network. Let's take Guest_02 as an example.  From our VM host server console, we type:

[root@vmserver ~]# ping win01
PING win01.internal.net (192.168.14.23) 56(84) bytes of data.
64 bytes from win01.internal.net (192.168.14.23): icmp_seq=1 ttl=128 time=2.13 ms
64 bytes from win01.internal.net (192.168.14.23): icmp_seq=2 ttl=128 time=0.518 ms
^C
--- win01.internal.net ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1360ms
rtt min/avg/max/mdev = 0.518/1.324/2.131/0.807 ms
[root@vmserver ~]# arp -a | grep win01
win01.internal.net (192.168.14.23) at 52:54:00:28:23:af [ether] on eth2

Note Guest_02's MAC address from Figure 4. That's the one replying and bridged into the internal network. This means that for all intents and purposes, Guest_02 is just another server on the internal network. Mission accomplished.

Happy KVM sponsored virtual server hosting!