Skip to main content

VirtualBox 5.1.27 + RHEL 7.4 (and others) + kernel update = suggest double reboot

For a long time now, every damn time I go to apply a kernel update, I have to rebuild the VirtualBox Guest Additions. If I have any vboxsf mounts set to mount at boot in /etc/fstab, I can look forward to a rescue-mode prompt. This is not specific to RHEL7, or to RHEL (I see plenty of reports of Ubuntu 11.04 with the same issue).

The 'dkms' (Dynamic Kernel Module Support) is meant to prevent this issue by triggering a rebuild when a kernel package is updated. It's installed,....  so why isn't it working. Time to get my hands dirty and learn a bit about dkms.



A bit of brief research first (kinda feels like cheating... let's just say its giving credit where credit is due):

  • This post on Ask Ubuntu points out that it needs to be a dkms-aware package; my Guest Additions come direct from VirtualBox (Oracle), not as a package; perhaps that's the problem.
  • dkms(8)
    • I should expect a dmkms.conf file, presumably somewhere under /etc.... I see a directory /etc/dkms/, with a .conf file and some .spec files (that seem to be for generating RPMs)... but nothing VirtualBox related.
    • 'dkms status' returns an empty output, so presumably nothing is registered.
  • yum history list / yum history info 95
    • Looking at the output from my last 'yum upgrade', I see nothing about DKMS, but that is probably to be expected.
  • /usr/share/doc/dkms-2.3/README.md
    • Never neglect /usr/share/doc/ !
  • lsmod | grep vbox
    • Surprisingly, this shows that vboxsf and vboxguest are loaded.... but X11 is too small for comfort, and obviously its usual driver is missing.
    • locate vbox | grep '\.ko$' (remembering that this data would have been current of when updatedb was last run -- before I upgrade the kernel) leads me to expect I should see a 'vboxvideo' in the output.
  • Well, that's interesting. The timestamps on /var/log/vboxadd-install{,-x11}.log match either when I rebooted or upgraded.... 
    • cat /proc/uptime; date --date='2283 seconds ago' puts my boot time at 9:57, which matches up quite nicely.
    • the 'End time' in yum history info 95 puts my upgrade window ending at 9:55
    • so therefore it seems VirtualBox (presumably via startup script) tried to install stuff (and may have partially succeeded with the exception of the vboxvideo driver).
Okay, that's useful. So I can forget about DKMS; that's a red herring (on my red hat).

Before we look at the log file, let's understand how this build is being triggered:

systemctl status 'vboxadd*'

This gives me three service units: vboxadd.service, vboxadd-service.service, and vboxadd-x11.service, all of which have an 'active' and apparently healthy status.

Let's have a closer look at that vboxadd-x11.service unit:


# systemctl cat vboxadd-x11.service 
# /usr/lib/systemd/system/vboxadd-x11.service
[Unit]
SourcePath=/opt/VBoxGuestAdditions-5.1.27/init/vboxadd-x11
Description=
Before=shutdown.target 
After=
Conflicts=shutdown.target

[Service]
Type=forking
Restart=no
TimeoutSec=5min
IgnoreSIGPIPE=no
KillMode=process
GuessMainPID=no
RemainAfterExit=yes
ExecStart=/opt/VBoxGuestAdditions-5.1.27/init/vboxadd-x11 start
ExecStop=/opt/VBoxGuestAdditions-5.1.27/init/vboxadd-x11 stop

[Install]
WantedBy=multi-user.target


(aside: despite vboxvideo not being available, shared clipboard is working)

So, we can see that this should happen before a shutdown (which is significant -- the running kernel would be a different version). According to the script that it runs, this output goes to vboxadd-install-x11.log

Okay, so understanding is locking into place. Let's have a look at those log files, specifically /var/log/vboxadd-install-x11.log


VirtualBox Guest Additions installation, Window System and desktop setup
(END)


... well, that's short, but at least we know that the 'setup' function in that script starts.

Hmmm, there is a _lot_ of sensitivity there to the version of X (look for x_version in that script), and I did have to use a test-build of Guest Additions to account for a newer (1.19.3) X.Org server on RHEL 7.4....  and that appears to be the issue (at least for me at present); the test-build Guest Additions don't support 1.19.*

        .... many other cases
        ....
        1.12.* | 1.13.* | 1.14.* | 1.15.* | 1.16.* | 1.17.* | 1.18.* )
            xserver_version="X.Org Server ${x_version_short}"
            vboxvideo_src=vboxvideo_drv_`echo ${x_version_short} | sed 's/\.//'`.so
            setupxorgconf=""
            test -f "${lib_dir}/${vboxvideo_src}" ||
            {
                echo "Warning: unknown version of the X Window System installed.  Not installing"
                echo "X Window System drivers."
                dox11config=""
                vboxvideo_src=""
            }
            ;;
        * )
            # For anything else, assume kernel drivers.
            dox11config=""
            ;;

I could hack up the script a little to add support for 1.19.... but I should probably just get a later test build; particularly if I go to submit a bug report.

You can see how I updated guest additions in my earlier post. But the script has not changed....

Hmmm, if you look in /opt/VBoxGuestAdditions-5.1.27/lib/VBoxGuestAdditions/, there are a lot of vboxvideo_drv_VERSION.so files, where VERSION is what is being calculated in the cases above. And there is no vboxvideo_drv_119.so file.... so what was I running before I upgraded my kernel? (X.org was the same version).

.... I really do dislike dealing with driver issues; its the worst part of dealing with VMs. So much promise, shot down by reality.

However, If I reboot the guest, it all comes right. Not sure if that was because:
  • Just needs a double-reboot to ensure that when it builds (at shutdown) its running the kernel it needs to be building for (I suspect that's probably what happened).
  • Newer version of Guest Additions test build (5.1.27 r117558)
But now at least the vboxvideo module is loaded... so what is X11 using:

[    10.465] (II) LoadModule: "vboxvideo"
[    10.465] (WW) Warning, couldn't open module vboxvideo
[    10.465] (II) UnloadModule: "vboxvideo"
[    10.465] (II) Unloading vboxvideo
[    10.465] (EE) Failed to load module "vboxvideo" (module does not exist, 0)

... apparently not vboxvideo. Seems to be using either vesa judging by 'using VT number 1'

[    10.466] (II) LoadModule: "fbdev"
[    10.466] (II) Loading /usr/lib64/xorg/modules/drivers/fbdev_drv.so
[    10.467] (II) Module fbdev: vendor="X.Org Foundation"
[    10.467]    compiled for 1.19.1, module version = 0.4.3
[    10.467]    Module class: X.Org Video Driver
[    10.467]    ABI class: X.Org Video Driver, version 23.0
[    10.467] (II) LoadModule: "vesa"
[    10.467] (II) Loading /usr/lib64/xorg/modules/drivers/vesa_drv.so
[    10.467] (II) Module vesa: vendor="X.Org Foundation"
[    10.467]    compiled for 1.19.1, module version = 2.3.2
[    10.467]    Module class: X.Org Video Driver
[    10.467]    ABI class: X.Org Video Driver, version 23.0
[    10.467] (II) modesetting: Driver for Modesetting Kernel Drivers: kms
[    10.467] (II) FBDEV: driver for framebuffer: fbdev
[    10.467] (II) VESA: driver for VESA chipsets: vesa
[    10.467] (++) using VT number 1

Okay, so I shouldn't expect great performance, but at least I can get a comfortable 2D environment until such time as VirtualBox development catches up with X.org drivers used in RHEL 7.4

I wonder if I should just get VMWare workstation.... my blog seems to only get updated lately with 'how to get working again after VirtualBox updates' type stories.

Thanks for reading, and I hope you got something out of it.

This is Cameron, signing off.
    --- Understanding begets Resolution






Comments

Popular posts from this blog

Use IPTables NOTRACK to implement stateless rules and reduce packet loss.

I recently struck a performance problem with a high-volume Linux DNS server and found a very satisfying way to overcome it. This post is not about DNS specifically, but useful also to services with a high rate of connections/sessions (UDP or TCP), but it is especially useful for UDP-based traffic, as the stateful firewall doesn't really buy you much with UDP. It is also applicable to services such as HTTP/HTTPS or anything where you have a lot of connections...

We observed times when DNS would not respond, but retrying very soon after would generally work. For TCP, you may find that you get a a connection timeout (or possibly a connection reset? I haven't checked that recently).

Observing logs, you might the following in kernel logs:
kernel: nf_conntrack: table full, dropping packet. You might be inclined to increase net.netfilter.nf_conntrack_max and net.nf_conntrack_max, but a better response might be found by looking at what is actually taking up those entries in your conne…

ORA-12170: TNS:Connect timeout — resolved

If you're dealing with Oracle clients, you may be familiar with the error message
ERROR ORA-12170: TNS:Connect timed out occurred I was recently asked to investigate such a problem where an application server was having trouble talking to a database server. This issue was blocking progress on a number of projects in our development environment, and our developers' agile post-it note progress note board had a red post-it saying 'Waiting for Cameron', so I thought I should promote it to the front of my rather long list of things I needed to do... it probably also helped that the problem domain was rather interesting to me, and so it ended being a late-night productivity session where I wasn't interrupted and my experimentation wouldn't disrupt others. I think my colleagues are still getting used to seeing email from me at the wee hours of the morning.

This can masquerade as a number of other error strings as well. Here's what you might see in the sqlnet.log f…

Getting MySQL server to run with SSL

I needed to get an old version of MySQL server running with SSL. Thankfully, that support has been there for a long time, although on my previous try I found it rather frustrating and gave it over for some other job that needed doing.

If securing client connections to a database server is a non-negotiable requirement, I would suggest that MySQL is perhaps a poor-fit and other options, such as PostgreSQL -- according to common web-consensus and my interactions with developers would suggest -- should be first considered. While MySQL can do SSL connections, it does so in a rather poor way that leaves much to be desired.

UPDATED 2014-04-28 for MySQL 5.0 (on ancient Debian Etch).

Here is the fast guide to getting SSL on MySQL server. I'm doing this on a Debian 7 ("Wheezy") server. To complete things, I'll test connectivity from a 5.1 client as well as a reasonably up-to-date MySQL Workbench 5.2 CE, plus a Python 2.6 client; just to see what sort of pain awaits.

UPDATE: 2014-0…