My experiences with XEN VGA passthrough

The following post gets quite technical, and employs a bit of jargon. You can read the whole anecdote if you want, or just scroll down to the bottom if you want some advice related to Linux Mint Debian Edition, XEN and VGA Passthrough.

I recently built myself a new computer, and after jumping through many hoops (involving Intel's Haswell Refresh I believe – seems trying a "Devil's Canyon" processor with an 8-series chipset isn't a good idea – D'oh #1!), I got a machine up and running with Linux Mint Debian Edition 64-bit. There is an nVidia card in one PCIe slot, an AMD card in another PCIe slot, and an integrated GPU on my Core i7 that is disabled in the BIOS. Once I had this all set up, I thought it would be child's play (well, not really, but quite simple) to follow this tutorial and get Windows Vista up and running, and using my AMD card, while leaving the nVidia card for Mint.

Boy, was I wrong! At first everything seemed to be going smoothly, but I had a niggling feeling that it was going too well – this was, after all, a tutorial for normal Mint, not the Debian Edition. My first stumbling block came at step 17, which didn't give the output hoped for. However, I assumed that this was because I skipped steps 9 and 10, opting for Synergy instead. I forged on, and ended up unable to pass through any devices to the VM – D'oh #2!

At this point, I had Vista up and running, able to control it via VNC, but with no VGA passthrough. Therefore, I backtracked, and I'm not sure what I did, but I ended up with a "no such file or directory" error upon attempting to start the VM, with the unhelpful question "Is xend running?". Seemingly I could no longer use the XM toolstack for whatever reason. Therefore, I switched to XL with the command:

sudo sed -i 's/TOOLSTACK=.*\+/TOOLSTACK="xl"/' /etc/default/xen

Of course, now problems with QEMU versions started to rear their ugly heads. XEN couldn't find the right QEMU binaries anymore, and following the instructions wasn't enough, because I was on Debian rather than Ubuntu. My suspicions were correct after all! I was eventually able to start the VM with the following config file:

firmware_override = '/usr/lib/xen-default/boot/hvmloader'
memory = 8192
name = 'vista'
vif = [ 'mac=00:16:3e:68:e1:01,bridge=xenbr0' ]
disk = [ 'phy:/dev/mapper/guest-vista,hda,w' , 'file:/home/mgkeyes/Vista.iso,hdc:cdrom,r' ]
#device_model = '/usr/lib/xen-default/bin/qemu-dm'
device_model_version = 'qemu-xen'
device_model_override = '/usr/bin/qemu-system-x86_64'
#pci=[ '02:00.0', '02:00.1' ]
pci=[ '02:00.1' ]

Now, you may notice that I've commented out the line passing through both the GPU and its sound chip, in favour of one passing through only the sound chip. This is because the sound chip is passed through successfully, and detected by Windows. The GPU, on the other hand, causes the VM to not boot, with the only way to shut it down being:

sudo xl destroy vista

Unfortunate, especially after I had spent so much time trying to get the Radeon driver blacklisted so the card was even available for passthrough.

I was puzzled. Surely my card should pass through fine. After all, all Radeon HD 7000 are supported, and I had bought a card from that series, albeit a low-end one. Or so I thought. What I had bought was a "Radeon R7 250", which I misinterpreted as "Radeon 7250". Turns out it's part of a new series of cards altogether – D'oh #3! It is with considerable shame that I divulge this.

=====The rest of this isn't really related to VGA passthrough, but it is interesting, and somewhat related to XEN. You can still scroll down to the bottom to see my lessons.=====

So, having cursed my inability to properly research components for my new build, I decided to wait until, some time down the line, support for these cards is properly programmed. (But in the meantime, I wonder how I will play Rayman Legends…) Having given up on the VM's graphics, I had a look at the host's. The "nouveau" open-source drivers were being used, which caused the Cinnamon desktop environment to run in software mode, and when I tried to install and run Steam, it didn't even start for me. Therefore, I decided to give the proprietary nVidia drivers a try. After all, they had worked well on my old system, with the same card (a GeForce GTX 550Ti by the way).

Bad idea. Cinnamon, instead of running in software mode, crashed upon login, leaving me in "Fallback Mode", which was very unpleasant. To make matters worse, Firefox also crashed any time it was even slightly over-taxed, making even regular browsing quite a frustrating experience. Also the screen went irrecoverably blank any time I tried to exit to command-line to install an updated version of the driver (obtained from nVidia's site). Defeated, I returned to the nouveau driver, an action which itself involved jumping through a few hoops.

On the up-side, it turned out that I could get proper OpenGL rendering by installing the libgl1-mesa-dri-experimental package. This got Cinnamon out of software mode, allowed me to run Steam, and even play through Portal! Afterwards, I tried a few other things, like playing recordings in MythTV, but ended up coming across this little kernel bug, which affects the version still being shipped with this distro (3.11). Still, I was happy, and was looking forward to seeing what would happen with Portal 2.

What I saw was really ugly, and unfortunately the system became too unresponsive to take a screenshot. nouveau decided to spam all ttys (accessible by Ctrl+Alt+F(1-6)) with error messages, making it basically impossible to even try to kill the game from the command line. After a hard reboot, I thought it might be the same kernel bug again, so I disabled the option "Wait for Vertical Sync" in the advanced video settings. It seems I was on to something, because this time the game didn't go ugly, but it did freeze with three dots left in the loading progress bar for the first map.

Having already found that exiting to command line no longer turned the screen blank, I decided to have another shot at installing the updated driver from nVidia's website. It installed successfully, but a reboot produced – you guessed it – a blank screen! D'oh #4!

At this point I realised that I was still booting into the XEN hypervisor, which wasn't much good to me when VGA passthrough wasn't working. Therefore, I decided to try the normal kernel, and what did I get? Cinnamon running perfectly and Portal 2 playable! I do have a few visual glitches in normal apps, but I can overlook them (for now).

Now, I publish this in the hope that others can benefit from my experience, so I'll try to boil it down to a few little lessons that I have learned, and that I hope others may be able to take away from this:

  • DO YOUR RESEARCH! If you see something like "Devil's Canyon" on a processor, be sure you know exactly what it means. Don't assume that "R7 250" means the same thing as "Radeon 7250". This is probably obvious to most, but I failed to do it, and, as mentioned, I am ashamed of that.

  • Try not to assume that tutorials for normal Mint work with the Debian Edition.

  • device_model_version = 'qemu-xen'
    device_model_override = '/usr/bin/qemu-system-x86_64

    seems to work on Linux Mint Debian Edition with the correct QEMU packages installed. These packages include qemu and qemu-system.

  • AMD R7 250 cards DO NOT WORK with VGA passthrough at this time, and it's probable that other members of the Rx 200 series don't either.

  • Proprietary nVidia drivers don't work well with the version of the Linux kernel used in the XEN hypervisor (at least in Debian, or it might just be a Linux Mint Debian Edition thing).

I hope this has been in some way helpful, or at least amusing! Suggestions for improvement or clarification are more than welcome.