gpu_accelerated_vm_with_kvm

Differences

This shows you the differences between two versions of the page.

Link to this comparison view

Both sides previous revision Previous revision
Next revision
Previous revision
gpu_accelerated_vm_with_kvm [30.09.2018 07:54] – [2nd Try: HP RGS] Pascal Sutergpu_accelerated_vm_with_kvm [29.01.2023 22:39] (current) – [Re-Try RDP + passthrough] Pascal Suter
Line 162: Line 162:
 in order to get the middle mous button (wheel) to click, i had to modify the config of my citrix receiver on my client machine. I had to edit the file ''$HOME/.ICAClient/wfclient.ini'' and set the entry for ''MouseSendsControlV'' to ''False''. This setting is helpful if you are used to copy/paste by selecting text and then clicking the mouse wheel on the target text entry field. however it is useless if you CAD software makes use of the middle mous click to rotat your Model :)  in order to get the middle mous button (wheel) to click, i had to modify the config of my citrix receiver on my client machine. I had to edit the file ''$HOME/.ICAClient/wfclient.ini'' and set the entry for ''MouseSendsControlV'' to ''False''. This setting is helpful if you are used to copy/paste by selecting text and then clicking the mouse wheel on the target text entry field. however it is useless if you CAD software makes use of the middle mous click to rotat your Model :) 
 ===== GPU Forwarding for KVM ===== ===== GPU Forwarding for KVM =====
 +First of all you need to enable VT-d and VT-x or they might be also called something like Intel Virtualization tech, in the BIOS. without these guys you won't be able to load the vfio module later on. 
 +
 Helpful links:  Helpful links: 
   * [[https://medium.com/@calerogers/gpu-virtualization-with-kvm-qemu-63ca98a6a172]]   * [[https://medium.com/@calerogers/gpu-virtualization-with-kvm-qemu-63ca98a6a172]]
Line 231: Line 233:
   cp -p windows10.img windows10HPrgs.img   cp -p windows10.img windows10HPrgs.img
 now let's boot it for the first time without the gpu installed, so we can make sure it boots all right:  now let's boot it for the first time without the gpu installed, so we can make sure it boots all right: 
-    qemu-system-x86_64 -enable-kvm -k de-ch -name windows -drive file=/bulk/vm/windows10HPrgs.img,if=virtio,format=raw,index=0 -m 6000 -device virtio-net-pci,netdev=net0,mac=52:54:00:01:00:01 -netdev tap,id=net0 -cdrom Win10_1803_English_x64.iso -drive file=virtio-win.iso,media=cdrom,index=1 -monitor tcp:127.0.0.1:5801,server,nowait -boot once=d -cpu host,kvm=off -smp cpus=8,cores=8 -monitor vc -drive if=pflash,format=raw,readonly,file=/usr/share/OVMF/OVMF_CODE.fd -drive if=pflash,format=raw,file=/bulk/vm/OVMF_VARS.windows10.fd -rtc base=localtime -balloon none -serial none -parallel none -M q35 -vnc :2+    qemu-system-x86_64 -enable-kvm -k de-ch -name windows -drive file=/bulk/vm/windows10HPrgs.img,if=virtio,format=raw,index=0 -m 9000 -device virtio-net-pci,netdev=net0,mac=52:54:00:01:00:01 -netdev tap,id=net0 -cdrom Win10_1803_English_x64.iso -drive file=virtio-win.iso,media=cdrom,index=1 -monitor tcp:127.0.0.1:5801,server,nowait -boot once=d -cpu host,kvm=off -smp cpus=8,cores=8 -monitor vc -drive if=pflash,format=raw,readonly,file=/usr/share/OVMF/OVMF_CODE.fd -drive if=pflash,format=raw,file=/bulk/vm/OVMF_VARS.windows10.fd -rtc base=localtime -balloon none -serial none -parallel none -M q35 -vnc :2
 during this first session I also uninstalled the Citrix VDA stuff, since this probably requires a reboot anyway during this first session I also uninstalled the Citrix VDA stuff, since this probably requires a reboot anyway
  
 now let's reboot with the gpu attached, so we can get to business now let's reboot with the gpu attached, so we can get to business
-  qemu-system-x86_64 -enable-kvm -k de-ch -name windows -drive file=/bulk/vm/windows10HPrgs.img,if=virtio,format=raw,index=0 -m 6000 -device virtio-net-pci,netdev=net0,mac=52:54:00:01:00:01 -netdev tap,id=net0 -cdrom Win10_1803_English_x64.iso -drive file=virtio-win.iso,media=cdrom,index=1 -monitor tcp:127.0.0.1:5801,server,nowait -boot once=d -cpu host,kvm=off -smp cpus=8,cores=8 -device vfio-pci,host=05:00.0 -monitor vc -drive if=pflash,format=raw,readonly,file=/usr/share/OVMF/OVMF_CODE.fd -drive if=pflash,format=raw,file=/bulk/vm/OVMF_VARS.windows10.fd -rtc base=localtime -balloon none -serial none -parallel none -M q35 -vnc :2+  qemu-system-x86_64 -enable-kvm -k de-ch -name windows -drive file=/bulk/vm/windows10HPrgs.img,if=virtio,format=raw,index=0 -m 9000 -device virtio-net-pci,netdev=net0,mac=52:54:00:01:00:01 -netdev tap,id=net0 -cdrom Win10_1803_English_x64.iso -drive file=virtio-win.iso,media=cdrom,index=1 -monitor tcp:127.0.0.1:5801,server,nowait -boot once=d -cpu host,kvm=off -smp cpus=8,cores=8 -device vfio-pci,host=05:00.0 -monitor vc -drive if=pflash,format=raw,readonly,file=/usr/share/OVMF/OVMF_CODE.fd -drive if=pflash,format=raw,file=/bulk/vm/OVMF_VARS.windows10.fd -rtc base=localtime -balloon none -serial none -parallel none -M q35 -vnc :2 
 +install the RGS sender and also the virtual mouse stuff 
 + 
 +to install the client on my ubuntu desktop sadly the ubuntu directory contains an rpm file with a script that executes an ''rpm -Uvh'' to install this package.. well, that won't work on ubuntu!  
 + 
 +so instead, let's use alien to convert the rpm and install it:  
 +<code> 
 +sudo apt install alien  
 +sudo alien rgreceiver*.rpm  
 +sudo dpkg -i rgreceiver*.deb 
 +</code> 
 +now to start rgreceiver we need to set the LD_LIBRARY_PATH and then start it:  
 +<code> 
 +export LD_LIBRARY_PATH=/opt/hpremote/rgreceiver/lib64:$LD_LIBRARY_PATH 
 +/opt/hpremote/rgreceiver/rgreceiver 
 +</code> 
 +to launch rgreceiver more conveniently by just typing ''rgreceiver'' in a new shell, let's add the ''LD_LIBRARY_PATH'' export and ''PATH'' export to our ''.bashrc'' to automatically execute each time you open a new shell:  
 +<code> 
 +echo 'export LD_LIBRARY_PATH=/opt/hpremote/rgreceiver/lib64:$LD_LIBRARY_PATH' >> ~/.bashrc 
 +echo 'export PATH=/opt/hpremote/rgreceiver:$PATH' >> ~/.bashrc 
 +</code> 
 +in the connection window that appears after starting the rgreceiver, you can click on settings and select to ''match receiver resolution'' which will start rgreceiver in fullscreen mode. to get out of fullscreen mode press ''leftCtrl + space'' and release space while keeping ''leftCtrl'' pressed, now press ''H'' and you will see a little menu at the top of the screen. to enter settings press ''M'' instead of ''H''  
 +==== connecting through a firewall ==== 
 +to enable access to your RDG service you need to allow both TCP and UDP traffic to Port 42966 to be routed to your RDG sender.  
 +==== First Impression ==== 
 +the performance is similar to Citrix.. the quality, lag and framerate seem very similar to Citrix, I'd say they are about the same. Installation is ALOT easier than citrix, the whole server infrastructure around it is not needed. Therefore i recon this setup will also run more stable in an environment like mine (not talking about big enterprise installs). price is okay as well. the big Downside is: it seems to be ment for fullscreen use only! the desktop size won't scale down if the receiver is run within a window and i haven't found a way yet to have the receiver in windowed mode without having scrollbars around it and needing to scroll. so that's definately a major downside of this solution.  
 + 
 +===== 3rd Try: Nice DCV ===== 
 +[[https://www.nice-software.com/download/nice-dcv-2016|Nice DCV]] is our third contestant. Priced at 300€ for a perpetual license this is the most expensive of the bunch. It needs a server component installed but it does not require any installation on the client side anymore.. it used to be RealVNC based but now they have moved to an HTML 5 solution which is nice as far as accessabiliy is concerned. The downside of this is, that HTML 5 clients are a lot more restricted for example when it comes to syncing your Clipboard. DCV requires you to press a button on their control panel to copy your local clipboard to the remote machine and vice versa.. To overcome that, there is a client that can be installed on workstations you use frequently, so you can enjoy both the comfort of a native client and the portability of a web based client which is absolutely awesome. well done!  
 + 
 +compared to the HP RGS, DCV supports on-the-fly desktop scaling, so when it is used in windowed mode on the native client or in a browser window, the desktop will resize as soon as you resize the viewer window. 
 + 
 +upon initial install you can use the software for 15 days free of chare to try it out. if you need more time, contact sales and they will give you a demo license that will allow you to try some more.  
 + 
 +so let's make another copy of our windows 10 image.. this time with rsync so we see the progress :)  
 +  rsync --progress windows10HPrgs.img windows10NiceDCV.img 
 +start it again withou the GPU attached to it for the first time:  
 +  qemu-system-x86_64 -enable-kvm -k de-ch -name windows -drive file=/bulk/vm/windows10NiceDCV.img,if=virtio,format=raw,index=0 -m 9000 -device virtio-net-pci,netdev=net0,mac=52:54:00:01:00:01 -netdev tap,id=net0 -cdrom Win10_1803_English_x64.iso -drive file=virtio-win.iso,media=cdrom,index=1 -monitor tcp:127.0.0.1:5801,server,nowait -boot once=d -cpu host,kvm=off -smp cpus=8,cores=8 -device vfio-pci,host=05:00.0 -monitor vc -drive if=pflash,format=raw,readonly,file=/usr/share/OVMF/OVMF_CODE.fd -drive if=pflash,format=raw,file=/bulk/vm/OVMF_VARS.windows10.fd -rtc base=localtime -balloon none -serial none -parallel none -M q35 -vnc :2 
 + 
 +download the latest dcv server and install it on the vm, then connect to ''https://vm-Ip:8443''  
 + 
 +if you want to use the client, simply download and install the package that matches your client os.  
 + 
 +so far i could only test this from a very slow connection in a hotel abroad and it is amazing how much faster both DCV and HP RGS are compared to normal RDP. DCV seems to perform very similar or maybe even a bit better than RGS with this extremely low bandwidth (too low really to use any sort of remote desktop with full HD, not to mention 4k).. so at the moment, except for the high priche tag, DCV is in my eyes the winner so far.  
 + 
 +===== retry DCV ===== 
 +dcv was running very unreliably.. it crashed often when i went to full screen with 4k and it had performance problems with resolutions in the 4k range.. i noticed however, that my method of copying the windows image and then first uninstall the HP stuff before i installed DCV was probably not the best as for example the display driver for the virtual display was not called DCV Display but something else, so i think there might have been some leftovers around which interfere with DCV.. so in order to get rid of all that i will retry the dcv installation with a new windows install.  
 +also i noticed that the performance of my virtual disk was lousy (10MB/s) so i am now using an LVM volume on a physical disk rather than an image on a zfs storage.  
 + 
 +here is the command line to start the vm:  
 + 
 +  qemu-system-x86_64 -enable-kvm -k de-ch -name windows -drive format=raw,file=/dev/vms/dcv,if=virtio,format=raw,index=0 -m 9000 -device virtio-net-pci,netdev=net0,mac=52:54:00:01:00:01 -netdev tap,id=net0 -cdrom Win10_1803_English_x64.iso -drive file=virtio-win.iso,media=cdrom,index=1 -monitor tcp:127.0.0.1:5801,server,nowait -boot once=d -cpu host,kvm=off -smp cpus=8,cores=8 -device vfio-pci,host=05:00.0 -monitor vc -drive if=pflash,format=raw,readonly,file=/usr/share/OVMF/OVMF_CODE.fd -drive if=pflash,format=raw,file=/bulk/vm/OVMF_VARS.windows10.fd -rtc base=localtime -balloon none -serial none -parallel none -M q35 -vnc :1 
 +   
 +as far as speed goes, this worked out a lot better, however i could not get DCV to run reliably.  
 + 
 +this time i've created a snapshot prior to installing DCV and i later cloned that snapshot to install RGS on a "plain" clone of the windows install in a second LVM volume. Here is my command to start the RGS VM:  
 +  qemu-system-x86_64 -enable-kvm -k de-ch -name windows -drive format=raw,file=/dev/vms/rgs,if=virtio,format=raw,index=0 -m 9000 -device virtio-net-pci,netdev=net0,mac=52:54:00:01:00:01 -netdev tap,id=net0 -cdrom Win10_1803_English_x64.iso -drive file=virtio-win.iso,media=cdrom,index=1 -monitor tcp:127.0.0.1:5801,server,nowait -boot once=d -cpu host,kvm=off -smp cpus=8,cores=8 -monitor vc -drive if=pflash,format=raw,readonly,file=/usr/share/OVMF/OVMF_CODE.fd -drive if=pflash,format=raw,file=/bulk/vm/OVMF_VARS.windows10.fd -rtc base=localtime -balloon none -serial none -parallel none -M q35 -vnc :1 -device vfio-pci,host=05:00.0 
 + 
 +RGS runs smooth and windows feels reasonably fast too with the SSD as storage behind it. however, When the vm is left running over night usually the next morning it hangs or is so slow that one often can't even shut it down properly. I don't know why that is, i've deactivated all power saving features of windows, screen blanking etc. but I still have this issue. The machine hangs the same when I connect vie windows RDP. Also in those cases where i can still see something, I usually get alot of horizontal lines as artefacts on the screen both with RGS and RDP. The only solution is then to shut down the VM and start it again, a Reboot (instead of a power cycle) will make the VM fast again but the Artefacts remain. so there is definately something wrong with my VM here. This needs further investigation!  
 + 
 +==== LVM stuff ==== 
 +just as a quick side note: here is how you create and revert to a snapshot in lvm.  using dd you can also copy a snapshot into a new lvm volume if the new volume is at least the size of the snapshotted one.  
 + 
 +create a snapshot:  
 +  lvcreate -s -L 50G -n win10base /dev/vms/dcv 
 +revert to a snapshot:  
 +  lvconvert --merge /dev/vms/win10base 
 + 
 +==== using a vm with just the forwarded GPU ==== 
 +this is the ultimate goal. so far i could not get this to work, whenver i booted a vm without a emulated graphics card i could no longer connect to it. and it was hard to debug as i had no "local" screen to connect to. however, i recently connected a monitor to the GPU and found out, that the reason why i could not connect was in fact very simple: once the emulated graphics adapter is no longer presence, the pci-id of the emulated NIC changes and windows detects it as a new ethernet adapter despite it having the same MAC. So the vm actually got a DHCP assigned ip rather than the fixed one i had configured. I could then connect to the dhcp assigned ip and set the same fixed ip on the "second" interface which now works fine.  
 + 
 +to start a vm without a emulated graphics card, use ''-nographic -vga none'' in your command line.  
 + 
 + qemu-system-x86_64 -enable-kvm -k de-ch -name windows -drive format=raw,file=/dev/vms/dcv,if=virtio,format=raw,index=0 -m 9000 -device virtio-net-pci,netdev=net0,mac=52:54:00:01:00:01 -netdev tap,id=net0 -cdrom Win10_1803_English_x64.iso -drive file=virtio-win.iso,media=cdrom,index=1 -monitor tcp:127.0.0.1:5801,server,nowait -boot once=d -cpu host,kvm=off -smp cpus=8,cores=8 -monitor vc -drive if=pflash,format=raw,readonly,file=/usr/share/OVMF/OVMF_CODE.fd -drive if=pflash,format=raw,file=/bulk/vm/OVMF_VARS.windows10.fd -rtc base=localtime -balloon none -serial none -parallel none -M q35 -vga none -nographic -device vfio-pci,host=05:00.0 
 + 
 +==== stability issues ==== 
 +i go to the point where whenever i booted my DCV VM the HOST system rebooted or rather reset! Finally I found a workaround that fixed this issue: I bought a little dummy hdmi display emulator which is basically an HDMI plug that you plug into the output of the graphics card and it emulates a display. Since i had no adapters readily available for my K1200 i switched to a GTX 1050Ti which i had in my other pc. So far it works the same as the K1200 did. DCV now starts and is usable same as RGS was before. However i Still have the issue, where after about a day or so, when i log back into my VM i get horizontal black lines across my remote desktop which move around as i re-size the window but stay stady "above" the various windows of Windows. This happend with RGS as well as with DCV and even when I use windows RDP! Usually the vm also becomes increasingly unresponsive when i try to re-connect after having this issue.  
 + 
 +I have now updated my Bios hosts bios to the latest version (it was actually quite outdated already) and have also enabled Coherency support for the IOMMU stuff in the bios. I usually don't like changing two parameters at once to sovle an issue, but for now lets just try it as it takes at least a day to reproduce the problem.. i can then roll back the bios setting and see if it was the update alone that resolved the issue should my current trials be successful.  
 + 
 +===== Re-Try RDP + passthrough ===== 
 +the key to a smooth user experience as far as framerate etc. goes seems to be to use AVC444 which on Windows supports NVENC (using the Nvidia Card to encode the video stream for RDP)  
 + 
 +to enable this on the windows side:  
 +  - Open Run (Win+R). Type gpedit.msc 
 +  - Computer Configuration -> Administrative Templates -> Windows Components -> Remote Desktop Services -> Remote Desktop Session Host -> Remote Session Environment 
 +    - Enable "Prioritize H.264/AVC 444 Graphics mode for Remote Desktop..." 
 +    - Enable "Configure H.264/AVC hardware encoding for Remote Desktop..." 
 + 
 + 
 +on the client side with xrdp, get the latest version (i worked with 3.0) from https://www.freerdp.com/ where they offer pre-built binaries for many distributions. the binaries are built on a daily basis and include the stable branch. with that, use the ''/gfx:AVC444'' or ''/gfx-h264:AVC444'' option to connect to the server  
 +  /opt/freerdp-nightly/bin/xfreerdp /gfx-h264:AVC444 /u:"Pascal Suter" /v:192.168.168.101 /dynamic-resolution 
 + 
 +to verify that nvenc is being used, open the task manager and go to the performance tab. the GPU should show video encoding workload. 
  
  
  • gpu_accelerated_vm_with_kvm.1538286859.txt.gz
  • Last modified: 30.09.2018 07:54
  • by Pascal Suter