====== Spectre and Meltdown fixes ====== This page should give you a short overview of what is currently available to update your Intel based server or Workstation to get as good as possible patched against Spectre ( CVE 2017-5753 (Bounds Check Bypass / Variant 1) and CVE 2017-5715 (Branch Target Injection / Variant 2)) and Meltdown (CVE 2017-5754 (rogue data cache load / Variant 3)). ===== What's needed? ===== **this update will need a reboot of your server for sure, don't just update and continue to work without rebooting** in general you need to update your kernel to the latest versions provided by your distribution of choice. by now, pretty much any distribution should have released patches. Here is how it's done... apt-get update && apt-get upgrade ... in ubuntu and .. yum update .. in RedHat and CentOS distributions. this will protect you against Variant 1 and Variant 3 vlunerabilities. In order to also protect against Variant 2, you further need to update the CPU's microcode (basically the CPU's firmware). Instructions on how to do that follow further down on this page ===== Different Distributions and their update status ===== here is a list of links with information about updates available from linux distributions I care most about: * [[https://security-tracker.debian.org/tracker/CVE-2017-5754|Debian]] stretch is fixed as of kernel version 4.9.65-3**+deb9u2** * [[https://wiki.ubuntu.com/SecurityTeam/KnowledgeBase/SpectreAndMeltdown|Ubuntu]] (fixes for all maintained versions available as of Jan. 9th) * [[https://forum.proxmox.com/threads/meltdown-and-spectre-linux-kernel-fixes.39110/|Proxmox VE]] debian based virtualization environment, fixed for Version 4 and 5 as explained in the linked forum post. * [[https://lists.centos.org/pipermail/centos-announce/2018-January/022696.html|CentOS]] (started syncing to mirrors on Jan 4) kernel-3.10.0-693.11.6.el7.x86_64.rpm and related packages fix the bug * [[https://access.redhat.com/errata/RHSA-2018:0007|RedHat Enterprise Linux RHEL 7]] (released 3.Jan) * [[https://www.suse.com/c/suse-addresses-meltdown-spectre-vulnerabilities/|Suse]] for further information read those pages or check out the [[https://meltdownattack.com/#faq-fix|meltdown webpage link section]] ===== Microcode Update - Yes it's necessary too! ===== After you have installed the latest OS updates, your system should be protected against Variant 1 and 3, in order to protect against Variant 2, you also need to install a newer microcode (firmware of the CPU). There are three ways to get to the latest microcode updates: ==== 1: Wait for your linux distribution to include it ==== Usually the microcode package is also a package in your distributions repository and is updated during a normal os upgrade. However, there where some [[https://newsroom.intel.com/news/intel-security-issue-update-addressing-reboot-issues/|stability issues]] with the microcodes released by Intel early this year ( around 18.1.18) so that for example RedHat removed them from their repos. after that, intel pulled the microcode updates back from their own webpage as well and went back to alpha and beta testing. they have now released a bunch of updated microcodes for their cpu's. generally what we see is, that they start with the newest cpu's and work their way back. I would expect, that the final microcode releases will make it into the respective microcode packages in your linux distribution. the Easiest way is just to run updates regularly and eventually you should get the latest microcode update for your cpu that will enable the Variant 2 fix. ==== 2: Download the latest microcode from intel and install it manually ==== besides waiting for your distro to include the update for you, you can also manually download the latest microcode package from intel (includes microcodes for all intel cpus in one package) and install that on your computer manually. Here is, how that's done: The latest officially released Microcodes can be found on the intel downloadcenter page. currently [[https://downloadcenter.intel.com/download/27431/Linux-Processor-Microcode-Data-File?v=t|this download here]] is the latest. there should be a banner at the top linking to a newer version, should this link no longer lead to the latest version. So far they have updated the link target together with new releases to always point to the latest release. - download the package of your choice to your System: cd /root/ wget - now move your existing microcode package to another location, so you can move it back in case you have these stability issues that some users had with the new ones. then unpack and load the new microcode: cd /lib/firmware/ mv intel-ucode /root/intel-ucode.old tar xvf /root/microcode-20180108.tgz echo 1 > /sys/devices/system/cpu/microcode/reload you can double check if your microcode was loaded using dmesg | grep microcode. you should now be up to date with the latest patches for all three Variants of the Spectre & Meltdown vulnerability. ==== 3: Update your BIOS ==== BIOS packages from the mainboard vendor should also include the latest microcode for the CPU upon release of the BIOS package. sometimes you can get microcode updates earlier through a BIOS update than you can get them through the intel microcode package download. However, if your mainboard producer supplies no mor BIOS updates or if they have a slow release cycle, the intel package might be the faster solution for you. Because Linux only loads the microcode from its own package when it's newer than the version loaded by the bios, a BOIS update that gets you the new Microcode will work even with old Linux Versions that might be out of maintenance. However, if you are using such a Distro, you probably aren't worried too much about security anyway and your system is hopefully only running in a well protected internal network with trusted users.. in that case, don't worry about Meltdown :) If you do worry about meltdown and want to upgrade the microcode through a new bios, you can find a list of the latest BIOS releases that contain Variant 2 fixes in their included microcode on this [[https://www.intel.com/content/www/us/en/support/articles/000026622/server-products.html|Intel-SA-00088 for Intel® Server Boards]] overview page. The list is updated as soon as new bioses become available. ===== Understanding the microcode release notes ===== Intel's Microcode releasenotes are somewhat cryptic. Here is how to read the following lines of the current release notes: -- Updates upon 20171117 release -- IVT C0 (06-3e-04:ed) 428->42a SKL-U/Y D0 (06-4e-03:c0) ba->c2 BDW-U/Y E/F (06-3d-04:c0) 25->28 HSW-ULT Cx/Dx (06-45-01:72) 20->21 Crystalwell Cx (06-46-01:32) 17->18 BDW-H E/G (06-47-01:22) 17->1b HSX-EX E0 (06-3f-04:80) 0f->10 SKL-H/S R0 (06-5e-03:36) ba->c2 HSW Cx/Dx (06-3c-03:32) 22->23 HSX C0 (06-3f-02:6f) 3a->3b BDX-DE V0/V1 (06-56-02:10) 0f->14 BDX-DE V2 (06-56-03:10) 700000d->7000011 KBL-U/Y H0 (06-8e-09:c0) 62->80 KBL Y0 / CFL D0 (06-8e-0a:c0) 70->80 KBL-H/S B0 (06-9e-09:2a) 5e->80 CFL U0 (06-9e-0a:22) 70->80 CFL B0 (06-9e-0b:02) 72->80 SKX H0 (06-55-04:b7) 2000035->200003c GLK B0 (06-7a-01:01) 1e->22 * ''-- Updates upon 20171117 release --'' This means, that this package contains the following updates since the last release. so only the microcodes for the cpus mentioned here where updated, everything else is identical to the last release. * ''SKL-U/Y D0 (06-4e-03:c0) ba->c2'' * ''SKL'' stands for Sky-Lake * ''06-4e-03'' is actually the most useful part, it tells you what cpu that is in cpu-family, model and stepping. you can get this information from ''/proc/cpuinfo'' with this command: grep -P "^(cpu family)|(model\s*:)|(stepping)" /proc/cpuinfo | tail -3 . ''06'' is the family, ''4e'' is the model in HEX format (use google or a scientific calculator to convert if you are lazy :)) and ''03'' is the stepping. * the last part ''ba->c2'' is the relevant part of the version number that changed. For this specific Skylake CPU the Spectre Patch is supposed to be in releases ''0xc2'' or newer, so this one here contains the patch. Sadly the list with all these releases is under NDA, so i can't share it here. But in general you can expect everything that is released starting with the current package to have the fix in place. * by the way, ''06-4e-03'' is also the filename of that microcode. * rather than browsing through the entire history of the release notes you can also check the version of a specific microcode file using this command: ''iucode_tool -l intel-ucode/06-4f-01'' ===== Minimalistic Fix on CentOS 7.4 ===== Should you, for some reason, not be able or willing to run a full update, I have here a minimalistic fix for your centos: - download the necessary update packages mkdir -p /opt/meltdown cd /opt/meltdown for p in kernel-abi-whitelists-3.10.0-693.11.6.el7.noarch.rpm kernel-debug-3.10.0-693.11.6.el7.x86_64.rpm kernel-debug-devel-3.10.0-693.11.6.el7.x86_64.rpm kernel-devel-3.10.0-693.11.6.el7.x86_64.rpm kernel-doc-3.10.0-693.11.6.el7.noarch.rpm kernel-headers-3.10.0-693.11.6.el7.x86_64.rpm kernel-tools-3.10.0-693.11.6.el7.x86_64.rpm kernel-tools-libs-3.10.0-693.11.6.el7.x86_64.rpm kernel-tools-libs-devel-3.10.0-693.11.6.el7.x86_64.rpm perf-3.10.0-693.11.6.el7.x86_64.rpm python-perf-3.10.0-693.11.6.el7.x86_64.rpm kernel-3.10.0-693.11.6.el7.src.rpm kernel-3.10.0-693.11.6.el7.x86_64.rpm; do wget http://mirror.centos.org/centos/7.4.1708/updates/x86_64/Packages/$p; done - create a repository: createrepo . - add your repository to yum mydir=`pwd` cat > /etc/yum.repos.d/CentOS-meltdown.repo < - run the update and reboot the machine: yum update reboot ===== Test-Tools ===== * [[https://github.com/speed47/spectre-meltdown-checker]] ===== Performance Impact ===== the fix for all this works in a way that it may affect the system performance negatively. Different sources claim different results reaching from no impact at all up to a 30% slowdown. As always, Benchmarks are probably not too representative for your realworld experience. In order to find out what the difference in performance is, you can simply disable the workaround on a patched kernel to run your workload once with and once without the patch. In CentOS (and probably other linux distributions as well) the workarounds can be enabled or disabled without a reboot using these commands: echo 0 > /sys/kernel/debug/x86/pti_enabled echo 0 > /sys/kernel/debug/x86/ibpb_enabled echo 0 > /sys/kernel/debug/x86/ibrs_enabled echo 0 > /sys/kernel/debug/x86/retp_enabled by default all three fixes are enabled, if you want to disable them permanently (=on every boot) you can add these three options to your kernel command line: noibrs noibpb nopti noretp spectre_v2=off the last ''spectre_v2=off'' is redhat/CentOS specific and might be redundant with the previous ones.