Differences

This shows you the differences between two versions of the page.

--- spectre_and_meltdown_fixes_-_release_dates_for_linux_distros [19.01.2018 15:35] – Pascal Suter
+++ spectre_and_meltdown_fixes_-_release_dates_for_linux_distros [24.10.2018 21:40] (current) – [Performance Impact] Pascal Suter
@@ Line 1: / Line 1: @@
-====== spectre and Meltdown fixes - release dates for linux distros ======
+====== Spectre and Meltdown fixes ======
-Unfortunately the [[https://meltdownattack.com|Meltdown and Spectre]] vulnerabilities became Public too early. So now we are all keen on getting updates for our servers to work around the respective design issues in modern CPU's ASAP. Originally, the official announcement was planned for Jan. 9, so we can expect that by then most OS's including most Linux Distributions will have a fix in their repos which can simply be installed by running whatever your regular update routine is. That's
+This page should give you a short overview of what is currently available to update your Intel based server or Workstation to get as good as possible patched against Spectre ( CVE 2017-5753 (Bounds Check Bypass / Variant 1) and CVE 2017-5715 (Branch Target Injection / Variant 2)) and Meltdown (CVE 2017-5754 (rogue data cache load / Variant 3)).
+===== What's needed? =====
+**this update will need a reboot of your server for sure, don't just update and continue to work without rebooting**
+in general you need to update your kernel to the latest versions provided by your distribution of choice. by now, pretty much  any distribution should have released patches.
+Here is how it's done...
   apt-get update && apt-get upgrade
-in ubuntu and
+... in ubuntu and ..
   yum  update
-in RedHat and CentOS distributions.
+.. in RedHat and CentOS distributions.
-**this update will need a reboot of your server for sure, don't just update and continue to work without rebooting**
+this will protect you against Variant 1 and Variant 3 vlunerabilities. In order to also protect against Variant 2, you further need to update the CPU's microcode (basically the CPU's firmware). Instructions on how to do that follow further down on this page
-However, now that the news is out, every OS vendor tries to be as fast as possible in pushing out patches, so possibly fixes will be released earlier.
+===== Different Distributions and their update status =====
-here is a list of links with information about updates available from linux distributions i care about:
+here is a list of links with information about updates available from linux distributions I care most about:
   * [[https://security-tracker.debian.org/tracker/CVE-2017-5754|Debian]] stretch is fixed as of kernel version 4.9.65-3**+deb9u2**
   * [[https://wiki.ubuntu.com/SecurityTeam/KnowledgeBase/SpectreAndMeltdown|Ubuntu]] (fixes for all maintained versions available as of Jan. 9th)
@@ Line 19: / Line 26: @@
 for further information read those pages or check out the [[https://meltdownattack.com/#faq-fix|meltdown webpage link section]]
-===== Minimalistic Fix on CentOS 7.4 =====
+===== Microcode Update - Yes it's necessary too! =====
-usually all you need todo today is simply update your linux to the latest patch level and you should be good. as of now (18.1.18) you might need to update the microcode (firmware) for your CPU manually, as some distributions (RHEL for example) have removet the intel firmware from their repos for the moment, due to some [[https://newsroom.intel.com/news/intel-security-issue-update-addressing-reboot-issues/|stability issues]].
+After you have installed the latest OS updates, your system should be protected against Variant 1 and 3, in order to protect against Variant 2, you also need to install a newer microcode (firmware of the CPU).
+There are three ways to get to the latest microcode updates:
+==== 1: Wait for your linux distribution to include it ====
+Usually the microcode package is also a package in your distributions repository and is updated during a normal os upgrade. However, there where some [[https://newsroom.intel.com/news/intel-security-issue-update-addressing-reboot-issues/|stability issues]] with the microcodes released by Intel early this year ( around 18.1.18) so that for example RedHat removed them from their repos.
+after that, intel pulled the microcode updates back from their own webpage as well and went back to alpha and beta testing. they have now released a bunch of updated microcodes for their cpu's. generally what we see is, that they start with the newest cpu's and work their way back. I would expect, that the final microcode releases will make it into the respective microcode packages in your linux distribution. the Easiest way is just to run updates regularly and eventually you should get the latest microcode update for your cpu that will enable the Variant 2 fix.
+==== 2: Download the latest microcode from intel and install it manually ====
+besides waiting for your distro to include the update for you, you can also manually download the latest microcode package from intel (includes microcodes for all intel cpus in one package) and install that on your computer manually. Here is, how that's done:
+The latest officially released Microcodes can be found on the intel downloadcenter page. currently [[https://downloadcenter.intel.com/download/27431/Linux-Processor-Microcode-Data-File?v=t|this download here]] is the latest. there should be a banner at the top linking to a newer version, should this link no longer lead to the latest version. So far they have updated the link target together with new releases to always point to the latest release.
+  - download the package of your choice to your System: <code>
+cd /root/
+wget <url>
+</code>
+  - now move your existing microcode package to another location, so you can move it back in case you have these stability issues that some users had with the new ones. then unpack and load the new microcode: <code>
+cd /lib/firmware/
+mv intel-ucode /root/intel-ucode.old
+tar xvf /root/microcode-20180108.tgz
+echo 1 > /sys/devices/system/cpu/microcode/reload
+</code>
+you can double check if your microcode was loaded using <code>dmesg | grep microcode</code>.
+you should now be up to date with the latest patches for all three Variants of the Spectre & Meltdown vulnerability.
+==== 3: Update your BIOS ====
+BIOS packages from the mainboard vendor should also include the latest microcode for the CPU upon release of the BIOS package. sometimes you can get microcode updates earlier through a BIOS update than you can get them through the intel microcode package download. However, if your mainboard producer supplies no mor BIOS updates or if they have a slow release cycle, the intel package might be the faster solution for you. Because Linux only loads the microcode from its own package when it's newer than the version loaded by the bios, a BOIS update that gets you the new Microcode will work even with old Linux Versions that might be out of maintenance. However, if you are using such a Distro, you probably aren't worried too much about security anyway and your system is hopefully only running in a well protected internal network with trusted users.. in that case, don't worry about Meltdown :)
+If you do worry about meltdown and want to upgrade the microcode through a new bios, you can find a list of the latest BIOS releases that contain Variant 2 fixes in their included microcode on this [[https://www.intel.com/content/www/us/en/support/articles/000026622/server-products.html|Intel-SA-00088 for Intel® Server Boards]] overview page. The list is updated as soon as new bioses become available.
+===== Understanding the microcode release notes =====
+Intel's Microcode releasenotes are somewhat cryptic. Here is how to read the following lines of the current release notes: <code>
+-- Updates upon 20171117 release --
+IVT C0		(06-3e-04:ed) 428->42a
+SKL-U/Y D0	(06-4e-03:c0) ba->c2
+BDW-U/Y E/F	(06-3d-04:c0) 25->28
+HSW-ULT Cx/Dx	(06-45-01:72) 20->21
+Crystalwell Cx	(06-46-01:32) 17->18
+BDW-H E/G	(06-47-01:22) 17->1b
+HSX-EX E0	(06-3f-04:80) 0f->10
+SKL-H/S R0	(06-5e-03:36) ba->c2
+HSW Cx/Dx	(06-3c-03:32) 22->23
+HSX C0		(06-3f-02:6f) 3a->3b
+BDX-DE V0/V1	(06-56-02:10) 0f->14
+BDX-DE V2	(06-56-03:10) 700000d->7000011
+KBL-U/Y H0	(06-8e-09:c0) 62->80
+KBL Y0 / CFL D0	(06-8e-0a:c0) 70->80
+KBL-H/S B0	(06-9e-09:2a) 5e->80
+CFL U0		(06-9e-0a:22) 70->80
+CFL B0		(06-9e-0b:02) 72->80
+SKX H0		(06-55-04:b7) 2000035->200003c
+GLK B0		(06-7a-01:01) 1e->22
+</code>
+  * ''-- Updates upon 20171117 release --'' This means, that this package contains the following updates since the last release. so only the microcodes for the cpus mentioned here where updated, everything else is identical to the last release.
+  * ''SKL-U/Y D0	(06-4e-03:c0) ba->c2''
+    * ''SKL'' stands for Sky-Lake
+    * ''06-4e-03'' is actually the most useful part, it tells you what cpu that is in cpu-family, model and stepping. you can get this information from ''/proc/cpuinfo'' with this command: <code>
+grep -P "^(cpu family)|(model\s*:)|(stepping)" /proc/cpuinfo | tail -3
+</code>. ''06'' is the family, ''4e'' is the model in HEX format (use google or a scientific calculator to convert if you are lazy :)) and ''03'' is the stepping.
+    * the last part ''ba->c2'' is the relevant part of the version number that changed. For this specific Skylake CPU the Spectre Patch is supposed to be in releases ''0xc2'' or newer, so this one here contains the patch. Sadly the list with all these releases is under NDA, so i can't share it here. But in general you can expect everything that is released starting with the current package to have the fix in place.
+  * by the way, ''06-4e-03'' is also the filename of that microcode.
+  * rather than browsing through the entire history of the release notes you can also check the version of a specific microcode file using this command: ''iucode_tool -l intel-ucode/06-4f-01''
+===== Minimalistic Fix on CentOS 7.4 =====
 Should you, for some reason, not be able or willing to run a full update, I have here a minimalistic fix for your centos:
@@ Line 48: / Line 118: @@
 reboot
 </code>
-===== Microcode Update - Yes it's necessary too! =====
-Your system should now be patched against Variant 1 and 3, in order to protect against Variant 2, you also need to install a newer microcode. Keep in mind however, that there where some stability issues with the microcodes released until today (18.1.18) so that for example RedHat removed them from their repos. however, this step can be reverted should you have issues with the new microcode (sudden reboots where reported), so if you can afford a few crashes at worst, you should still give it a try and only revert when the system gets too unstable for you to work with:
-  - go to [[https://downloadcenter.intel.com/download/27431/Linux-Processor-Microcode-Data-File?v=t|the intel download center]] and download the latest microcode package.
-  - <code>
-cd /root/
-wget <url copied from downloadcenter>
-</code>
-  - now move your existing microcode package to another location, so you can move it back in case you have these stability issues that some users had with the new ones. then unpack and load the new microcode: <code>
-cd /lib/firmware/
-mv intel-ucode /root/intel-ucode.old
-tar xvf /root/microcode-20180108.tgz
-echo 1 > /sys/devices/system/cpu/microcode/reload
-</code>
-you can double check if your microcode was laded using <code>dmesg | grep microcode</code>.
-you should now up to date with the latest patches for all three Variants of the Spectre & Meltdown vulnerability.
 ===== Test-Tools =====
   * [[https://github.com/speed47/spectre-meltdown-checker]]
 ===== Performance Impact =====
-the fix for all this works in a way that it may affect the system performance negatively. Different sources claim different results reaching from no impact at all up to a 30% slowdown. As always, Benchmarks are probably not too representative for your realworld experience. In order to find out what the difference in performance is, you can simply disable the workaround on a patched kernel to run your workload once with and once without the patch. the workaround can be disabled by passing the boot option ''pti=off''.
+the fix for all this works in a way that it may affect the system performance negatively. Different sources claim different results reaching from no impact at all up to a 30% slowdown. As always, Benchmarks are probably not too representative for your realworld experience. In order to find out what the difference in performance is, you can simply disable the workaround on a patched kernel to run your workload once with and once without the patch.
+In CentOS (and probably other linux distributions as well) the workarounds can be enabled or disabled without a reboot using these commands:
+<code>
+echo 0 > /sys/kernel/debug/x86/pti_enabled
+echo 0 > /sys/kernel/debug/x86/ibpb_enabled
+echo 0 > /sys/kernel/debug/x86/ibrs_enabled
+echo 0 > /sys/kernel/debug/x86/retp_enabled
+</code>
+by default all three fixes are enabled, if you want to disable them permanently (=on every boot) you can add these three options to your kernel command line:
+<code>
+noibrs noibpb nopti noretp spectre_v2=off
+</code>
+the last ''spectre_v2=off'' is redhat/CentOS specific and might be redundant with the previous ones.