Intel Reboot Issues With Meltdown-Spectre Patches
The efforts to mitigate the threat of the Meltdown and Spectre exploits is officially WORSE than the threat itself. Many Intel systems are randomly and spontaneously rebooting after installing Intel Spectre 2 patches. No shit. Here is our continuing coverage of the Intel Spectre Reboot Issue!
Article Update History
On 11 January 2018the WSJ reported that Intel was quietly asking their cloud computing customers to hold off installing Meltdown and Spectre patches because “the patches have bugs of their own“. Specifically, there were three bugs in the microcode patches they released.
In a blog post posted on the same day, Intel Executive Vice President and General Manager of the Intel Data Center Group, Navin Shenoy confirmed that Intel received reports of “higher system reboots” after applying those updates.
Basically, these systems would randomly and spontaneously reboot after installing those patches. Not something you want your computer to do, never mind servers that cater to tens or hundreds of thousands of users.
not confirmed Spectre 2 mitigations as the cause in the Coffee Lake, Kaby LakeSkylakeIvy Bridge and Sandy Bridge platforms that are also affected.
In fact, Intel shared that “The progress we have made in identifying a root cause for Haswell and Broadwell will help us address issues on other platforms. Please be assured we are working quickly to address these issues.”
All of the systems suffering from spontaneous reboot issues were running on Haswell, Broadwell, SkylakeKaby Lake and the latest Coffee Lake CPUs. Workstation and server CPUs based on Ivy Bridge and Sandy Bridge were also affected, but thankfully not their desktop brethren.
On 8 February 2018Intel revealed that some of the microcode updates that they suspected were buggy, were actually not buggy. They include :
- The Intel Skylake H/S/U/Y Desktop Processors
- The Intel Xeon E3-1200 v5 Processor Family (Skylake)
We prepared the full list of CPUs affected by the buggy Intel Spectre 2 patches, but it is a VERY LONG LIST with 801 CPUsso we split them into three sections.
As you can see, many more server and workstation CPUs are affected than desktop and mobile CPUs combined. That’s because Intel prioritised the patching of their server and workstation CPUs, over desktop and mobile CPUs.
When he first posted on the spontaneous reboot issue, Navin said that Intel was working to “understand, diagnose and address this reboot issue“.
In his latest update, he shared that Intel had already issued an early version of the new microcode updates to their partners for tests, and will release them “once that testing has been completed“.
These new microcode updates basically have Spectre 2 mitigations removed. This will restore stability to the affected Intel CPUs, while Intel fixes the problems in those mitigations.
On 7 February 2018Navin Shenoy announced that Intel has released “production microcode updates for several Skylake-based platforms” to their OEM customers and industry partners, with more platform updates “in coming days“.
The schedule was updated on 12 February and 20 February with more details, including production (final), pre-beta and beta versions of the new Intel Spectre microcode updates.
While Intel initially advised end-users to “apply updates” from system and operating system providers, they have now changed their guidance, as of 22 January 2018 :
- We recommend that OEMs, Cloud service providers, system manufacturers, software vendors and end users stop deployment of current versions on the below platforms, as they may introduce higher than expected reboots and other unpredictable system behavior.
- We also ask that our industry partners focus efforts on testing early versions of the updated solution for Broadwell and Haswell we started rolling out this weekend, so we can accelerate its release. We expect to share more details on timing later this week.
- For those concerned about system stability while we finalize the updated solutions, we are also working with our OEM partners on the option to utilize a previous version of microcode that does not display these issues, but removes the Variant 2 (Spectre) mitigations. This would be delivered via a BIOS update, and would not impact mitigations for Variant 1 (Spectre) and Variant 3 (Meltdown).
Please note that there has been no actual recorded threat or attack using the Meltdown or Spectre exploits. The damage, or risk of damage, every time your system or server spontaneously reboot is FAR WORSE than the (currently) non-existent threat of a Meltdown or Spectre exploit.
Therefore, we recommend that you DO NOT apply any microcode update for your Intel systemif you are using any Intel processor manufactured since 2011.
If you have already applied the latest Intel Spectre microcode update, and are affected by spontaneous reboots; you should upgrade to the new firmware (if they are available), or revert to the older firmware.
If you like our work, you can help support our work by visiting our sponsorsparticipating in the Tech ARP Forumsor even donating to our fund. Any help you can render is greatly appreciated!