More ways for firmware to screw you
Mar. 17th, 2012 10:18 am
mjg59
Some of my recent time has been devoted to making our boot media more Mac friendly, which has entailed rather a lot of rebooting. This would have been fine, if tedious, except that some number of boots would fall over with either a clearly impossible kernel panic or userspace segfaulting in places that made no sense. Something was clearly wrong. Crashes that shouldn't happen are generally an indication of memory corruption. The question is how that corruption is being triggered. Hunting that down wasn't terribly easy.
My first thought was that we were possibly managing to load the kernel over a region used by UEFI code. UEFI defines two types of code - boot services and runtime services. While runtime services code and data must be preserved by the OS, in theory boot services code and data is available to the OS once the firmware has exited. In practice, that's not true. It seemed entirely possible that the kernel might be ending up on top of some of that boot services code or data and getting trodden on. Grub now has code to avoid putting the kernel on boot services, so testing the latest code seemed like a good plan. But no, crashes still happened.
That pretty much ruled out the bootloader. My next thought was that executing some of the firmware code was triggering a write to some other memory that contained the kernel. Josh Boyer suggested the next trick, which was to try marking the kernel read-only to see whether anything was hitting it. x86 lets you mark pages as read-only - any attempt to write to them should take a fault. UEFI functions are executed in the context of the kernel, so share the same page tables. That let me rule this out, since everything still went just as wrong and I wasn't taking an extra fault first.
However, at this point I was reasonably happy that it wasn't the kernel itself being overwritten - faults were occurring in userspace code as well. That was a pretty strong indication that what was happening was continuing to happen once userspace had started, so it wasn't a direct response to a firmware call. I made sure of that by stubbing out all the calls that could be triggered after kernel initialisation, and saw the same failures. Once all attempts to be clever have failed, it's time to just start using brute force. The kernel lets you reserve areas of RAM by passing arguments like memmap=0xlength$0xstart to block length bytes starting at start from being used. It took a while, but I finally found a 256MB range that made a difference - reserving it resulted in the machine booting reliably, letting the OS use it resulted in occasional crashes.
Definite progress. Comparing that memory range to the EFI memory map was helpful. There were several blocks of UEFI boot services data present there, which really seemed like too much of a coincidence. By reserving each of them in turn, I'd traced it down to a single 31MB region of boot service data - that is, memory reserved by the firmware for use by the UEFI boot services. Per spec, this is available to the OS once the boot environment has been exited. Nothing other than the OS should be touching this after boot, but something clearly was. Tracking down what was far easier than I expected, although the first attempt was a failure. Setting it read-only should have triggered a fault, but didn't. That was rather confusing. But, rather than give up, I patched the kernel to fill the region with 0xff at kernel init. Then I booted the system, read it back and looked for values that weren't 0xff. I got this:
How do we fix this? Unsure. With luck disconnecting the UEFI driver in the bootloader should quiesce the hardware, but without testing I'm not sure of that yet. For now it's just another example of firmware managing to break expectations in deeply strange ways.
My first thought was that we were possibly managing to load the kernel over a region used by UEFI code. UEFI defines two types of code - boot services and runtime services. While runtime services code and data must be preserved by the OS, in theory boot services code and data is available to the OS once the firmware has exited. In practice, that's not true. It seemed entirely possible that the kernel might be ending up on top of some of that boot services code or data and getting trodden on. Grub now has code to avoid putting the kernel on boot services, so testing the latest code seemed like a good plan. But no, crashes still happened.
That pretty much ruled out the bootloader. My next thought was that executing some of the firmware code was triggering a write to some other memory that contained the kernel. Josh Boyer suggested the next trick, which was to try marking the kernel read-only to see whether anything was hitting it. x86 lets you mark pages as read-only - any attempt to write to them should take a fault. UEFI functions are executed in the context of the kernel, so share the same page tables. That let me rule this out, since everything still went just as wrong and I wasn't taking an extra fault first.
However, at this point I was reasonably happy that it wasn't the kernel itself being overwritten - faults were occurring in userspace code as well. That was a pretty strong indication that what was happening was continuing to happen once userspace had started, so it wasn't a direct response to a firmware call. I made sure of that by stubbing out all the calls that could be triggered after kernel initialisation, and saw the same failures. Once all attempts to be clever have failed, it's time to just start using brute force. The kernel lets you reserve areas of RAM by passing arguments like memmap=0xlength$0xstart to block length bytes starting at start from being used. It took a while, but I finally found a 256MB range that made a difference - reserving it resulted in the machine booting reliably, letting the OS use it resulted in occasional crashes.
Definite progress. Comparing that memory range to the EFI memory map was helpful. There were several blocks of UEFI boot services data present there, which really seemed like too much of a coincidence. By reserving each of them in turn, I'd traced it down to a single 31MB region of boot service data - that is, memory reserved by the firmware for use by the UEFI boot services. Per spec, this is available to the OS once the boot environment has been exited. Nothing other than the OS should be touching this after boot, but something clearly was. Tracking down what was far easier than I expected, although the first attempt was a failure. Setting it read-only should have triggered a fault, but didn't. That was rather confusing. But, rather than give up, I patched the kernel to fill the region with 0xff at kernel init. Then I booted the system, read it back and looked for values that weren't 0xff. I got this:
00000000 ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff |................| * 001568a0 ff ff ff ff ff ff ff ff ff ff ff ff 84 00 00 00 |................| 001568b0 00 20 a7 ac 46 00 00 00 00 00 00 00 00 00 06 01 |. ..F...........| 001568c0 c2 0b 0c 00 ff ff ff ff ff ff ff ff ff ff ff ff |................| 001568d0 ff ff 0a 04 f0 03 82 0d 40 00 00 00 ff ff ff ff |........@.......| 001568e0 ff ff 00 21 00 36 9a 80 ff ff ff ff ff ff 00 7e |...!.6.........~| 001568f0 00 09 43 48 41 2d 47 75 65 73 74 01 04 02 04 0b |..CHA-Guest.....| 00156900 16 32 08 0c 12 18 24 30 48 60 6c 2d 1a 0e 18 1a |.2....$0H`l-....| 00156910 ff ff 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................| 00156920 00 00 00 00 00 00 00 dd 09 00 10 18 02 00 10 01 |................| 00156930 00 00 dd 1e 00 90 4c 33 0e 18 1a ff ff 00 00 00 |......L3........| 00156940 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................| 00156950 00 00 bd ea f8 b3 ff ff ff ff ff ff ff ff ff ff |................| 00156960 ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff |................| * 022fb000That's a lot of 0xffs (around 31MB of them) with one small section that contains an 802.11 probe packet with the SSID of the hospital across the road from my house. Apple support network booting off wireless networks. It seems that the firmware brought up the wireless card, associated with this network (it's the only public one nearby) and then left the card DMAing packets into RAM. The read-only page attribute only applies to CPU-initiated accesses, so it could do this without triggering a page fault. It also explained why it was so random - whether memory corruption occurred would depend on whether a packet appeared between that memory being used by the OS and the kernel reinitialising the wireless card. It certainly explains why I couldn't reproduce it when I left the machine repeatedly rebooting on the bus home.
How do we fix this? Unsure. With luck disconnecting the UEFI driver in the bootloader should quiesce the hardware, but without testing I'm not sure of that yet. For now it's just another example of firmware managing to break expectations in deeply strange ways.
You always have the best war stories
Date: 2012-03-17 11:30 pm (UTC)Re: You always have the best war stories
Date: 2012-03-18 06:17 pm (UTC)http://os.inf.tu-dresden.de/vfiasco/
http://os.inf.tu-dresden.de/L4Re/
http://os.inf.tu-dresden.de/L4Re/doc/
http://demo.tudos.org/
enjoy :)
kind regards
Remo
Re: You always have the best war stories
Date: 2012-03-18 07:30 pm (UTC)Nasty
Date: 2012-03-18 12:14 am (UTC)Um...
Date: 2012-03-18 12:50 am (UTC)That section of memory is used by BootSrv (specifically part of the boot device chooser or the wireless firmware I'm guessing, I'm kinda curious to know if thats right), which gets data punted into it (by the wireless firmware?), but the boot-time wireless network handling doesn't necessarily notice that BootSrv has ended so it should cut that shit out right now, right?
Re: Um...
Date: 2012-03-18 01:03 am (UTC)Loading the wireless driver?
Date: 2012-03-18 01:50 am (UTC)Re: Loading the wireless driver?
Date: 2012-03-18 01:57 am (UTC)Truly stupid
Date: 2012-03-18 03:24 am (UTC)My bet is that Apple simply 'knows' not to use any UEFI memory until all devices have been reinitialized by the OS - or they even have a way to pass that memory region back to the OS.
Can we have Linux reserve all such regions until we walked the list of known devices and put them in a known state? At least for Macs?
Dirk
Re: Truly stupid
Date: 2012-03-18 03:31 am (UTC)Re: Truly stupid
Date: 2012-03-18 04:05 am (UTC)-hpa
Bootloader really does not dig us out properly.
Date: 2012-03-18 10:03 pm (UTC)Next is lock the boot services memory until after the kernel knows it has passed a safe point in init that should have caught up with rogues normally by reseting everything that can be. To reset a card you don't need its firmware thank you pci spec. Once the card is reset it will forget all dma transfers it had been requested todo.
Remember after resetting everything most likely will lose like the EFI provided file-system driver and other things as well. So when you do the reset you better not need anything EFI is providing.
Reason for this way what comes to my mind is firmware updates so cure one driver get a new problem driver.
So the wifi driver is still on what other dma transfers are still on we don't exactly know. This is why I see the flush everything point is the only way. Really there is no reason why the Linux kernel could not be altered to be directly loaded by EFI.
I/OMMU
Date: 2012-03-18 08:35 am (UTC)UEFI: was it worth it?
Date: 2012-03-18 08:36 am (UTC)But from a detached engineering perspective: was it worth it pushing the equivalent of a full blown OS (in terms of complexity) down to the BIOS level?
How many nasty bugs await us from there, given that the BIOS makers won't have the chance to afford the QA typically given to OSes (and that is barely sufficient!)?
Re: UEFI: was it worth it?
Date: 2012-03-18 01:43 pm (UTC)What we will end up with is a perennial stream of randomly failing hardware with no chance to fix it. Your only hope of getting your driver fixed is the hardware manufacturer, who cannot be arsed to provide it since once you've bought their product, you are no longer their customer.
Re: UEFI: was it worth it?
Date: 2012-03-18 01:47 pm (UTC)Re: UEFI: was it worth it?
Date: 2012-03-19 09:44 am (UTC)I am not a HW expert but I have read that UEFI drivers could be available to the OS for basic services. Is it possible/required or forbidden?
Re: UEFI: was it worth it?
Date: 2012-03-19 11:33 am (UTC)Re: UEFI: was it worth it?
Date: 2012-03-18 04:30 pm (UTC)And that seems to be the core point.
Make it their problem.
UEFI scribbling into your memory? Trying to call into freed boot services? Invalid ACPI tables? Missing or inconsistent ASPM settings? => Device is broken => Give it back and demand a refund.
The sad part, that this strategy (while simple and in my eyes even the correct answer to the problem) will not work. Most people will never notice how many workarounds an OS or other software is using to make everything work. And if Linux would print out "your device is broken, get the vendor to repair it", the people would probably blame Linux and not the vendor. And then the vendors will start to sell "Windows only devices" / "MacOS only devices" and any standard violations will be "its supposed to do that". :-(
Re: UEFI: was it worth it?
Date: 2012-03-18 06:58 pm (UTC)BUT... the hardware vendor can only possibly test the configurations that they expect people to actually use. Once you start installing your own OSes on the hardware then you are using a unsupported configuration and this means that you are on your own for testing these sorts of things. If you find bugs maybe the vendor will care, maybe not.
The ultimate solution, of course, is to buy hardware from vendors that actually expect you to run the configurations you want. Meaning if you want to run Linux on a laptop then you buy a laptop the vendor that has Linux as a _expressly_ supported configuration.
I am not trying to downplay the attempts of above kernel developer struggling to get Linux to work properly on Apple hardware. What he is doing is very valuable as it is finding problems and fixing them and this will benefit more then just Apple hardware users... I am just saying that for the average user that wants their stuff to 'just work' purchasing Apple laptop to run Linux is probably the last thing you'd want to do. There are a lot of Linux-friendly vendors out there.
Re: UEFI: was it worth it?
Date: 2012-03-18 08:13 pm (UTC)...buy hardware from vendors that actually expect you to run the configurations you want.
That is the problem, not the solution.
Vendors only test "boots Windows" and ship it, because its cheaper to not do any more test. And they do this, because they know, they will get away with it, because "nobody" is complaining about it.
If someone is claiming to implement UEFI, then they better really do implement UEFI. And test against the UEFI spec. And not implement something-not-quite-unlike-UEFI that is just able to boot Windows and/or MacOS.
Nobody is demanding that the vendors try every possible software configuration. That is obvious impossible, as with any somewhat open interface. But what they do need to do (and should be able to do): Test their side of the interface. The problem here (and in all the other cases that count as "the firmware screwing you) is that this implementation is breaking this interface. And not that the combination of this implementation and linux is breaking. If Linux is violating the UEFI (or any other spec), then fix Linux. But if the vendor is violating the spec that he is claiming to support, then it is his duty to fix this. Independent of what this violation is breaking.
And buying from a Linux supporting vendor will not help you, if that vendor only tests "boots Linux". Because the next kernel version can change the order of device initializations or its address space or ... And then it will break.
I think it would be terrible step backwards to switch from standards (even bloated ones like UEFI) to vendor approved OS lists. Because that would be the consequence of your suggestion. Then the vendors can cut even the few remaining later fixes, because the device was always only sold as "works with Windows version XYZ". Want to upgrade to Windows 8? Better buy a new board.
It might still work for Microsoft, because they can exert enough pressure to the vendors because of there market share (-> see secure boot and who will probably be the only firm that gets their keys into any hardware). But even with "Linux-friendly" vendors, suspect you be stuck to a specific kernel soon. Just look how "easy" it is to update the kernel (or system libraries) on an android phone.
Re: UEFI: was it worth it?
Date: 2012-03-20 05:22 pm (UTC)1. This is a mac. All bets are off. If you want something standards compliant buy something else.
2. UEFI "windows" systems generally behave much better (compared to macs), and from a vendor perspective, why is windows not a good test of UEFI? Windows is by far the most widely used UEFI supporting OS. Much of the spec is open to interpretation. Your best bet it so support the most widely used interpretation.
Re: UEFI: was it worth it?
Date: 2012-03-20 05:26 pm (UTC)Really? They're pretty much equivalently bad, in my experience. I've found plenty of EFI bugs that are present on various Windows machines but not on Macs.
Re: UEFI: was it worth it?
Date: 2012-03-26 10:17 pm (UTC)http://support.apple.com/kb/HT1237
http://refit.sourceforge.net/info/
Macs are using EFI 1.10
Any assumption about boot services behavior in Macs even though it's Intel platform can't be made for Windows PCs.
It's a nice find though on errant behavior of EFI. (Notice not blaming Apple here), it could be the UEFI UNDI driver by that IHV in the Mac itself not properly behaving.
Which firmware is this?
Date: 2012-03-18 02:19 pm (UTC)Because I've got a MacBookPro7,1, and I'm seeing a "EFI Firmware update 2.5" in the MacOSX software update, which I was just about to install when I saw your post.
"This update enables Lion Recovery from an Internet connection on MacBook Pro (13-inc, Mid 2010) models."
Will installing this break my Linux and render it unstable/unusable?
Re: Which firmware is this?
Date: 2012-03-18 02:22 pm (UTC)Re: Which firmware is this?
Date: 2012-03-18 05:54 pm (UTC)Off-topic: Apple is full of weirdness. Just two days ago decided to update to Lion, but it wouldn't let me at first, because MacOS likes to have 128MB unallocated between partitions. Even booting from DVD (Snow Leopard/Lion) was impossible. Had to use target disk mode to shrink a partition. Oh well - at least target disk mode is a pretty cool feature for a firmware to have.
About UEFI
Date: 2012-03-18 08:59 pm (UTC)It is very annoying and time consuming, but the best way to deal with this totally different system is to actually learn the UEFI and its quirks, and use the tools to see what's going on.
Re: About UEFI
Date: 2012-03-18 09:47 pm (UTC)Which is pretty scary, since it is HUGE.
using IOMMU to prevent DMA access
Date: 2012-03-18 10:10 pm (UTC)no subject
Date: 2012-03-20 05:57 pm (UTC)some operation will allocate boot-service type memory if the operation is invoked.
Unfortunately, I have seen that some motherboard vendor shipped suspect custom modification which caused more problem than the original Aptio F/W.
I think it is a good reason to explain why it should be self-sustained although which can't prevent buggy F/W from crashing your OS entirely.
Nobody's perfect ...
Date: 2012-03-21 03:54 pm (UTC)Re: Nobody's perfect ...
Date: 2012-03-21 03:59 pm (UTC)Re: Nobody's perfect ...
Date: 2012-03-21 04:21 pm (UTC)Re: Nobody's perfect ...
Date: 2012-03-21 04:27 pm (UTC)Possible solution to UEFI
Date: 2012-04-10 03:32 pm (UTC)I had a dream about the UEFI situation last night in which I found a solution to the problem. I am not an engineer or computer scientist, so the dream my have been influenced by the 50mg of benadryl I took before bed.
In my dream I asked someone if the major server and desktop distributions like Fedora, Centos, Debian, Ubuntu, OpenSuse created a properly signed version of their current kernel and a basic, standard userland of utilities -- Perhaps also signed X or Wayland code as well. Could the rest of the userland applications be installed and run without signing if they were written to a universal and signed implementation of the LSB? Sort of a Linux version of Windows Compatibility mode where applications that could run under all supporting distros even after the signed kernel and other components were upgraded to newer versions.
I don't remember how or if the person answered me, but I woke up wishing that such a solution could be found and I would not have to by Windows to run Ubuntu in the future.
ExitBootServices event
Date: 2014-09-11 11:44 am (UTC)So, Apple's SNP (Simple Network Protocol) driver is buggy then. The UEFI Driver Writer's Guide explains that network drivers need to register an EVT_SIGNAL_EXIT_BOOT_SERVICES event. When ExitBootServices() is called, the firmware signals this event, and the handler / callback function for the event is invoked. In the function, the driver needs to stop all in-flight DMA, exactly in order to prevent the situation that you describe.
Apple's driver apparently didn't do this at the time you wrote this post. edk2/OvmfPkg/VirtioNetDxe/ does (as an open source example), mapping the "DMA transfer" concept to "virtio transfer".
Hurray, my MacBook can boot from wlan now
Date: 2015-03-24 08:19 pm (UTC)That problem hit my machine, I guess.
I've got a MacBookPro6,2 running debian and a homebrew linux-3.18.8
I use suspend-to-disk to boot OSX sometimes for gaming etc. and that was solid rock.
Last week I hat to upgrade to yosesmite and since then resume to linux is screwed up:
(cold boot!)
[ 12.631120] PM: Starting manual resume from disk
[ 12.631124] PM: Hibernation image partition 254:3 present
[ 12.631125] PM: Looking for hibernation image.
[ 12.631633] PM: Image signature found, resuming
[ 12.637379] PM: Preparing processes for restore.
[ 12.637385] Freezing user space processes ... (elapsed 0.001 seconds) done.
[ 12.638550] PM: Loading hibernation image.
[...]
[ 12.642240] PM: Basic memory bitmaps created
[ 12.719237] PM: Using 3 thread(s) for decompression.
[ 12.719237] PM: Loading and decompressing image data (439192 pages)...
[ 12.840040] PM: Image loading progress: 0%
[ 13.846117] PM: Image loading progress: 10%
[ 14.215101] PM: Image loading progress: 20%
[ 14.564934] PM: Image loading progress: 30%
[ 14.921618] PM: Image loading progress: 40%
[ 15.269658] PM: Image loading progress: 50%
[ 15.621066] PM: Image loading progress: 60%
[ 15.984618] PM: Image loading progress: 70%
[ 16.349571] PM: Image loading progress: 80%
[ 16.653994] PM: Invalid LZO uncompressed length
[ 16.654056] PM: Read 1756768 kbytes in 3.93 seconds (447.01 MB/s)
[ 16.654604] PM: Error -1 resuming
[ 16.654609] PM: Failed to load hibernation image, recovering.
[ 16.710503] PM: Basic memory bitmaps freed
[ 16.710508] Restarting tasks ... done.
[ 16.715622] PM: Hibernation image not present or could not be loaded.
More evidence from a dmesg file before the upgrade:
cold boot before yosesmite upgrade
grep 'enabling device' dmesg.3.18.8.ok
[ 0.375133] pci 0000:00:1a.0: enabling device (0000 -> 0001)
[ 0.375447] pci 0000:00:1d.0: enabling device (0000 -> 0001)
[ 0.653059] pcieport 0000:00:1c.3: enabling device (0000 -> 0003)
[ 1.812399] tg3 0000:02:00.0: enabling device (0000 -> 0002)
[ 11.846913] snd_hda_intel 0000:00:1b.0: enabling device (0000 -> 0002)
[ 11.847358] snd_hda_intel 0000:01:00.1: enabling device (0000 -> 0002)
[ 11.856775] nouveau 0000:01:00.0: enabling device (0006 -> 0007)
[ 11.871167] bcma-pci-bridge 0000:03:00.0: enabling device (0000 -> 0002)
and after
grep 'enabling device' dmesg.3.18.8.osxupgraded.ffcked
[ 0.649500] pcieport 0000:00:1c.3: enabling device (0000 -> 0003)
[ 17.268327] snd_hda_intel 0000:00:1b.0: enabling device (0000 -> 0002)
[ 17.268657] snd_hda_intel 0000:01:00.1: enabling device (0000 -> 0002)
[ 17.275151] nouveau 0000:01:00.0: enabling device (0006 -> 0007)
Looks like, some devices are now 'warm' on boot.
I don't remember that my machine was able to boot via wlan, now it can.
Is there a way to disable this 'feature'? It looks like there was no firmware upgrade:
DMI: Apple Inc. MacBookPro6,2/Mac-F22586C8, BIOS MBP61.88Z.0057.B0F.1112091028 12/09/11
Any progress solving that problem?
regards,
chris
I got bitten by this one too
Date: 2015-10-10 03:32 am (UTC)We hit this exact issue at VMware trying to boot ESXi on Macs, around the same time you hit it. It's pretty obvious with a debug build of ESXi because on such builds, we fill all of the supposedly free memory with a bit pattern and occasionally check that the bit pattern hasn't been disturbed.
We filed problem report 9303938 with Apple about it back in 2012, but I don't seem to have the ability to look at that report's status, so I don't know how/if Apple replied to it.
For ESXi 6.0, I implemented a workaround in our bootloader that involved turning off bus mastering on the offending device. Ugh.
It does seem like Apple has fixed this it in recent firmware, FWIW. I just upgraded my group's Mac mini 6,2 from B03 to B09, and it doesn't seem to require the workaround anymore.
Still here in 2016!
Date: 2016-02-02 07:27 pm (UTC)The tale continues ....
Date: 2016-04-12 02:55 pm (UTC)So I fixed it installing linux on it, just running on IGD.
Testing hibernate/resume, it worked but after n-times I got:
"memory size mismatch".
WHAT????
After some try and error, I can say this:
Booting on battery may give extra 4k free RAM !!!!
dmesg | grep totalpages:
powered:
[ 0.000000] On node 0 totalpages: 1013414
battery:
[ 0.000000] On node 0 totalpages: 1013415
I was not able to reproduce this with the other macbook .... until I did a SMC reset.
Another way to waste some RAM is using the alt-key-bootloader on startup ... and/or if there was a bootable usb-device plugged in.