GrapheneOS Memory Tagging Notifications Lead To a Large Uptick In Reports To 3rd Party App Developers

GrapheneOS users on 8th/9th gen Pixels are making a massive contribution to getting memory corruption bugs in the open source ecosystem thanks to the nice crash report notifications created by our hardware memory tagging feature. One of the latest fixes:

https://github.com/mullvad/mullvadvpn-app/pull/6727/files

Someone should report C.GoString being broken in Go's cgo. Reading an entire page before and after an object that's passed is incredibly broken undefined behavior. They're relying on memory allocation and memory protection having page granularity at a low level which is wrong.

GrapheneOS users have repeatedly found memory corruption bugs in WireGuard-based apps on Android. It's possible most of these are largely caused by memory corruption in the Go runtime because they're playing fast and loose with memory accesses outside the bounds of objects...

GrapheneOS always uses heap memory tagging for every process in the base OS with a single exception (camera HAL). Our implementation is guaranteed to catch all small/linear overflows and even use-after-free until a certain number of allocation cycles for that size class occur.

It has a 14/15 chance to catch any other kind of heap corruption for the standard system allocators.

Since it catches memory corruption as the read or write occurs, it produces very useful tracebacks for devs. We provide them to users with a UI to copy it to report bugs to devs.

Our users on 8th/9th gen Pixels can enable it for all user installed apps via Settings > Security & privacy > Exploit protection > Memory tagging. Use the per-app toggle for incompatible apps and report the bugs to them. It's not used for most user installed apps by default yet.

Apps can mark themselves as compatible with memory tagging to opt-in to having it on GrapheneOS. We also have an app compatibility database where we can add known compatible apps to enable it by default and incompatible ones which skips them with the global default opt-in toggle.

Hardware memory tagging in the security-focused asymmetric mode has very low overhead. Latent memory corruption bugs occurring during regular use in many apps is the only blocker for us enabling it by default for every user installed app as we already do for all base OS apps.

One of the memory corruption bugs in Go being caught by memory tagging on GrapheneOS was reported to Go in September 2018 and is still unfixed today:

https://github.com/golang/go/issues/27610

Reading outside bounds of objects from other languages is a serious memory safe violation, not benign.