Ah, stupid me. I've been banging my head on the wall for almost an hour trying to figure out why Chrome is not picking up my web manifest. I recently modified the server to return the strictest possible CSP headers for my use case. I used default-src 'none' but I didn't explicitly set the manifest-src property.

Service worker does not successfully serve the manifest start_url.

So there you go. If you get these kinds of errors in Lighthouse audit be sure to check your CSP headers.