I wouldn't let that stop you: adding a service work won't in any way degrade the experience of Safari users. When used for caching, it's pure progressive enhancement.
Sure, some browsers will have a more robust experience in adverse network conditions, but the experience once a page has loaded will be identical. It's also difficult to accidentally make the worker a hard dependency of a site, since even supported browsers will load the site without the worker on first visit.
I might be dumb but can't you kind of polyfill this behavior by making your SPA cache itself on load and response to everything as events, like shown in the repo linked in this article?
Sure, some browsers will have a more robust experience in adverse network conditions, but the experience once a page has loaded will be identical. It's also difficult to accidentally make the worker a hard dependency of a site, since even supported browsers will load the site without the worker on first visit.