One other big contributor to app size we've noticed at Khan Academy: extensive use of value types, particularly ones whose fields require more than a couple words of storage.
The big picture observation is that for value types with storage larger than a few words, several instructions must be emitted per call and per storage word because they cannot be passed-by-value in registers. And Swift often emits more calls than you can see (e.g. thunks, protocol witnesses, weak sentinels, etc). This is not new to Swift—the same requirement exists when passing large C values around—but we use value types a lot more in Swift for various reasons, so the issue becomes more salient.
In terms of remediation, I audited our app for all structs larger than a few words. I just did this manually; there were something like 120 structs to look at. For each, I converted it to a class then evaluated the impact on generated code size. Only four structs had meaningful impact on generated code size (to the tune of ~13MB), and happily, they were fully immutable, so they retained their value semantics even when converted to classes. If they had not already been fully immutable, I would have had to spend some time either adapting the classes to achieve value semantics, or adapting their clients to tolerate reference semantics.
Then I audited our app for all enums larger than a few words. These can be made pass-by-reference by using Swift’s `indirect` feature, which implicitly boxes associated storage. We had one enum for which this made a substantial difference, to the tune of several MB.
Then I had to make sure runtime performance hadn’t been too badly damaged by all the new dynamic allocations and indirections. In the end, I observed nothing noticeable. We don’t have formal repeatable performance tests, though—it would have been interesting to see the impact on those.
C++ has this issue, too; it largely handles it by using reference arguments when consuming large stack values. In the future, it’s possible for Swift to optimize many cases (especially intramodule) where this occurs by allowing deeper stack frames to reference values types stored in parent stack frames when it can prove it’s safe. Rust has a lot of fanciness here you might find interesting!
There is some work ongoing to improve this situation in Swift 4.0. Take/copy/destroy operations for enums are now outlined and shared between call sites, and a further improvement is in the works to pass large structs indirectly even when they are otherwise loadable (no address-only fields).
Another recent improvement is copy-on-write existentials. There's no code size win here, but it improves runtime performance by avoiding copying the payload when passing existentials by value.
I think they have some more infrastructure that can audit stuff automatically in the same way you did, but I'm basing that off of a conversation I had and I'm not familiar enough with the codebase to find it; maybe it's just this set of tests.
We ran into this issue in the rewrite as well. We ended up having to convert all of our models from structs to classes (similar to your case they were immutable, so it wasn't a huge difference).
The big picture observation is that for value types with storage larger than a few words, several instructions must be emitted per call and per storage word because they cannot be passed-by-value in registers. And Swift often emits more calls than you can see (e.g. thunks, protocol witnesses, weak sentinels, etc). This is not new to Swift—the same requirement exists when passing large C values around—but we use value types a lot more in Swift for various reasons, so the issue becomes more salient.
In terms of remediation, I audited our app for all structs larger than a few words. I just did this manually; there were something like 120 structs to look at. For each, I converted it to a class then evaluated the impact on generated code size. Only four structs had meaningful impact on generated code size (to the tune of ~13MB), and happily, they were fully immutable, so they retained their value semantics even when converted to classes. If they had not already been fully immutable, I would have had to spend some time either adapting the classes to achieve value semantics, or adapting their clients to tolerate reference semantics.
Then I audited our app for all enums larger than a few words. These can be made pass-by-reference by using Swift’s `indirect` feature, which implicitly boxes associated storage. We had one enum for which this made a substantial difference, to the tune of several MB.
Then I had to make sure runtime performance hadn’t been too badly damaged by all the new dynamic allocations and indirections. In the end, I observed nothing noticeable. We don’t have formal repeatable performance tests, though—it would have been interesting to see the impact on those.
C++ has this issue, too; it largely handles it by using reference arguments when consuming large stack values. In the future, it’s possible for Swift to optimize many cases (especially intramodule) where this occurs by allowing deeper stack frames to reference values types stored in parent stack frames when it can prove it’s safe. Rust has a lot of fanciness here you might find interesting!