You can use Directory API with ADC, but you can't use the Domain Wide delegation (used with the `with_subject`) with a user account, or with GCE/GKE not "prepared" for these GSuite scopes.
My opinion on the authentication part is: It's hard for beginners, and when it's hard, the "humans" go to the simplest solution, and, in terms of security, means the less secure.
It's my day to day toil to educate my colleagues on this and I really pray for a simplification about the security.
Impersonation is powerful, but, as you said, it's boilerplate for few things... And you alternative with KMS (or Secret manager) is great. I think you can share this story, it will be very interesting!!
The problem of impersonation is that only Java and Python Google Auth libraries implement them. The other languages have to implement this manually... Another 200 lines of boilerplate code!!
Anyway, I know that a IAM v2 is in progress and I put a lot on hope in this new evolution (I know a little about some features and it's exciting!). Maybe in 2021!!