AWS Application Load Balancer ALBs support HTTP/2, and one would think we could switch over to these and immediately enjoy the benefits of HTTP/2.
Is there any possible a way to terminate HTTP/2 on ALB, so that the web servers don't need to be configured to support HTTP/2?
They already don't need to be.
On Application Load Balancer (ELB/2.0) that's how it works out of the box. The ALB speaks to the instances using HTTP/1.1 while speaking HTTP/2 to the browser if the browser supports it, otherwise HTTP/1.x.
ELB doesn't support HTTP2, ALB does, and so does CloudFront. You could put ELB into TCP mode and support HTTP2 using your web server, but that removes the protection you get from an ELB (it protects you again a bunch of attacks, DDOS, SYN Flood, etc).
If you need HTTP/2 your options, in the order I think best, are: