Reverse proxying multiple different dockerized HTTPS services

I have two services, both neatly dockerized, each with their own docker-compose.yml.

Each services has its own domain name.

Both services uses HTTPS and have Lets Encrypt certificates.

Both services are completely self contained and listen on 80 and 443.

.--[http]-[https]--.    .--[http]-[https]--.
|                  |    |                  |
|    Service A     |    |    Service B     |
|                  |    |                  |
'------------------'    '------------------'

How do I set up a reverse proxy so that I can launch both services on the same host?

.--------------[http]-[https]--------------.
|                                          |
|              REVERSE PROXY               |
|                                          |
'---+--------+---------------+--------+----'
    |        |               |        |
   http     https           http     https
   req.     req.            req.     req.
   domainA  domainA         domainB  domainB
     |        |              |        |
     V        V              V        V
.--[http]-[https]--.    .--[http]-[https]--.
|                  |    |                  |
|    Service A     |    |    Service B     |
|                  |    |                  |
'------------------'    '------------------'

I would like to keep the services self contained. I.e. I would like to avoid extracting the certificates from the service containers.

From my experiments, it seems like nginx can't forward HTTPS requests without the certs, although it seems like it should be possible to achieve this. Perhaps with some other reverse proxy software?

I have control over both services docker-compose.yml. I can change ports and so on if needed.

Ideally I would like to find a simple daemon that accepts a config that looks something like

[service a]
domain: domainA.tld
localPort: 8080

[service b]
domain: domainB.tld
localPort: 8081

I've searched like crazy but haven't found any such tool.