Cloudfront how to failover between multiple custom origins?

I have a Cloudfront distribution and two origin domains that are not hosted in AWS.

I want to failover to the second domain if a file is not found in the first domain.

I've created two behaviours, but if the file is 404 in the first behaviour Cloudfront does not continue to check further behaviours.

What I want to happen is if the first behaviour is 404 then Cloudfront tries the next behaviour.

How could I make this work?