I have a few Server 2008 Hyper-V hosts all working fine with Virtual Machine Manager. I got a new server and installed Server 2008 R2 and was unable to manage it with VMM; so I upgraded to VMM R2. Now, I am unable to manage any of my old Server 2008 machines, but the Server 2008 R2 machine works great.
Is there a way to use System Center Virtual Machine Manager in a mixed Server2008/Server2008R2 environment?
The mixed environment you describe is indeed a supported environment for VMM 2008 R2 (this FAQ lists the supported virtualization platforms).
I would suggest reviewing the steps outlined in this blog post on TechNet to make sure you didn't miss anything (particularly the post-upgrade update of each of the managed entities with the new agent).
Additional details regarding the VMM upgrade process can be found in the VMM 2008 R2 Deployment Guide in VMM TechNet.