I have an install of Exchange 2010 without an Edge Transport Server. The only way I can get it to receive e mail from the internet is by allowing anonymous access to the receive connector but this in turn means the server can be used as a relay. Testing has proved that this is indeed the case.
I've Googled this extensively but can't seem to find any solution, other than installing the Edge Transport role somewhere which unfortunately isn't an option.
I'm hopeful that I'm just missing something obvious - previously I've always used Exchange 2003.
Allowing anonymous access in the receive connector does not mean it's an open relay. Configuring anonymous access simply means your server will accept a connection without the sending server authenticating. Relay means your server will send email to domains despite your mail system not being authoritative for that domain. So long as you do not specify the sending server can relay, it can only deliver email to the domains your server is configured to be authoritative for. Those are the accepted domains. If it's working otherwise, you configured it incorrectly.
If you are allowing direct internet email to your Hub Transport server, be sure to configure antispam agents on the receiving HT server(s). http://technet.microsoft.com/en-us/library/bb201691.aspx
See my answer here:
What ports to open for mail server?
In order to receive email, you must accept anonymous connections on port 25.
For your connector for port 587, do not allow anonymous connections.
And make sure you NEVER check externally secured unless you know exactly what you are doing. That is how open relays are setup.