In regards to a website that you want to keep private, in a situation where you can't restrict/allow to ip's for your users and password protection cannot be implemented. But you can have your users use a custom user-agent.
For a first line of defence I am thinking about denying all user-agents and allowing one ambiguous one.
The Apache documentation says:
Access control by User-Agent is an unreliable technique, since the User-Agent header can be set to anything at all, at the whim of the end user.
But I figure in the reverse fashion of denying all and only accepting one it could be quite effective.
My questions are 1.) giving the situation does this seem like a good solution 2.) is there a way for people to figure out which user-agents are allowed on a given server?
The documentation which you quoted is warning specifically against what you are suggesting.
Unless your traffic is passing through an encrypted (HTTPS) channel (seems that's probably not the case if you don't have the option to use any other means of authentication) a casual observer anywhere between your client network and the destination network will be able to see which user agents are "accepted" by your service - i.e. you have no guarantee of privacy.