At new customer I ound on an 2003 install the situation described in the title. A production database is installed in the instance used by SBS for monitoring the system.
Is it recommended to keep things in such a situation, or am I supposed to use an ad hoc SQLServer instance?
Is there some official recommendation about the case?
A SQL Instance is just an instance of the MSSQL Engine. Obviously you can create multiple instances on a server to segregate things. In this case I think the concern you have is really over the name of the instance as the OS uses this instance for other internal processes.
If you take the Single Responsibility Principle into account it makes a lot of sense to either spin up another instance to store production databases so they are segregated from internal utility databases. Or better yet spin up a different server all together to store production data.
There aren't any overall best practices when it comes to instances and combining roles and responsibilities, often just preferences. Microsoft would probably advise you not to use that instance for other data.
Personally, I would recommend the same, spin up another instance and keep your data there. Use the internal utility instances for just those purposes.