I'm looking for advice/best-practices/+ve/-ve or other idea's on deployment and maintenance of many (50) Web Services and Web Service Applications on IIS across an application server cluster of 10 machines for Service Orientated Client (WinApp) Software.
Currently we have one solution that contains many Web Service Projects. Software is developed separately and later integrated when ready for release. This involves the integration manager (unfamiliar with the code), adding the project to the main solution and ensuring everything works, e.g. references etc. aren't broken. This means we end up with one site in IIS that serves many web services.
An recent alternative idea, is to have multiple sites - possibly grouping related web services into one site, but keeping other web services in different sites, although this increases maintenance (as there are more sites) it means parts of the software can be updated separably and tested, by the dev who wrote the code.
May be relevant: We use source control for the code, have many users (around 1million hits a day) and can tolerate 0 downtime during working hours.
Currently to update a single web service, the application server must be taken out of the cluster, updated, then added back into the cluster.
Is there a more efficient/flexible, yet still maintainable way of going about this?
I can share some thoughts from what we are currently doing at the new Stack Overflow datacenter that might give you some ideas. There is a staging server to build to first as test. The staging server is a replica of all the production servers. When a new site is added, after testing I will sync the applicationhost.config with a simple powershell script. For further updates to a site that is already deployed we use cruise control for .NET which allows for automated builds across the web servers. I played with IIS Shared configuration but found it to be a bit buggy.
To automate taking a server out of the cluster you might be able to add a health check to your load balancer which will take it out the cluster automatically.
I like your idea of grouping the IIS sites by the developer or dev teams. We do something similar and it allows the devs to own their own code. As admin you can still own the servers but not get in the way of the developers.
I also just heard an interesting talk about continuous deployment you (pushing to production frequently) which you can read a little bit about here. You might be able to get some ideas from that.
Ultimately you have to design what works best for your company and culture.