I'm using a dedicated server for my app. Now I got a webmail app that I also want to work on the same server. I want the webmail app to work using a subdomain from the site, for example if my site uses the domain mysite.com, then I want the webmail app to use webmail.mysite.com.
I basically want to add a new website in IIS and map the domain webmail.mysite.com to it so that any requests with the host name webmail.mysite.com go the that website, how can I do that?
Your main website will typically be the default website, that means that all requests without a specific host-header will be fielded by that site. I'm assuming that you only have one IP assigned to that server. If not, add a comment and I will explain what to do if you have a spare IP address (it's even easier).
What you need to do, is in the IIS admin tool, to create a second website. If you then edit the properties of that website, go to the tab that says bindings (sorry, I don't have it installed locally currently and I'm explaining by memory, and my system was in Spanish anyway), and modify your host name to be webmail.mysite.com.
Then, all requests that ask for that specific (sub)domain will be served from the second site. Be aware that your DNS (or your local hosts file, if you're still testing) need to be configured so that webmail.mysite.com correctly point to your IP address.
if you need to point your subdomain to another IP, add an A record to your DNS zone pointing to that IP