I think that it is easy to see some benefits of separating BOOT and DATA into two partitions or drives; it makes for each backup.
However, I was always under the impression that it is better to separate your BOOT from your DATA drives to increase performance. For this reason, I had 4 drives, configured in two mirrors (BOOT & DATA).
Because drives are so large now, I'd prefer to take two terabyte drives, mirror them, and partition them into BOOT and DATA.
Am I losing any performance benefit here?
Traditionally boot drives also contained paging files (on Windows systems). Putting the paging file on separate spindles could provide some benefit but it's highly dependent on the amount of memory in the server and how much paging actually goes on.
My basic systems where RAID1 or RAID5, I would sperate off 20gigs usually for the system drive and then rest for the data drive.
Logic is, if there if is some massive file system issue with the system drive, hopefully it won't cross into the data drive. That would let me format the system drive if needed and leave the data alone.
Backups also happened, but if it saved me copying a few hundred gigs of data back over it was worth it.
The size of the system drive would be dependant on what the server was, the exchange server got more room then a basic file server.
Also kept things neat, everyone knew the E: drive on every server was all data.
For servers that use some kind of SAN for their data, it makes absolute sense as boot-from-SAN is still a bit dodgy even after all these years. A pair of local mirrored drives for the boot/OS volumes, and any data volumes supplied by the SAN.
For DAS servers it is less obvious. Some workloads, like databases, are happier when on their very own spindles. Other workloads are low enough disk I/O that it just plain doesn't matter.
So yes, there is some performance benefit here, but it depends greatly on where your bottlenecks are and how your storage is attached to the server.
There isn't much of a reason to seperate the OS from data (or applications) unless there are specfic files that would be of benefit to have on seperate spindles (in order to reduce contention or be placed on disks with higher performance).
Seperate partitions increase backup complexity and application recovery complexity. As far as keeping it seperate for reinstallation, the application will still have to be reinstalled and I would consider reinstalling the OS and application to a system with "live" data on a seperate disk to be a risk since it's certainly a possiblility that the app would reinitialize the data or the drive could get wiped during reinstall.