We currently run two DNS systems. The Windows domain uses the Windows DNS for an internal domain. Our BIND9 installation serves our public domain DNS. Since we run BIND on Unix it has been dubbed the "Unix DNS". All of our Unix/Linux systems have FQDNs of our public domain. All of the Windows systems joined to the domain have FQDNs of our internal network, but also have a BIND entry their hostname.publicdomain.edu if they need to be accessed from the Internet.
We are looking to consolidate our DNS systems when we move our domain to Windows 2008 R2. The goal is to use BIND to do everything. I know this is possible, but details from Microsoft are a bit scarce. We would like to ditch or minimize using or relying on the Windows DNS and DHCP. We would also like to consider moving to a single domain space rather than just having our BIND installation host the separate zone our Windows system is currently handling.
I have done a fair amount of research on doing this and have enough information to move forward with a test environment, but I am looking for others who have already done this or something similar.
Pros? Cons? Gotchas?
I haven't done it before, but I know from a combination of documentation from Microsoft and talking with Customers that it's feasible to use BIND9 to support AD.
AD requires SRV RR's. BIND9 can do that w/ no problem. If AD doesn't have the ability to resolve the various SRV RR's that it wants to register, lots of nasty things happen. Replication breaks, the knowledge consistency checker (which builts intra-site replication topologies) breaks, and logons from clients break. The "A" record for the domain, which resolves to all of the domain controllers in the domain, by default, is used by DFS on client computers to resolve the domain 'SYSVOL' location and get a DFS referral to the closest copy of the files, so it needs to exist or group policy will fail to work properly.
Dynamic DNS is highly preferred, because AD domain controllers want to register a fair number of records (SRV RR's, sites, etc). Dynamic DNs is not required, but you'll have to manually update the zone when new DC's are added or old ones are removed. BIND 9.5.0 supports GSS-TSIG, which Microsoft clients use to perform dynamic updates. You'll have to integrate BIND with Kerberos provided by AD to get this to work, but you really should since it will give you secure dynamic updates.
Anecdotally, I've heard people talk about "random strangeness" with using BIND9 instead of Microsoft DNS. I've never worked directly in one of these networks, but I've heard this from people for whom I have respect for their opinions. I'd guess that there may well be slight artifacts of the Microsoft DNS implementation that "play" better with AD than BIND9 does. DNS is DNS, but I doubt that Microsoft tests AD with BIND9 as comprehensively as they do with their own DNS server.
If you're going to have such a unified DNS infrastructure as you suggest I'd highly recommend using "views" to limit access to the _msdcs.domain.suffix zone to networks where domain-member computers will be located. There's no sense in letting the Internet see any information about your domain controller computers, sites, etc. There's good information, for an attacker, in the DNS that supports AD.
If I read you right, it sounds like you might want to change the existing DNS domain name for the Active Directory domain, too. Changing that may be problematic if it's not undertaken with due care. There are ramifications for Exchange, DFS, Certificate Services, and trust relationships with other domains. More in-depth information is available from Microsoft at: http://download.microsoft.com/download/9/6/5/965e6899-e086-4b3e-8ed6-516ea07ea225/domain-rename-intro.doc It's very feasible to rename a domain, but it needs to be a planned and coordinated activity.
Your title mentions ISC DHCPD, but your question doesn't really address it. I don't recall what dynamic DNS update functionality is present in the ISC DHCPD, but you can always configure domain member computers to perform their own DNS A and PTR record registration rather than relying on the DHCP server to do it. (I've always thought having the DHCP register A records, as the stock Microsoft configuration works, is kind of silly.) There are no "special" options handed out by the Microsoft DHCP server that the ISC DHCP server can't handle (no proprietary extensions to the protocol, etc). I've supported Windows domain member client computers with a variety of DHCP servers (ISC, embedded in Cisco devices, Windows) with no ill effects.