In a domain name registrar you have the option to set name servers as such: ns1.example.com, etc. and you don't have to specify the IP address of the name servers.
Well, what if I create a set of name servers and have them set as someone else's domain?
What is stopping the domain name from resolving to the lookalike DNS servers I just created ?
example:
Say you have a domain, peanuthead.com. You go to digital ocean and create two name servers with BInd9: ns1.peanuthead.com and ns2.peanuthead.com. you go back to your registrar and set your name servers to ns1.peanuthead.com and ns2.peanuthead.com. this means when you go to peanuthead.com it will query thoes dns servers.
What if someone randomly creates two name servers that you created, ns1.peanuthead.com and ns2.peanuthead.com? what stops the domain peanuthead.com from querying those name servers created by someone else?
You can stand up a DNS server and have it serve requests for whatever domain you'd like. That would be a completely pointless exercise, though, because requests for those domains would never hit your server.
Well, as you pointed out, nameserver records are set at the domain registrar. If they're set properly, requests for your domain will only ever be referred to those servers.
You seem to be quite confused. If you are in control of example.com, only you and you alone can create DNS records in that domain. Sure, anyone else can configure their DNS server to be authoritative for that domain, but again, that's pointless as queries for example.com will only ever be directed to the specified nameservers.
This question is nonsensical, as no one else can create records in your domain, and no one but you can edit the nameserver records at your registrar.
I think this is the main cause of confusion:
That is only true if you have already specified the ip address of the name server somewhere else (perhaps with another registrar). Setting your dns server to a domain name instead of IP address makes the assumption that you've set up the domain name of your server correctly.
Some registrars even disallow recursive dns references where the dns server is a subdomain of the domain the dns server is managing. Some allow it but also allows you to register the IP address of the dns servers so that their root servers can know where to start dns lookup. Then there are some who can't be bothered to validate your settings and allow you to create broken and non-working domain setups.