If I'm not mistaken, it's determined by the NIC binding order in the Advanced Settings in the network connections folder. You can verify it by changing the binding order of the various NIC's and running nslookup as a test.

To expand on my answer, citing the article that Evan linked, here is an excerpt from said article:

The DNS Client service queries the DNS servers in the following order:

1. The DNS Client service sends the name query to the first DNS server on the preferred adapter’s list of DNS servers and waits one second for a response.

2. If the DNS Client service does not receive a response from the first DNS server within one second, it sends the name query to the first DNS servers on all adapters that are still under consideration and waits two seconds for a response.

3. If the DNS Client service does not receive a response from any DNS server within two seconds, the DNS Client service sends the query to all DNS servers on all adapters that are still under consideration and waits another two seconds for a response.

4. If the DNS Client service still does not receive a response from any DNS server, it sends the name query to all DNS servers on all adapters that are still under consideration and waits four seconds for a response.

5. If it the DNS Client service does not receive a response from any DNS server, the DNS client sends the query to all DNS servers on all adapters that are still under consideration and waits eight seconds for a response.

The preferred adapter in step 1 being the adapter that's listed first in the binding order.

In Windows 10 you must update the metric of each interface in the order you want.

1. Goto Control Panel > Network and Internet > Network Connections
2. Right click the desired connection (Higher Priority Connection)
3. Click Properties > Internet Protocol Version 4
4. Click Properties > Advanced
5. Uncheck 'Automatic Metric'
6. Enter 10 in 'Interface Metric'
7. Click OK

References:

• answers.microsoft.com
• Configure the Order of Network Interfaces - Microsoft Docs

I used 10 for my LAN, 20 for WLAN, and 100 for VPN Interfaces (I prefer local DNS at work, this may be reverse for others). Remember lower metric = higher priority.

See this article in Microsoft Support for An explanation of the Automatic Metric feature for IPv4 routes.

I suspect this works because part of the route selection is to then set the source IP of the interface with the highest metric priority (lowest number) as you are not using the same IP address for all connections, whereas a dual-homed BGP connection would use the same source IP, but select different next-hop gateways pending destination IP.

For Server, install the DNS service and do something purposeful/intelligent, I do not agree with doing this on Server OS.

To specifically answer the question, Windows uses the first DNS records associated with the highest ranking network interface (an InterfaceMetric with a lower value has higher rank). Examples below show how to obtain and change that interface ranking value. When there are no manual DNS servers configured on an interface, Windows relies on DNS servers provided by any DHCP server associated with a network interface.

A common scenario where this issue pops up is when you have one or more VPNs that provide a DNS server for resources defined for that virtual network.

However, the public DNS server assigned by a WiFi router is taking precedent and either claiming that hosts don't exist or are responding with the unexpected IP address.

To resolve this, we need to explicitly dictate which network interface should take precedence by specifying its associated "metric". (Currently, this metric appears to only be settable via PowerShell ... see below)

Change Metric of Network Interface (e.g., the VPN)

First, list local interfaces and their metrics

C:\> Get-NetIPInterface | Select-Object -Property InterfaceAlias, InterfaceMetric | Sort-Object -Property InterfaceMetric

InterfaceAlias               InterfaceMetric
--------------               ---------------
vEthernet (DockerNAT)                     15
Npcap Loopback Adapter                    25
Ethernet 11                               25
Local Area Connection* 1                  25
Local Area Connection* 2                  25
Npcap Loopback Adapter                    25
Ethernet 2                                35
Wi-Fi                                     55
Bluetooth Network Connection              65
Loopback Pseudo-Interface 1               75
MY VPN                                   200   <--- There's my NIC with my fave DNS
vEthernet (Default Switch)              5000

Then, set metric of network interface

(NOTE: Make sure you do this from an administrative PowerShell console)

C:\>  Set-NetIPInterface -InterfaceAlias "MY VPN" -InterfaceMetric 10

C:\> Get-NetIPInterface | Select-Object -Property InterfaceAlias, InterfaceMetric | Sort-Object -Property InterfaceMetric

InterfaceAlias               InterfaceMetric
--------------               ---------------
MY VPN                                    10   <--- Awesome!!! My fave DNS is now on top!!
vEthernet (DockerNAT)                     15
Npcap Loopback Adapter                    25
Ethernet 11                               25
Local Area Connection* 1                  25
Local Area Connection* 2                  25
Npcap Loopback Adapter                    25
Ethernet 2                                35
Wi-Fi                                     55
Bluetooth Network Connection              65
Loopback Pseudo-Interface 1               75
vEthernet (Default Switch)              5000

Testing Effect of Interface Metric

In this example, before changing the metric value, pinging a local resource that also has a publicly-defined IP returns its public IP (instead of the internal IP returned by my VPN's DNS server).

Example When Bad! :(

> ping 10640-TEST

Pinging 10640-TEST.example.com [XX.163.171.155] with 32 bytes of data:
Reply from XX.163.171.155: bytes=32 time=80ms TTL=45
Reply from XX.163.171.155: bytes=32 time=76ms TTL=45

Example When Good! :)

After changing the metric on the network interface, I now see the expected internal IP address that is returned from the DNS server associated with my VPN:

> ping 10640-TEST

Pinging 10640-TEST.example.com [192.168.100.44] with 32 bytes of data:
Reply from 192.168.100.44: bytes=32 time=42ms TTL=127
Reply from 192.168.100.44: bytes=32 time=52ms TTL=127

This page describes the algorithm used by Windows to perform DNS queries. It's not in depth enough to give you all the answers you're looking for, but some time w/ a sniffer and this article ought to be all you need to determine what's happening in your specific situation.

I had exactly the same problem. I've spent a day trying to figure it out.

If you have several network cards and if you specify a DNS server in each of them do you know what DNS server will be used?

Well, you can check it with:

nslookup 192.168.3.6

So you see the server that your PC actually uses.

The question is: how does windows choose which DNS server to use and how can we change it.

We especially need it when we use VPN clients.

The answer from qwerty2010 is right and correct. But you can use it ONLY when you have a NIC in control panel - network and sharing center - change adapter settings - list of nics

Only if you see the nic that is reponsible for your VPN client you can use the graphical way.

So you go Network and sharing centre -> Change adapter settings -> Advanced -> Advanced settings -> and you move UP the nic with DNS you'd like to USE.

However if for instance you use Shrew vpn client - you will not have any nics.

What do you do? You open registry.

HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\Class\{4D36E972-E325-11CE-BFC1-08002BE10318}

And you look in subfolders 0000, 0001 etc the folder with

DriverDesc = Shrew Soft Virtual Adapter

Then you copy to the clipboard

NetCfgInstanceId = {B498E7DE-7257-48F6-AD32-60E470030F05}

Now you go

[HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\services\Tcpip\Linkage]

and you open key = Bind. You'll see the list like this

\Device\{1DF89CE3-CAAD-4EB7-A53F-AD16BC1D5EFD}
\Device\{70126DBE-B44D-4392-9417-0CABD6E384B1}
\Device\{D5127F8E-E7BB-4661-AE5A-A922614173D0}
\Device\{C44039AB-6801-4A9B-A736-3B12782FF411}
\Device\{85231D0F-CD05-4774-A983-632C5D83AC62}
\Device\{7E87BC81-8C58-4E05-9FA0-7897A6AA5CCE}
\Device\{3A1A3EFC-A9DE-4BCA-BAF6-81C7074487E0}
\Device\{8D41EDFC-04AC-4537-B5D5-0D54EB51A023}

All you have to do is put to the top

\Device\{B498E7DE-7257-48F6-AD32-60E470030F05}
\Device\{1DF89CE3-CAAD-4EB7-A53F-AD16BC1D5EFD}
\Device\{70126DBE-B44D-4392-9417-0CABD6E384B1}
\Device\{D5127F8E-E7BB-4661-AE5A-A922614173D0}
\Device\{C44039AB-6801-4A9B-A736-3B12782FF411}
\Device\{85231D0F-CD05-4774-A983-632C5D83AC62}
\Device\{7E87BC81-8C58-4E05-9FA0-7897A6AA5CCE}
\Device\{3A1A3EFC-A9DE-4BCA-BAF6-81C7074487E0}
\Device\{8D41EDFC-04AC-4537-B5D5-0D54EB51A023}

That's all. There is no need to reboot.

Now your windows will use DNS specified in shrew vpn nic when you use vpn connection.

On Windows 10 and 11, DNS priority is determined by interface metric, this is documented here - the DNS servers on the interface with the lowest metric value will be used first.

It is important to note that an interface actually has two interface metric values, one for IPv4 and one for IPv6.

You can configure these either via:

Control Panel -> Network and Sharing Center -> Change adapter settings -> Ethernet Properties -> Internet Protocol Version 4/6 -> Advanced -> Interface Metric

Or via Powershell:

Set-NetIPInterface -InterfaceAlias "Ethernet" -AddressFamily IPv4 -InterfaceMetric 50
Set-NetIPInterface -InterfaceAlias "Ethernet" -AddressFamily IPv6 -InterfaceMetric 50

For the purposes of DNS the lowest of the two values is considered, in other words the list of interfaces is sorted by the min(Ipv4Metric, Ipv6Metric).

Each interface (e.g. Ethernet or WiFi) then has a list of DNS servers, but within that list IPv6 servers come first.

Here is an example scenario:

Windows first finds the best interface - Ethernet 2 since it has the lowest metric, and will therefore use DNS server 10.2.2.2

The next best interface is Ethernet 1, Windows will attempt 2001:db8::1 since IPv6 DNS servers are used first. And then finally 10.1.1.1

Note: See my experiments that prove this

As this might help you or others, you can disable the windows round-robin usage of the listed dns to make the dns usage more predictable. Try setting the RoundRobin=0 in the HKEY_LOCAL_MACHINE\System\CurrentControlSet\Services\DNS\Parameters (reference) to disable the dns round-robing.

There is also a "timeout" for rotating the servers, you can also try to setting it to zero:

 HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\Dnscache\Parameters
 ServerPriorityTimeLimit=0

It doesn't decide randomly. You're connected to a router which gets it IP from a company which has DNS servers. They get your request unless you alter the IP manually to another DNS, for instance: OpenDns. Or perhaps you decide on having your own DNS servers. That works too. Just enter the IP adresses in the network center and all should be well. And yes, you'll have to set those manually.

You can also adjust a file in C:\Windows\System32 called "hosts" with no extension. It allows you to have URL-requests redirected instead of being sent to the standard DNS. To a local server(which must be running and listening to port 80)