Home / server / Questions / 706184
In Process
Michael Renner
Asked: 2015-07-17 02:45:12 +0800 CST

What causes the Linux neighbour table confirmation counter to be reset

  • 772

Given three linux hosts on the same broadcast domain, the sender constantly pings an IP bound to receiver1.

When moving the IP address from receiver1 to receiver2 it takes a quite long time for the ARP/neighbour cache to be invalidated on sender.

While monitoring the neighbour cache statistics, the confirmation counter is reset in irregular fashion despite no inbound traffic from the neighbour entry has been received.

An example:

sender: 172.22.5.79 / 00:1b:21:cd:a3:11
receiver1: b0:83:fe:da:7e:95
receiver2: b0:83:fe:da:83:ab
moving ip: 172.22.15.188

IP moved betwen 10:33:06 and 10:33:07.

tcpdump on the sender:

10:33:02.967762 00:1b:21:cd:a3:11 > b0:83:fe:da:7e:95, ethertype IPv4 (0x0800), length 98: 172.22.5.79 > 172.22.15.188: ICMP echo request, id 20133, seq 31, length 64
10:33:02.968855 b0:83:fe:da:7e:95 > 00:1b:21:cd:a3:11, ethertype IPv4 (0x0800), length 98: 172.22.15.188 > 172.22.5.79: ICMP echo reply, id 20133, seq 31, length 64
10:33:03.968901 00:1b:21:cd:a3:11 > b0:83:fe:da:7e:95, ethertype IPv4 (0x0800), length 98: 172.22.5.79 > 172.22.15.188: ICMP echo request, id 20133, seq 32, length 64
10:33:03.980530 b0:83:fe:da:7e:95 > 00:1b:21:cd:a3:11, ethertype IPv4 (0x0800), length 98: 172.22.15.188 > 172.22.5.79: ICMP echo reply, id 20133, seq 32, length 64
10:33:04.970580 00:1b:21:cd:a3:11 > b0:83:fe:da:7e:95, ethertype IPv4 (0x0800), length 98: 172.22.5.79 > 172.22.15.188: ICMP echo request, id 20133, seq 33, length 64
10:33:04.970714 b0:83:fe:da:7e:95 > 00:1b:21:cd:a3:11, ethertype IPv4 (0x0800), length 98: 172.22.15.188 > 172.22.5.79: ICMP echo reply, id 20133, seq 33, length 64
10:33:05.969581 00:1b:21:cd:a3:11 > b0:83:fe:da:7e:95, ethertype IPv4 (0x0800), length 98: 172.22.5.79 > 172.22.15.188: ICMP echo request, id 20133, seq 34, length 64
10:33:05.969748 b0:83:fe:da:7e:95 > 00:1b:21:cd:a3:11, ethertype IPv4 (0x0800), length 98: 172.22.15.188 > 172.22.5.79: ICMP echo reply, id 20133, seq 34, length 64
10:33:06.968585 00:1b:21:cd:a3:11 > b0:83:fe:da:7e:95, ethertype IPv4 (0x0800), length 98: 172.22.5.79 > 172.22.15.188: ICMP echo request, id 20133, seq 35, length 64
10:33:06.968685 b0:83:fe:da:7e:95 > 00:1b:21:cd:a3:11, ethertype IPv4 (0x0800), length 98: 172.22.15.188 > 172.22.5.79: ICMP echo reply, id 20133, seq 35, length 64
10:33:07.967738 00:1b:21:cd:a3:11 > b0:83:fe:da:7e:95, ethertype IPv4 (0x0800), length 98: 172.22.5.79 > 172.22.15.188: ICMP echo request, id 20133, seq 36, length 64
10:33:08.967756 00:1b:21:cd:a3:11 > b0:83:fe:da:7e:95, ethertype IPv4 (0x0800), length 98: 172.22.5.79 > 172.22.15.188: ICMP echo request, id 20133, seq 37, length 64
10:33:09.967761 00:1b:21:cd:a3:11 > b0:83:fe:da:7e:95, ethertype IPv4 (0x0800), length 98: 172.22.5.79 > 172.22.15.188: ICMP echo request, id 20133, seq 38, length 64
10:33:10.967752 00:1b:21:cd:a3:11 > b0:83:fe:da:7e:95, ethertype IPv4 (0x0800), length 98: 172.22.5.79 > 172.22.15.188: ICMP echo request, id 20133, seq 39, length 64
10:33:11.967769 00:1b:21:cd:a3:11 > b0:83:fe:da:7e:95, ethertype IPv4 (0x0800), length 98: 172.22.5.79 > 172.22.15.188: ICMP echo request, id 20133, seq 40, length 64
10:33:12.967750 00:1b:21:cd:a3:11 > b0:83:fe:da:7e:95, ethertype IPv4 (0x0800), length 98: 172.22.5.79 > 172.22.15.188: ICMP echo request, id 20133, seq 41, length 64
10:33:13.967749 00:1b:21:cd:a3:11 > b0:83:fe:da:7e:95, ethertype IPv4 (0x0800), length 98: 172.22.5.79 > 172.22.15.188: ICMP echo request, id 20133, seq 42, length 64
10:33:14.967760 00:1b:21:cd:a3:11 > b0:83:fe:da:7e:95, ethertype IPv4 (0x0800), length 98: 172.22.5.79 > 172.22.15.188: ICMP echo request, id 20133, seq 43, length 64
10:33:15.967735 00:1b:21:cd:a3:11 > b0:83:fe:da:7e:95, ethertype IPv4 (0x0800), length 98: 172.22.5.79 > 172.22.15.188: ICMP echo request, id 20133, seq 44, length 64
10:33:16.967745 00:1b:21:cd:a3:11 > b0:83:fe:da:7e:95, ethertype IPv4 (0x0800), length 98: 172.22.5.79 > 172.22.15.188: ICMP echo request, id 20133, seq 45, length 64
10:33:17.967756 00:1b:21:cd:a3:11 > b0:83:fe:da:7e:95, ethertype IPv4 (0x0800), length 98: 172.22.5.79 > 172.22.15.188: ICMP echo request, id 20133, seq 46, length 64
10:33:18.967784 00:1b:21:cd:a3:11 > b0:83:fe:da:7e:95, ethertype IPv4 (0x0800), length 98: 172.22.5.79 > 172.22.15.188: ICMP echo request, id 20133, seq 47, length 64
10:33:19.967762 00:1b:21:cd:a3:11 > b0:83:fe:da:7e:95, ethertype IPv4 (0x0800), length 98: 172.22.5.79 > 172.22.15.188: ICMP echo request, id 20133, seq 48, length 64
10:33:20.967775 00:1b:21:cd:a3:11 > b0:83:fe:da:7e:95, ethertype IPv4 (0x0800), length 98: 172.22.5.79 > 172.22.15.188: ICMP echo request, id 20133, seq 49, length 64
10:33:21.967748 00:1b:21:cd:a3:11 > b0:83:fe:da:7e:95, ethertype IPv4 (0x0800), length 98: 172.22.5.79 > 172.22.15.188: ICMP echo request, id 20133, seq 50, length 64
10:33:22.967737 00:1b:21:cd:a3:11 > b0:83:fe:da:7e:95, ethertype IPv4 (0x0800), length 98: 172.22.5.79 > 172.22.15.188: ICMP echo request, id 20133, seq 51, length 64
10:33:23.967743 00:1b:21:cd:a3:11 > b0:83:fe:da:7e:95, ethertype IPv4 (0x0800), length 98: 172.22.5.79 > 172.22.15.188: ICMP echo request, id 20133, seq 52, length 64
10:33:24.967773 00:1b:21:cd:a3:11 > b0:83:fe:da:7e:95, ethertype IPv4 (0x0800), length 98: 172.22.5.79 > 172.22.15.188: ICMP echo request, id 20133, seq 53, length 64
10:33:25.967757 00:1b:21:cd:a3:11 > b0:83:fe:da:7e:95, ethertype IPv4 (0x0800), length 98: 172.22.5.79 > 172.22.15.188: ICMP echo request, id 20133, seq 54, length 64
10:33:26.967760 00:1b:21:cd:a3:11 > b0:83:fe:da:7e:95, ethertype IPv4 (0x0800), length 98: 172.22.5.79 > 172.22.15.188: ICMP echo request, id 20133, seq 55, length 64
10:33:27.967755 00:1b:21:cd:a3:11 > b0:83:fe:da:7e:95, ethertype IPv4 (0x0800), length 98: 172.22.5.79 > 172.22.15.188: ICMP echo request, id 20133, seq 56, length 64
10:33:28.971748 00:1b:21:cd:a3:11 > b0:83:fe:da:7e:95, ethertype IPv4 (0x0800), length 98: 172.22.5.79 > 172.22.15.188: ICMP echo request, id 20133, seq 57, length 64
10:33:29.971744 00:1b:21:cd:a3:11 > b0:83:fe:da:7e:95, ethertype IPv4 (0x0800), length 98: 172.22.5.79 > 172.22.15.188: ICMP echo request, id 20133, seq 58, length 64
10:33:30.971757 00:1b:21:cd:a3:11 > b0:83:fe:da:7e:95, ethertype IPv4 (0x0800), length 98: 172.22.5.79 > 172.22.15.188: ICMP echo request, id 20133, seq 59, length 64
10:33:31.971758 00:1b:21:cd:a3:11 > b0:83:fe:da:7e:95, ethertype IPv4 (0x0800), length 98: 172.22.5.79 > 172.22.15.188: ICMP echo request, id 20133, seq 60, length 64
10:33:32.971782 00:1b:21:cd:a3:11 > b0:83:fe:da:7e:95, ethertype IPv4 (0x0800), length 98: 172.22.5.79 > 172.22.15.188: ICMP echo request, id 20133, seq 61, length 64
10:33:33.971793 00:1b:21:cd:a3:11 > b0:83:fe:da:7e:95, ethertype IPv4 (0x0800), length 98: 172.22.5.79 > 172.22.15.188: ICMP echo request, id 20133, seq 62, length 64
10:33:34.971790 00:1b:21:cd:a3:11 > b0:83:fe:da:7e:95, ethertype IPv4 (0x0800), length 98: 172.22.5.79 > 172.22.15.188: ICMP echo request, id 20133, seq 63, length 64
10:33:35.971779 00:1b:21:cd:a3:11 > b0:83:fe:da:7e:95, ethertype IPv4 (0x0800), length 98: 172.22.5.79 > 172.22.15.188: ICMP echo request, id 20133, seq 64, length 64
10:33:36.971765 00:1b:21:cd:a3:11 > b0:83:fe:da:7e:95, ethertype IPv4 (0x0800), length 98: 172.22.5.79 > 172.22.15.188: ICMP echo request, id 20133, seq 65, length 64
10:33:37.971769 00:1b:21:cd:a3:11 > b0:83:fe:da:7e:95, ethertype IPv4 (0x0800), length 98: 172.22.5.79 > 172.22.15.188: ICMP echo request, id 20133, seq 66, length 64
10:33:38.971756 00:1b:21:cd:a3:11 > b0:83:fe:da:7e:95, ethertype IPv4 (0x0800), length 98: 172.22.5.79 > 172.22.15.188: ICMP echo request, id 20133, seq 67, length 64
10:33:39.971734 00:1b:21:cd:a3:11 > b0:83:fe:da:7e:95, ethertype IPv4 (0x0800), length 98: 172.22.5.79 > 172.22.15.188: ICMP echo request, id 20133, seq 68, length 64
10:33:40.971752 00:1b:21:cd:a3:11 > b0:83:fe:da:7e:95, ethertype IPv4 (0x0800), length 98: 172.22.5.79 > 172.22.15.188: ICMP echo request, id 20133, seq 69, length 64
10:33:41.971749 00:1b:21:cd:a3:11 > b0:83:fe:da:7e:95, ethertype IPv4 (0x0800), length 98: 172.22.5.79 > 172.22.15.188: ICMP echo request, id 20133, seq 70, length 64
10:33:42.971755 00:1b:21:cd:a3:11 > b0:83:fe:da:7e:95, ethertype IPv4 (0x0800), length 98: 172.22.5.79 > 172.22.15.188: ICMP echo request, id 20133, seq 71, length 64
10:33:43.971766 00:1b:21:cd:a3:11 > b0:83:fe:da:7e:95, ethertype IPv4 (0x0800), length 98: 172.22.5.79 > 172.22.15.188: ICMP echo request, id 20133, seq 72, length 64
10:33:44.971753 00:1b:21:cd:a3:11 > b0:83:fe:da:7e:95, ethertype IPv4 (0x0800), length 98: 172.22.5.79 > 172.22.15.188: ICMP echo request, id 20133, seq 73, length 64
10:33:45.971757 00:1b:21:cd:a3:11 > b0:83:fe:da:7e:95, ethertype IPv4 (0x0800), length 98: 172.22.5.79 > 172.22.15.188: ICMP echo request, id 20133, seq 74, length 64
10:33:46.971794 00:1b:21:cd:a3:11 > b0:83:fe:da:7e:95, ethertype IPv4 (0x0800), length 98: 172.22.5.79 > 172.22.15.188: ICMP echo request, id 20133, seq 75, length 64
10:33:47.971750 00:1b:21:cd:a3:11 > b0:83:fe:da:7e:95, ethertype IPv4 (0x0800), length 98: 172.22.5.79 > 172.22.15.188: ICMP echo request, id 20133, seq 76, length 64
10:33:48.971765 00:1b:21:cd:a3:11 > b0:83:fe:da:7e:95, ethertype IPv4 (0x0800), length 98: 172.22.5.79 > 172.22.15.188: ICMP echo request, id 20133, seq 77, length 64
10:33:49.971744 00:1b:21:cd:a3:11 > b0:83:fe:da:7e:95, ethertype IPv4 (0x0800), length 98: 172.22.5.79 > 172.22.15.188: ICMP echo request, id 20133, seq 78, length 64

Contents of the neighbour table on the sender for the same time frame:

Thu Jul 16 10:32:56 UTC 2015  172.22.15.188 dev eth1 lladdr b0:83:fe:da:7e:95 ref 1 used 464/0/464 probes 4 REACHABLE
Thu Jul 16 10:32:57 UTC 2015  172.22.15.188 dev eth1 lladdr b0:83:fe:da:7e:95 ref 1 used 465/0/465 probes 4 REACHABLE
Thu Jul 16 10:32:58 UTC 2015  172.22.15.188 dev eth1 lladdr b0:83:fe:da:7e:95 ref 1 used 466/0/466 probes 4 REACHABLE
Thu Jul 16 10:32:59 UTC 2015  172.22.15.188 dev eth1 lladdr b0:83:fe:da:7e:95 ref 1 used 467/0/467 probes 4 REACHABLE
Thu Jul 16 10:33:00 UTC 2015  172.22.15.188 dev eth1 lladdr b0:83:fe:da:7e:95 ref 1 used 468/0/468 probes 4 REACHABLE
Thu Jul 16 10:33:01 UTC 2015  172.22.15.188 dev eth1 lladdr b0:83:fe:da:7e:95 ref 1 used 469/0/469 probes 4 REACHABLE
Thu Jul 16 10:33:02 UTC 2015  172.22.15.188 dev eth1 lladdr b0:83:fe:da:7e:95 ref 1 used 470/0/470 probes 4 REACHABLE
Thu Jul 16 10:33:03 UTC 2015  172.22.15.188 dev eth1 lladdr b0:83:fe:da:7e:95 ref 1 used 471/0/471 probes 4 REACHABLE
Thu Jul 16 10:33:04 UTC 2015  172.22.15.188 dev eth1 lladdr b0:83:fe:da:7e:95 ref 1 used 472/0/472 probes 4 REACHABLE
Thu Jul 16 10:33:05 UTC 2015  172.22.15.188 dev eth1 lladdr b0:83:fe:da:7e:95 ref 1 used 473/0/473 probes 4 REACHABLE
Thu Jul 16 10:33:06 UTC 2015  172.22.15.188 dev eth1 lladdr b0:83:fe:da:7e:95 ref 1 used 474/0/474 probes 4 REACHABLE
Thu Jul 16 10:33:07 UTC 2015  172.22.15.188 dev eth1 lladdr b0:83:fe:da:7e:95 ref 1 used 475/0/475 probes 4 REACHABLE
Thu Jul 16 10:33:08 UTC 2015  172.22.15.188 dev eth1 lladdr b0:83:fe:da:7e:95 ref 1 used 476/0/476 probes 4 REACHABLE
Thu Jul 16 10:33:09 UTC 2015  172.22.15.188 dev eth1 lladdr b0:83:fe:da:7e:95 ref 1 used 477/1/477 probes 4 REACHABLE
Thu Jul 16 10:33:10 UTC 2015  172.22.15.188 dev eth1 lladdr b0:83:fe:da:7e:95 ref 1 used 478/2/478 probes 4 REACHABLE
Thu Jul 16 10:33:11 UTC 2015  172.22.15.188 dev eth1 lladdr b0:83:fe:da:7e:95 ref 1 used 479/3/479 probes 4 REACHABLE
Thu Jul 16 10:33:12 UTC 2015  172.22.15.188 dev eth1 lladdr b0:83:fe:da:7e:95 ref 1 used 480/0/480 probes 4 REACHABLE
Thu Jul 16 10:33:13 UTC 2015  172.22.15.188 dev eth1 lladdr b0:83:fe:da:7e:95 ref 1 used 481/1/481 probes 4 REACHABLE
Thu Jul 16 10:33:14 UTC 2015  172.22.15.188 dev eth1 lladdr b0:83:fe:da:7e:95 ref 1 used 482/2/482 probes 4 REACHABLE
Thu Jul 16 10:33:15 UTC 2015  172.22.15.188 dev eth1 lladdr b0:83:fe:da:7e:95 ref 1 used 483/3/483 probes 4 REACHABLE
Thu Jul 16 10:33:16 UTC 2015  172.22.15.188 dev eth1 lladdr b0:83:fe:da:7e:95 ref 1 used 484/4/484 probes 4 REACHABLE
Thu Jul 16 10:33:17 UTC 2015  172.22.15.188 dev eth1 lladdr b0:83:fe:da:7e:95 ref 1 used 485/5/485 probes 4 REACHABLE
Thu Jul 16 10:33:18 UTC 2015  172.22.15.188 dev eth1 lladdr b0:83:fe:da:7e:95 ref 1 used 486/6/486 probes 4 REACHABLE
Thu Jul 16 10:33:19 UTC 2015  172.22.15.188 dev eth1 lladdr b0:83:fe:da:7e:95 ref 1 used 488/0/488 probes 4 REACHABLE
Thu Jul 16 10:33:20 UTC 2015  172.22.15.188 dev eth1 lladdr b0:83:fe:da:7e:95 ref 1 used 489/0/489 probes 4 REACHABLE
Thu Jul 16 10:33:21 UTC 2015  172.22.15.188 dev eth1 lladdr b0:83:fe:da:7e:95 ref 1 used 490/1/490 probes 4 REACHABLE
Thu Jul 16 10:33:22 UTC 2015  172.22.15.188 dev eth1 lladdr b0:83:fe:da:7e:95 ref 1 used 491/2/491 probes 4 REACHABLE
Thu Jul 16 10:33:23 UTC 2015  172.22.15.188 dev eth1 lladdr b0:83:fe:da:7e:95 ref 1 used 492/3/492 probes 4 REACHABLE
Thu Jul 16 10:33:24 UTC 2015  172.22.15.188 dev eth1 lladdr b0:83:fe:da:7e:95 ref 1 used 493/4/493 probes 4 REACHABLE
Thu Jul 16 10:33:25 UTC 2015  172.22.15.188 dev eth1 lladdr b0:83:fe:da:7e:95 ref 1 used 494/5/494 probes 4 REACHABLE
Thu Jul 16 10:33:26 UTC 2015  172.22.15.188 dev eth1 lladdr b0:83:fe:da:7e:95 ref 1 used 495/6/495 probes 4 REACHABLE
Thu Jul 16 10:33:27 UTC 2015  172.22.15.188 dev eth1 lladdr b0:83:fe:da:7e:95 ref 1 used 496/7/496 probes 4 REACHABLE
Thu Jul 16 10:33:28 UTC 2015  172.22.15.188 dev eth1 lladdr b0:83:fe:da:7e:95 ref 1 used 497/8/497 probes 4 REACHABLE
Thu Jul 16 10:33:29 UTC 2015  172.22.15.188 dev eth1 lladdr b0:83:fe:da:7e:95 ref 1 used 498/9/498 probes 4 REACHABLE
Thu Jul 16 10:33:30 UTC 2015  172.22.15.188 dev eth1 lladdr b0:83:fe:da:7e:95 ref 1 used 499/10/499 probes 4 REACHABLE
Thu Jul 16 10:33:31 UTC 2015  172.22.15.188 dev eth1 lladdr b0:83:fe:da:7e:95 ref 1 used 500/11/500 probes 4 REACHABLE
Thu Jul 16 10:33:32 UTC 2015  172.22.15.188 dev eth1 lladdr b0:83:fe:da:7e:95 ref 1 used 501/12/501 probes 4 REACHABLE
Thu Jul 16 10:33:33 UTC 2015  172.22.15.188 dev eth1 lladdr b0:83:fe:da:7e:95 ref 1 used 502/13/502 probes 4 REACHABLE

An full dump of the network traffic showed that there was no further communication between the nodes except the outbound ICMP packets from the sender.

The situation would rectify itself randomly between 5 and 30 minutes when the timer eventually expires or after sending gratuitous ARP requests from receiver2 after the IP has moved.

Further reading of

didn't provide insights on what's going on here.

To clarify, I'd expect that, with default settings, the ARP entry points to the new MAC address after 50 seconds at the worst.

According to arp(7) the respective tunables are

  • base_reachable_time: [..an..] entry is considered to be valid for at least a random value between base_reachable_time/2 and 3*base_reachable_time/2. Default: 30s
  • delay_first_probe_time: Delay before first probe after it has been decided that a neighbor is stale. Default: 5s

3 * 30 / 2 + 5 = 50

linux

0 Answers