We just had an incident on one of our webservers. Several PHP applications run on it and started spitting out the following error message:
php_network_getaddresses: getaddrinfo failed: Name or service not known
This message seems to be network/dns related. The server logs clearly indicated that around the time that the first error occurred, Ubuntu started an unattended-upgrades session. I will attach the output from dpkg.log below.
We use an external database server.
We decided to reboot the webserver, which solved the issue, but the question now is:
Is there any way that I can find out the underlying cause of this issue so that I can prevent it in the future?
Output from unattended-upgrades.log:
2017-03-21 15:01:56,279 INFO Starting unattended upgrades script
2017-03-21 15:01:56,279 INFO Allowed origins are: ['o=Ubuntu,a=xenial', 'o=Ubuntu,a=xenial-security']
2017-03-21 15:02:03,365 INFO Packages that will be upgraded: libc-bin libc-dev-bin libc6 libc6-dev libfreetype6 locales multiarch-support
2017-03-21 15:02:03,365 INFO Writing dpkg log to '/var/log/unattended-upgrades/unattended-upgrades-dpkg.log'
2017-03-21 15:02:12,851 INFO All upgrades installed
Output from dpkg.log:
2017-03-21 15:02:03 startup archives unpack
2017-03-21 15:02:03 upgrade libc6-dev:amd64 2.23-0ubuntu5 2.23-0ubuntu6
2017-03-21 15:02:03 status half-configured libc6-dev:amd64 2.23-0ubuntu5
2017-03-21 15:02:03 status unpacked libc6-dev:amd64 2.23-0ubuntu5
2017-03-21 15:02:03 status half-installed libc6-dev:amd64 2.23-0ubuntu5
2017-03-21 15:02:04 status half-installed libc6-dev:amd64 2.23-0ubuntu5
2017-03-21 15:02:04 status unpacked libc6-dev:amd64 2.23-0ubuntu6
2017-03-21 15:02:04 status unpacked libc6-dev:amd64 2.23-0ubuntu6
2017-03-21 15:02:04 upgrade libc-dev-bin:amd64 2.23-0ubuntu5 2.23-0ubuntu6
2017-03-21 15:02:04 status half-configured libc-dev-bin:amd64 2.23-0ubuntu5
2017-03-21 15:02:04 status unpacked libc-dev-bin:amd64 2.23-0ubuntu5
2017-03-21 15:02:04 status half-installed libc-dev-bin:amd64 2.23-0ubuntu5
2017-03-21 15:02:04 status triggers-pending man-db:amd64 2.7.5-1
2017-03-21 15:02:04 status half-installed libc-dev-bin:amd64 2.23-0ubuntu5
2017-03-21 15:02:04 status unpacked libc-dev-bin:amd64 2.23-0ubuntu6
2017-03-21 15:02:04 status unpacked libc-dev-bin:amd64 2.23-0ubuntu6
2017-03-21 15:02:04 upgrade libc6:amd64 2.23-0ubuntu5 2.23-0ubuntu6
2017-03-21 15:02:04 status triggers-pending libc-bin:amd64 2.23-0ubuntu5
2017-03-21 15:02:04 status half-configured libc6:amd64 2.23-0ubuntu5
2017-03-21 15:02:04 status unpacked libc6:amd64 2.23-0ubuntu5
2017-03-21 15:02:04 status half-installed libc6:amd64 2.23-0ubuntu5
2017-03-21 15:02:05 status half-installed libc6:amd64 2.23-0ubuntu5
2017-03-21 15:02:05 status unpacked libc6:amd64 2.23-0ubuntu6
2017-03-21 15:02:05 status unpacked libc6:amd64 2.23-0ubuntu6
2017-03-21 15:02:05 startup packages configure
2017-03-21 15:02:05 configure libc6:amd64 2.23-0ubuntu6 <none>
2017-03-21 15:02:05 status unpacked libc6:amd64 2.23-0ubuntu6
2017-03-21 15:02:05 status unpacked libc6:amd64 2.23-0ubuntu6
2017-03-21 15:02:05 status half-configured libc6:amd64 2.23-0ubuntu6
2017-03-21 15:02:05 status installed libc6:amd64 2.23-0ubuntu6
2017-03-21 15:02:05 trigproc libc-bin:amd64 2.23-0ubuntu5 <none>
2017-03-21 15:02:05 status half-configured libc-bin:amd64 2.23-0ubuntu5
2017-03-21 15:02:05 status installed libc-bin:amd64 2.23-0ubuntu5
2017-03-21 15:02:05 trigproc man-db:amd64 2.7.5-1 <none>
2017-03-21 15:02:05 status half-configured man-db:amd64 2.7.5-1
2017-03-21 15:02:05 status installed man-db:amd64 2.7.5-1
2017-03-21 15:02:05 startup archives unpack
2017-03-21 15:02:05 upgrade locales:all 2.23-0ubuntu5 2.23-0ubuntu6
2017-03-21 15:02:05 status half-configured locales:all 2.23-0ubuntu5
2017-03-21 15:02:05 status unpacked locales:all 2.23-0ubuntu5
2017-03-21 15:02:05 status half-installed locales:all 2.23-0ubuntu5
2017-03-21 15:02:06 status triggers-pending man-db:amd64 2.7.5-1
2017-03-21 15:02:06 status half-installed locales:all 2.23-0ubuntu5
2017-03-21 15:02:06 status unpacked locales:all 2.23-0ubuntu6
2017-03-21 15:02:06 status unpacked locales:all 2.23-0ubuntu6
2017-03-21 15:02:06 upgrade libc-bin:amd64 2.23-0ubuntu5 2.23-0ubuntu6
2017-03-21 15:02:06 status half-configured libc-bin:amd64 2.23-0ubuntu5
2017-03-21 15:02:06 status unpacked libc-bin:amd64 2.23-0ubuntu5
2017-03-21 15:02:06 status half-installed libc-bin:amd64 2.23-0ubuntu5
2017-03-21 15:02:06 status half-installed libc-bin:amd64 2.23-0ubuntu5
2017-03-21 15:02:06 status unpacked libc-bin:amd64 2.23-0ubuntu6
2017-03-21 15:02:06 status unpacked libc-bin:amd64 2.23-0ubuntu6
2017-03-21 15:02:06 trigproc man-db:amd64 2.7.5-1 <none>
2017-03-21 15:02:06 status half-configured man-db:amd64 2.7.5-1
2017-03-21 15:02:07 status installed man-db:amd64 2.7.5-1
2017-03-21 15:02:07 startup packages configure
2017-03-21 15:02:07 configure libc-bin:amd64 2.23-0ubuntu6 <none>
2017-03-21 15:02:07 status unpacked libc-bin:amd64 2.23-0ubuntu6
2017-03-21 15:02:07 status unpacked libc-bin:amd64 2.23-0ubuntu6
2017-03-21 15:02:07 status unpacked libc-bin:amd64 2.23-0ubuntu6
2017-03-21 15:02:07 status unpacked libc-bin:amd64 2.23-0ubuntu6
2017-03-21 15:02:07 status unpacked libc-bin:amd64 2.23-0ubuntu6
2017-03-21 15:02:07 status unpacked libc-bin:amd64 2.23-0ubuntu6
2017-03-21 15:02:07 status half-configured libc-bin:amd64 2.23-0ubuntu6
2017-03-21 15:02:07 status installed libc-bin:amd64 2.23-0ubuntu6
2017-03-21 15:02:07 startup archives unpack
2017-03-21 15:02:07 upgrade multiarch-support:amd64 2.23-0ubuntu5 2.23-0ubuntu6
2017-03-21 15:02:07 status half-configured multiarch-support:amd64 2.23-0ubuntu5
2017-03-21 15:02:07 status unpacked multiarch-support:amd64 2.23-0ubuntu5
2017-03-21 15:02:07 status half-installed multiarch-support:amd64 2.23-0ubuntu5
2017-03-21 15:02:07 status half-installed multiarch-support:amd64 2.23-0ubuntu5
2017-03-21 15:02:07 status unpacked multiarch-support:amd64 2.23-0ubuntu6
2017-03-21 15:02:07 status unpacked multiarch-support:amd64 2.23-0ubuntu6
2017-03-21 15:02:07 startup packages configure
2017-03-21 15:02:07 configure multiarch-support:amd64 2.23-0ubuntu6 <none>
2017-03-21 15:02:07 status unpacked multiarch-support:amd64 2.23-0ubuntu6
2017-03-21 15:02:07 status half-configured multiarch-support:amd64 2.23-0ubuntu6
2017-03-21 15:02:07 status installed multiarch-support:amd64 2.23-0ubuntu6
2017-03-21 15:02:07 startup archives unpack
2017-03-21 15:02:07 upgrade libfreetype6:amd64 2.6.1-0.1ubuntu2 2.6.1-0.1ubuntu2.1
2017-03-21 15:02:07 status triggers-pending libc-bin:amd64 2.23-0ubuntu6
2017-03-21 15:02:07 status half-configured libfreetype6:amd64 2.6.1-0.1ubuntu2
2017-03-21 15:02:07 status unpacked libfreetype6:amd64 2.6.1-0.1ubuntu2
2017-03-21 15:02:07 status half-installed libfreetype6:amd64 2.6.1-0.1ubuntu2
2017-03-21 15:02:07 status half-installed libfreetype6:amd64 2.6.1-0.1ubuntu2
2017-03-21 15:02:07 status unpacked libfreetype6:amd64 2.6.1-0.1ubuntu2.1
2017-03-21 15:02:07 status unpacked libfreetype6:amd64 2.6.1-0.1ubuntu2.1
2017-03-21 15:02:07 trigproc libc-bin:amd64 2.23-0ubuntu6 <none>
2017-03-21 15:02:07 status half-configured libc-bin:amd64 2.23-0ubuntu6
2017-03-21 15:02:07 status installed libc-bin:amd64 2.23-0ubuntu6
2017-03-21 15:02:07 startup packages configure
2017-03-21 15:02:07 configure libc-dev-bin:amd64 2.23-0ubuntu6 <none>
2017-03-21 15:02:07 status unpacked libc-dev-bin:amd64 2.23-0ubuntu6
2017-03-21 15:02:07 status half-configured libc-dev-bin:amd64 2.23-0ubuntu6
2017-03-21 15:02:07 status installed libc-dev-bin:amd64 2.23-0ubuntu6
2017-03-21 15:02:07 configure libc6-dev:amd64 2.23-0ubuntu6 <none>
2017-03-21 15:02:07 status unpacked libc6-dev:amd64 2.23-0ubuntu6
2017-03-21 15:02:07 status half-configured libc6-dev:amd64 2.23-0ubuntu6
2017-03-21 15:02:07 status installed libc6-dev:amd64 2.23-0ubuntu6
2017-03-21 15:02:07 configure locales:all 2.23-0ubuntu6 <none>
2017-03-21 15:02:07 status unpacked locales:all 2.23-0ubuntu6
2017-03-21 15:02:07 status unpacked locales:all 2.23-0ubuntu6
2017-03-21 15:02:07 status half-configured locales:all 2.23-0ubuntu6
2017-03-21 15:02:11 status installed locales:all 2.23-0ubuntu6
2017-03-21 15:02:11 configure libfreetype6:amd64 2.6.1-0.1ubuntu2.1 <none>
2017-03-21 15:02:11 status triggers-pending libc-bin:amd64 2.23-0ubuntu6
2017-03-21 15:02:11 status unpacked libfreetype6:amd64 2.6.1-0.1ubuntu2.1
2017-03-21 15:02:11 status half-configured libfreetype6:amd64 2.6.1-0.1ubuntu2.1
2017-03-21 15:02:11 status installed libfreetype6:amd64 2.6.1-0.1ubuntu2.1
2017-03-21 15:02:11 trigproc libc-bin:amd64 2.23-0ubuntu6 <none>
2017-03-21 15:02:11 status half-configured libc-bin:amd64 2.23-0ubuntu6
2017-03-21 15:02:11 status installed libc-bin:amd64 2.23-0ubuntu6
2017-03-21 15:02:11 startup packages configure
We saw this hit our environment and it happened again 5 hours later. Considering rolling back to an older version of libc6 patch.
Here is the Ubuntu bug for this issue that confirms what we're all seeing: https://bugs.launchpad.net/ubuntu/+source/eglibc/+bug/1674532
Update: the issue has since been fixed.
See comments #16, #17 and #18 in the above report. Updating
libcshould resolve the issue. A reboot is recommended to make sure that any running processes that are running during the update that make use of this library keep functioning the way they should.