I found that after the latest leap second insertion (2016-12-31 23:59:60), our CentOS7 application that has worker threads sleeping for 1 second between jobs, started to wake the sleeping threads immediately instead of in a second. In general, all sleeps are awake 1 second ahead of expected wake time.
The simplest and working solution is to reboot the box. But that is not desirable in our case. Is there a way to fix this without rebooting?
PS. For reference, here's a simple program in C++ that reproduces the issue.
#include <boost/date_time.hpp>
#include <boost/thread.hpp>
#include <iostream>
using namespace std;
// this has to be run in a thread to be able to detect the issue
void check_thread()
{
size_t expected_delay = 1000;
cout << "Expected delay: " << expected_delay << " ms" << endl;
boost::posix_time::ptime t1 = boost::posix_time::microsec_clock::universal_time();
boost::this_thread::sleep(boost::posix_time::milliseconds(1000));
boost::posix_time::ptime t2 = boost::posix_time::microsec_clock::universal_time();
size_t actual_delay = (t2 - t1).total_milliseconds();
cout << "Actual delay: " << actual_delay << " ms" << endl;
if (abs(expected_delay - actual_delay) > 900) {
cout << "Too big delay difference: " << (expected_delay - actual_delay) << endl;
cout << "Possible leap second issue" << endl;
}
else {
cout << "No issues found" << endl;
}
}
int main()
{
boost::thread_group g;
g.create_thread(check_thread);
g.join_all();
return 0;
}
Building:
g++ sleep_test.cpp -Wl,-Bstatic -lboost_thread -lboost_system -lboost_date_time -Wl,-Bdynamic -rdynamic -pthread
Is your system time synched with
ntpd
orptp
? If not, update yourtzdata
package.Resolve Leap Second Issues in Red Hat Enterprise Linux
In addition to what Troy said, on RHEL7 systems that had not updated tzdata by the moment the leap second is applied, and not running ntpd, an extra step is required - manually set time 1 second forward, then revert it: