user2845840's questions

While trying to build PHP 5.3.3 on Debian jessie (save yourself some sanity by not asking why) with OpenSSL extension enabled, I encountered undefined reference to symbol 'X509_free@@OPENSSL_1.0.0' in the linking step of the Makefile and couldn't figure out why (libssl-dev is present and libssl.so contains the requested symbol). I tried to build the official Debian package that was included in Debian squeeze, using a pbuilder-based chroot-environment that contains a Debian jessie and therefore newer libraries (like OpenSSL 1.0.1 instead of 0.9.8). I confirmed that PHP 5.3.3 is compatible with OpenSSL 1.0.1 by building it outside the chroot-environment using ./configure --disable-all --with-openssl=/usr; make. The resulting CGI- and CLI-binaries are indeed linked against OpenSSL, which can be confirmed via ldd sapi/cli/php. And yet inside the chroot-environment, it failed.

Don't bother anwering, I already figured out most of the reason and a workaround/hack to enable a successful build.

We are administering a MySQL server for one of our customers that has >100 databases with about 50 tables each, many of them InnoDB tables. The server crashed and I'm trying to find the culprit. When restarting with innodb_force_recovery = 2, I can connect and don't see any errors in the error.log. More importantly, mysqlcheck --all-databases reports "Ok" for all tables. But when I remove innodb_force_recovery, the server crashes again, writes a stack trace to error.log and can only be stopped with kill -9.

How am I supposed to find the offending database under these circumstances and what would cause mysqlcheck to miss the corrupted tables? Please don't tell me to just ignore it and restore all databases from dumps. That might be acceptable for one or two databases and if it occurs only once in a blue moon, but I've had problems with the same server more than once and restoring everything from dumps just requires too much time and manual work to do that every time.

The server version is 5.5.46 and innodb_file_per_table is active.

Excerpt of error.log as requested (does the The tablespace free space info is corrupt mean there is an error that is not in a specific table and that can not be corrected?):

180222 17:13:48 mysqld_safe Starting mysqld daemon with databases from /home/mysql
180222 17:13:48 [Warning] 'THREAD_CONCURRENCY' is deprecated and will be removed in a future release.
180222 17:13:48 [Note] /usr/libexec/mysqld (mysqld 5.5.46) starting as process 26242 ...
180222 17:13:48 [Note] Plugin 'FEDERATED' is disabled.
180222 17:13:48 InnoDB: The InnoDB memory heap is disabled
180222 17:13:48 InnoDB: Mutexes and rw_locks use InnoDB's own implementation
180222 17:13:48 InnoDB: Compressed tables use zlib 1.2.3
180222 17:13:48 InnoDB: Using Linux native AIO
180222 17:13:48 InnoDB: Initializing buffer pool, size = 128.0M
180222 17:13:49 InnoDB: Completed initialization of buffer pool
180222 17:13:49 InnoDB: highest supported file format is Barracuda.
180222 17:13:49  InnoDB: Waiting for the background threads to start
180222 17:13:50 InnoDB: 5.5.46 started; log sequence number 1632912830888
180222 17:13:50 [Note] Server hostname (bind-address): '0.0.0.0'; port: 3306
180222 17:13:50 [Note]   - '0.0.0.0' resolves to '0.0.0.0';
180222 17:13:50 [Note] Server socket created on IP: '0.0.0.0'.
180222 17:13:50 [Note] Event Scheduler: Loaded 0 events
180222 17:13:50 [Note] /usr/libexec/mysqld: ready for connections.
Version: '5.5.46'  socket: '/var/lib/mysql/mysql.sock'  port: 3306  MySQL Community Server (GPL) by Remi
InnoDB: Dump of the tablespace extent descriptor:  len 40; hex 000000000000000200000000061600000000126e00000004ffffffffffffffffffffffffffffbfaa; asc                    n                    ;
InnoDB: Serious error! InnoDB is trying to free page 512
InnoDB: though it is already marked as free in the tablespace!
InnoDB: The tablespace free space info is corrupt.
InnoDB: You may need to dump your InnoDB tables and recreate the whole
InnoDB: database!
InnoDB: Please refer to
InnoDB: http://dev.mysql.com/doc/refman/5.5/en/forcing-innodb-recovery.html
InnoDB: about forcing recovery.
180222 17:13:50  InnoDB: Assertion failure in thread 2499464080 in file fsp0fsp.c line 3309
InnoDB: We intentionally generate a memory trap.
InnoDB: Submit a detailed bug report to http://bugs.mysql.com.
InnoDB: If you get repeated assertion failures or crashes, even
InnoDB: immediately after the mysqld startup, there may be
InnoDB: corruption in the InnoDB tablespace. Please refer to
InnoDB: http://dev.mysql.com/doc/refman/5.5/en/forcing-innodb-recovery.html
InnoDB: about forcing recovery.
16:13:50 UTC - mysqld got signal 6 ;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
We will try our best to scrape up some info that will hopefully help
diagnose the problem, but since we have already crashed,
something is definitely wrong and this may fail.

key_buffer_size=268435456
read_buffer_size=1048576
max_used_connections=0
max_threads=512
thread_count=0
connection_count=0
It is possible that mysqld could use up to
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 1314506 K  bytes of memory
Hope that's ok; if not, decrease some variables in the equation.

Thread pointer: 0x0
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 0 thread_stack 0x30000
/usr/libexec/mysqld(my_print_stacktrace+0x33)[0x842a1f3]
/usr/libexec/mysqld(handle_fatal_signal+0x42b)[0x82d9d3b]
[0x7bc420]
[0x7bc410]
/lib/libc.so.6(gsignal+0x50)[0x626b10]
/lib/libc.so.6(abort+0x101)[0x628421]
/usr/libexec/mysqld[0x85012e7]
/usr/libexec/mysqld[0x850147e]
/usr/libexec/mysqld[0x849c0b1]
/usr/libexec/mysqld[0x84a8a61]
/usr/libexec/mysqld[0x8561fef]
/usr/libexec/mysqld[0x85570a9]
/usr/libexec/mysqld[0x847b082]
/usr/libexec/mysqld[0x846bf04]
/usr/libexec/mysqld[0x846dad4]
/lib/libpthread.so.0[0x50d912]
/lib/libc.so.6(clone+0x5e)[0x6d347e]
The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains
information that should help you find out what is causing the crash.
180222 17:13:50 mysqld_safe Number of processes running now: 0
180222 17:13:50 mysqld_safe mysqld restarted

We're doing a graceful restart of our Apache server processes on the first day of every month. Usually it works fine. But this month one of our servers seemed to ignore that. We noticed it because the first few days of December were missing from the log files (which are NEW log files, the old ones being deleted after the graceful restart). This would be expected if the Apache processes ignored the graceful restart (i.e. the USR1 signal sent from Apache parent to all of its children) and kept on running, writing to the old (i.e. now deleted) log files. But the documentation ( https://httpd.apache.org/docs/current/stopping.html ) does not say anything about ignoring the signal and I don't know what circumstances would lead to that. Since the documentation says the USR1 signal advise[s] the children to exit after their current request, I guess ignoring it possible. The affected Apache is version 2.2.31.

I installed mod_security2 on several dozen servers (each with several dozen VHosts) and don't have the time to configure it for every VHost. In the default configuration, it produces copious amounts of false positives in log files, so I chose to let it run in DetectionOnly-mode (it doesn't block anything, but I still get detailed logs for most hacking attempts) for all but a select few VHosts.

I was happy with this setup until I discovered that log files on some servers grew to several Gigabytes in less than 3 weeks. I decided to switch off logging for the handful of VHosts that produced the lion's share of those log entries. There are several different ways to do this, I eventually settled on making new rules with very specific triggers that all have "nolog,phase:1,t:none,ctl:secAuditEngine=Off" as action. This succeeds insofar as the amount of entries in the audit log is reduced to manageable levels.

But I still get Gigabytes of logs because I can't seem to prevent mod_security2 from writing to the error log. I tried SecDebugLogLevel 0 since it's the only configuration directive concerned with error logging (that I was able to find, anyway), but to no avail. The only thing that seems to help is SecRuleEngine Off, which defeats the purpose of installing mod_security2 in the first place.

Am I missing something? No matter what I try, it appears as if I can only control the amount of logging to the audit log, while having no control over the amount of logging to the error log.

The situation I am facing: I set up a CentOS 5.9 machine that is supposed to be as close as possible a copy of another CentOS 5.9 machine, but not a clone. Using yum, I installed (as far as possible) all packages on the target machine that are also installed on the source machine (as reported by yum list installed and rpm -qa). Afterwards I used md5deep to find files that are still missing on the target machine. I found a lot more missing files than I expected and thereby discovered my current problem.

On both the target & source machines the package traceroute-3:2.0.1-6.el5.i386 is installed (says yum list installed). On both machines repoquery --info --show-duplicates traceroute lists only this version of the package and tells me that it can be found in the base-repository. On both machines rpm -ql traceroute-2.0.1-6.el5.i386 lists the same files. On both machines rpm -V traceroute-2.0.1-6.el5.i386 tells me that the installation went well (no files modified/missing). A quick look into traceroute-2.0.1-6.el5.i386.rpm shows only a binary file (/bin/traceroute), symlinks and documentation, so I don't expect compilation to take place when i install that package (but I'm not an RPM expert and it was only quick look, not a thorough examination). file /bin/traceroute tells me /bin/traceroute: ELF 32-bit LSB executable, Intel 80386, version 1 (SYSV), for GNU/Linux 2.6.9, dynamically linked (uses shared libs), stripped on both machines. /bin/traceroute has the same size on both machines.

And yet, when I execute md5sum /bin/traceroute on both machines, I get different results. xxd & diff show that several dozen bytes differ and the differences seem to be in the actual code (have not disassembled them yet, but the changed parts are not human readable strings). traceroute is just one random example, there are several dozen more differing files in /bin, /sbin & /lib.

Any idea what is going on? I was always under the impression that installing the same package twice would result in exactly the same files as long as compilation/editing was not part of the installation. At the least I was hoping that rpm -V would tell me about files that have been tampered with. It would help if yum had a facility to tell me from what repository it downloaded the RPM-file, but to the best of my knowledge this facility does not exist in my version of yum (3.2.22). To be clear: repoquery does not tell me what repository a package has been downloaded from, it only tells me where it is available. If the same package is available in 2 or more enabled repositories, repoquery lists all packages. Yum can decide which of the duplicates it wants and it does not seem to record this decision in any log files.