Will Martin's questions

I need another pair of eyes to help debug an issue. This will be a long message. I'm going to provide a lot of context because I'm not sure which details might be important.

We have two servers: dev, and production. The existing servers are running RHEL7 and due to hit end of life. I am in the process of building two new RHEL9 VMs to take their place.

The servers host a variety of apps. They're independent of one another and each resides in its own subdirectory off the document root. The document root is located in /var/www/html.

The one that's giving me trouble is a fresh, shiny installation of Drupal 10.2. I installed it to /var/www/drupal using composer, and then created a symbolic link at /var/www/html/databases that points to /var/www/drupal/web. I had to adjust the permissions a bit to make sure the Apache user could do everything it needed to, but pretty soon I had a functional installation of Drupal on both dev and prod.

Then the one on dev stopped working, shortly after I cloned a copy of our custom theme to the themes folder. When I visit, I get this error message:

The website encountered an unexpected error. Try again later.

The ssl_access_log shows this error:

10.133.0.30 - - [21/Feb/2024:13:55:07 -0600] "GET /databases/ HTTP/1.1" 500 61

Oh boy, a 500 error message. The error message that could mean literally anything and therefore means nothing at all. Terrific.

I set my LogLevel to debug, reproduced the issue and checked the ssl_error_log, where it reports the following:

[Wed Feb 21 13:55:07.118063 2024] [ssl:info] [pid 7769:tid 7944] [client 10.133.0.30:64845] AH01964: Connection to child 81 established (server undcflwebdev0.und.edu:443)
[Wed Feb 21 13:55:07.118300 2024] [ssl:debug] [pid 7769:tid 7944] ssl_engine_kernel.c(2391): [client 10.133.0.30:64845] AH02043: SSL virtual host for servername undcflwebdev0.und.edu found
[Wed Feb 21 13:55:07.118492 2024] [core:debug] [pid 7769:tid 7944] protocol.c(2460): [client 10.133.0.30:64845] AH03155: select protocol from , choices=h2,http/1.1 for server undcflwebdev0.und.edu
[Wed Feb 21 13:55:07.120744 2024] [ssl:debug] [pid 7769:tid 7944] ssl_engine_kernel.c(2251): [client 10.133.0.30:64845] AH02041: Protocol: TLSv1.3, Cipher: TLS_AES_256_GCM_SHA384 (256/256 bits)
[Wed Feb 21 13:55:07.120862 2024] [socache_shmcb:debug] [pid 7769:tid 7944] mod_socache_shmcb.c(508): AH00831: socache_shmcb_store (0x16 -> subcache 22)
[Wed Feb 21 13:55:07.120889 2024] [socache_shmcb:debug] [pid 7769:tid 7944] mod_socache_shmcb.c(745): AH00842: expiring 11 and reclaiming 0 removed socache entries
[Wed Feb 21 13:55:07.120901 2024] [socache_shmcb:debug] [pid 7769:tid 7944] mod_socache_shmcb.c(765): AH00843: we now have 0 socache entries
[Wed Feb 21 13:55:07.120911 2024] [socache_shmcb:debug] [pid 7769:tid 7944] mod_socache_shmcb.c(862): AH00847: insert happened at idx=0, data=(0:32)
[Wed Feb 21 13:55:07.120917 2024] [socache_shmcb:debug] [pid 7769:tid 7944] mod_socache_shmcb.c(865): AH00848: finished insert, subcache: idx_pos/idx_used=0/1, data_pos/data_used=0/230
[Wed Feb 21 13:55:07.120923 2024] [socache_shmcb:debug] [pid 7769:tid 7944] mod_socache_shmcb.c(530): AH00834: leaving socache_shmcb_store successfully
[Wed Feb 21 13:55:07.121011 2024] [ssl:debug] [pid 7769:tid 7944] ssl_engine_kernel.c(415): [client 10.133.0.30:64845] AH02034: Initial (No.1) HTTPS request received for child 81 (server undcflwebdev0.und.edu:443)
[Wed Feb 21 13:55:07.121357 2024] [authz_core:debug] [pid 7769:tid 7944] mod_authz_core.c(815): [client 10.133.0.30:64845] AH01626: authorization result of Require all granted: granted
[Wed Feb 21 13:55:07.121390 2024] [authz_core:debug] [pid 7769:tid 7944] mod_authz_core.c(815): [client 10.133.0.30:64845] AH01626: authorization result of <RequireAny>: granted
[Wed Feb 21 13:55:07.121491 2024] [authz_core:debug] [pid 7769:tid 7944] mod_authz_core.c(815): [client 10.133.0.30:64845] AH01626: authorization result of Require all granted: granted
[Wed Feb 21 13:55:07.121514 2024] [authz_core:debug] [pid 7769:tid 7944] mod_authz_core.c(815): [client 10.133.0.30:64845] AH01626: authorization result of <RequireAny>: granted
[Wed Feb 21 13:55:07.121680 2024] [proxy:debug] [pid 7769:tid 7944] mod_proxy.c(1520): [client 10.133.0.30:64845] AH01143: Running scheme unix handler (attempt 0)
[Wed Feb 21 13:55:07.121703 2024] [proxy_ajp:debug] [pid 7769:tid 7944] mod_proxy_ajp.c(795): [client 10.133.0.30:64845] AH00894: declining URL fcgi://localhost/var/www/html/databases/index.php
[Wed Feb 21 13:55:07.121719 2024] [proxy_fcgi:debug] [pid 7769:tid 7944] mod_proxy_fcgi.c(1069): [client 10.133.0.30:64845] AH01076: url: fcgi://localhost/var/www/html/databases/index.php proxyname: (null) proxyport: 0
[Wed Feb 21 13:55:07.121736 2024] [proxy_fcgi:debug] [pid 7769:tid 7944] mod_proxy_fcgi.c(1078): [client 10.133.0.30:64845] AH01078: serving URL fcgi://localhost/var/www/html/databases/index.php
[Wed Feb 21 13:55:07.121758 2024] [proxy:debug] [pid 7769:tid 7944] proxy_util.c(2568): AH00942: FCGI: has acquired connection for (*:80)
[Wed Feb 21 13:55:07.121776 2024] [proxy:debug] [pid 7769:tid 7944] proxy_util.c(2626): [client 10.133.0.30:64845] AH00944: connecting fcgi://localhost/var/www/html/databases/index.php to localhost:8000
[Wed Feb 21 13:55:07.121792 2024] [proxy:debug] [pid 7769:tid 7944] proxy_util.c(2662): [client 10.133.0.30:64845] AH02545: fcgi: has determined UDS as /run/php-fpm/www.sock
[Wed Feb 21 13:55:07.122008 2024] [proxy:debug] [pid 7769:tid 7944] proxy_util.c(2852): [client 10.133.0.30:64845] AH00947: connected /var/www/html/databases/index.php to httpd-UDS:0
[Wed Feb 21 13:55:07.122082 2024] [proxy:debug] [pid 7769:tid 7944] proxy_util.c(3222): AH02823: FCGI: connection established with Unix domain socket /run/php-fpm/www.sock (*:80)
[Wed Feb 21 13:55:07.150902 2024] [proxy:debug] [pid 7769:tid 7944] proxy_util.c(2584): AH00943: FCGI: has released connection for (*:80)
[Wed Feb 21 13:55:07.151140 2024] [ssl:debug] [pid 7769:tid 7944] ssl_engine_io.c(1151): [client 10.133.0.30:64845] AH02001: Connection closed to child 81 with standard shutdown (server undcflwebdev0.und.edu:443)

There aren't any obvious smoking guns, but I think the problem might be this line:

[Wed Feb 21 13:55:07.121703 2024] [proxy_ajp:debug] [pid 7769:tid 7944] mod_proxy_ajp.c(795): [client 10.133.0.30:64845] AH00894: declining URL fcgi://localhost/var/www/html/databases/index.php

If I understand correctly, FCGI is the interface between Apache and PHP. If FCGI is declining the connection, then the PHP script won't be interpreted and everything grinds to a halt.

We have SELinux installed, and it often causes problems, so I temporarily turned it off. But that didn't help -- the problem occurs even when SELinux is been put in permissive mode. So I don't think it's an SELinux issue.

Everything else on the server is working fine. I have multiple PHP apps living in their own folders that work without a hitch. So I reasoned perhaps it's something about the symbolic link. I deleted the link and replaced it with an ordinary folder named "databases" containing an index.html file that says "Hello world". It worked great. I renamed that index.html to index.php and it immediately failed with the same error messages as above. On a whim, I copied that index.php file to "test.php" and accessed that. And it worked fine!

Meanwhile, every other PHP app on the machine continues to work smoothly. Also, the installation of Drupal on the production server is running just fine despite being in a symbolic link. As far as I can tell, the problem only occurs for the exact path /var/www/html/databases/index.php, regardless of whether databases is an actual folder or a symbolic link. Literally every other PHP file on the server works fine.

I did some googling. I did not find anyone who ran into quite the same problem. I did find some people reporting the error code AH00894 on servers with reverse proxies in place. That gave me pause, because I do indeed have a reverse proxy set up. One of the other apps on the server is ancient, crufty and annoyingly irreplaceable. It won't run in anything newer than PHP 5.6. So I've got a docker container running providing a PHP 5.6 environment. Apache is configured to talk to the docker instance through a reverse proxy, thus:

# Listen to all traffic on port 80.
<VirtualHost *:80>

   # Keep the original host name.
   ProxyPreserveHost On

   # Redirect all traffic for the /archon folder to :8080 instead.
   ProxyPass /archon http://localhost:8080/
   ProxyPassReverse /archon http://localhost:8080/

</VirtualHost>

And again in ssl.conf I have the following bits:

SSLProxyEngine On

ProxyPreserveHost On

# Prevent SSL offloading
RequestHeader set X-Forwarded-Proto "https"
RequestHeader set X-Forwarded-Port "443"

# Redirect all traffic for the /archon folder to :8080 instead.
ProxyPass /archon http://localhost:8080/
ProxyPassReverse /archon http://localhost:8080/

The proxied app works fine. I can go to https://dev.example.com/archon/ and it loads up. But I wondered if maybe the proxying is somehow interfering with the drupal instance.

So I disabled all of the proxy stuff. I commented out the virtualhost and the all the Proxy config lines in ssl.conf, and restarted Apache. I verified that archon stopped working as expected, and then tried my Drupal. It generated the exact same error. So I turned all the proxy stuff back on.

At this point I've been banging my head against this problem for two days. If I've read the logs correctly, the problem is that FCGI is refusing the connection. But I don't know WHY it's refusing. Why do all of the other PHP apps on the server work fine? The production server is configured identically to the dev server, but production works and dev doesn't. Why? Every other PHP file on the server works like a charm. Why? What is it about this one URL that consistently dies when everything else works?

I could maybe rename the index.php file to something else and then use a .htaccess file to identify that file name as the index of the folder. That might be a workaround. But it's awfully awkward, and I worry that renaming Drupal's core file would cause unexpected issues. I don't know if any of Drupal's code has that file name hard coded anywhere.

I'm stuck. I don't know what else to try to figure out what the hell is wrong with the wretched thing. The new servers are supposed to go live Monday the 26th. I am rapidly running out of time, and this is my last blocker. Any suggestions would be appreciated.

I have a small Linux server (Ubuntu Server, 12.04 LTS) which is serving as a print release server. Essentially, I have defined a CUPS printer, and added Option job-hold-until indefinite to the entry in printers.conf so guests to the library can pay cash for their printouts, and someone at the desk can click "Release Job" in the CUPS web interface to send it to the printer.

The problem is that periodically the CUPS web interface will stop responding. The error log will fill up with messages like this:

E [10/Sep/2013:13:28:04 -0500] Unable to create certificate file /var/run/cups/certs/0 - Too many open files
E [10/Sep/2013:13:28:04 -0500] [CGI] Unable to create pipe for /usr/lib/cups/cgi-bin/jobs.cgi - Too many open files
E [10/Sep/2013:13:28:04 -0500] Unable to create certificate file /var/run/cups/certs/0 - Too many open files

The bulk of the messages have to do with files that have to do with SSL encryption for the web interface. It appears to be opening them and then never closing them, until it starts hitting the "Too many open files" error.

We don't really need SSL encryption for this. It's already fire-walled off to just IPs for staff computers inside the building. So the first time this happened, I attempted to disable SSL entirely by adding DefaultEncryption Never to cupsd.conf. We can access the web interface without SSL now, but the problem recurred this morning anyway.

So ... how do I persuade CUPS to 1) close the files when it's done with them, or B) not open them in the first place?

One of the programs in my cgi-bin folder is misbehaving. It's a compiled binary for which I do not have the source code. I would like to set up some kind of monitor to gather a list of the files that the program opens when it gets executed and dump that into a text file.

It looks like lsof might have that info, but since the CGI program is transient and typically finishes execution in under a second, it's not really practical to run lsof from the command line at the same time. For that matter, I'm not even sure if the CGI program would show up as itself or if the activity would be registered to the httpd process.

The server is running Red Hat Enterprise Linux 6.4 with Apache 2.2.15, and I have root access.

Suggestions?

I have a 64-bit Redhat Enterprise Linux server (version 6.3) that doesn't want to let me install the glibc.i686 module for 32-bit compatibility. When I try yum install glibc.i686 as root, it gives me a long string of errors like this:

Transaction Check Error:
  file /lib/libc.so.6 from install of glibc-2.12-1.80.el6_3.6.i686 conflicts with file from package libc6-6:2.15-1.x86_64
  file /lib/libm.so.6 from install of glibc-2.12-1.80.el6_3.6.i686 conflicts with file from package libc6-6:2.15-1.x86_64
  file /lib/libpthread.so.0 from install of glibc-2.12-1.80.el6_3.6.i686 conflicts with file from package libc6-6:2.15-1.x86_64

Here's the output of rpm -qa grepping for glibc and libc6:

# rpm -qa | grep glibc
glibc-common-2.12-1.80.el6_3.6.x86_64
glibc-2.12-1.80.el6_3.6.x86_64

# rpm -qa | grep libc6
libc6-2.15-1.x86_64

Google has not produced anything terribly helpful despite my best efforts.

Why do these two packages conflict, and how do I make them play nicely together?

EDIT:

Here is the output of yum repolist:

# yum repolist
Loaded plugins: product-id, rhnplugin, subscription-manager
Updating certificate-based repositories.
Unable to read consumer identity
repo id                   repo name                                                   status
rhel-x86_64-server-6      Red Hat Enterprise Linux Server (v. 6 for 64-bit x86_64)    8,824
repolist: 8,824

I am writing a BASH shell script to upload all the files in a directory to a remote server and then delete them. It'll run every few hours via a CRON job.

My complete script is below. The basic problem is that the part that's supposed to figure out whether the file uploaded successfully or not doesn't work. The SFTP command's exit status is always "0" regardless of whether the upload actually succeeded or not.

How can I figure out whether a file uploaded correctly or not so that I can know whether to delete it or let it be?

#!/bin/bash

# First, save the folder path containing the files.
FILES=/home/bob/theses/*

# Initialize a blank variable to hold messages.
MESSAGES=""
ERRORS=""


# These are for notifications of file totals.
COUNT=0
ERRORCOUNT=0

# Loop through the files.
for f in $FILES
do
    # Get the base filename
    BASE=`basename $f`

    # Build the SFTP command. Note space in folder name.
    CMD='cd "Destination Folder"\n'
    CMD="${CMD}put ${f}\nquit\n"


    # Execute it.
    echo -e $CMD | sftp -oIdentityFile /home/bob/.ssh/id_rsa [email protected]

    # On success, make a note, then delete the local copy of the file.
    if [ $? == "0" ]; then
        MESSAGES="${MESSAGES}\tNew file: ${BASE}\n"
        (( COUNT=$COUNT+1 ))

        # Next line commented out for ease of testing
        #rm $f
    fi

    # On failure, add an error message.
    if [ $? != "0" ]; then
        ERRORS="${ERRORS}\tFailed to upload file ${BASE}\n"
        (( ERRORCOUNT=$ERRORCOUNT+1 ))
    fi

done

SUBJECT="New Theses"

BODY="There were ${COUNT} files and ${ERRORCOUNT} errors in the latest batch.\n\n"

if [ "$MESSAGES" != "" ]; then
    BODY="${BODY}New files:\n\n${MESSAGES}\n\n"
fi

if [ "$ERRORS" != "" ]; then
    BODY="${BODY}Problem files:\n\n${ERRORS}"
fi

# Send a notification. 
echo -e $BODY | mail -s $SUBJECT [email protected]

Due to some operational considerations that make my head hurt, I cannot use SCP. The remote server is using WinSSHD on windows, and does not have EXEC privileges, so any SCP commands fail with the message "Exec request failed on channel 0". The uploading therefore has to be done via the interactive SFTP command.

I have a legacy app that needs FTP and cannot do SFTP.

My solution is:

• put an FTP server in place using VSFTPD
• configure the firewall to accept port 21 connections only from localhost
• Set up an SSH connection from the client with the legacy app
• Tunnel the FTP through SSH

I'm wondering though if I can configure VSFTPD to ignore connections from anywhere but localhost on its own, in addition to the firewall. Belt and bracers both.

I have a RHEL box running PHP 5.3.3, which was installed using the binary packages provided by yum. I have installed the php-pdo package:

# yum info php-pdo
Loaded plugins: product-id, rhnplugin, subscription-manager
Updating Red Hat repositories.
Installed Packages
Name        : php-pdo
Arch        : x86_64
Version     : 5.3.3
Release     : 3.el6_1.3
Size        : 168 k
Repo        : installed
From repo   : rhel-x86_64-server-6
Summary     : A database access abstraction module for PHP applications
URL         : http://www.php.net/
License     : PHP
Description : The php-pdo package contains a dynamic shared object that will add
            : a database access abstraction layer to PHP.  This module provides
            : a common interface for accessing MySQL, PostgreSQL or other
            : databases.

It appears to be working correctly for SQLite databases, but not MySQL. There's no file including pdo_mysql.so in /etc/php.d, and there is no copy of pdo_mysql.so in /usr/lib64/php/modules.

I'm pretty sure I just need the driver file and a line in the PHP configuration. A yum search pdo mysql didn't turn up any useful packages, and Google has failed me. If I were on Ubuntu or Debian, I'd apt-get install php5-mysql and be done with it.

So ... where in Red Hat land do I get a copy of pdo_mysql.so, and install it properly?

I use the mysql command line client frequently. Sometimes I wind up typing in my password dozens of times per day, and I'm getting really tired of it.

SSH has a neat utility called ssh-agent which lets you type in your password just once at the beginning of the day. Is there a similar utility for MySQL?

Okay, I have this code:

RewriteCond %{HTTPS} off
RewriteCond %{REQUEST_URI} ^/borrowing/ill/request\.php$
RewriteRule ^.*$ https://%{HTTP_HOST}%{REQUEST_URI} [R,L]

The way I would expect this to work is:

1. A request for /borrowing/ill/request.php comes in on HTTP.
2. The rule matches.
3. The server redirects to HTTPS.
4. The rule does not match, because HTTPS is now on.

The way it actually works is:

1. A request for /borrowing/ill/request.php comes in on HTTP.
2. The rule matches.
3. The server redirects to HTTPS.
4. The rule matches.
5. The server redirects to HTTPS.
6. The rule matches.
7. The server redirects to HTTPS ...

And so on.

I know that the second condition (matching the file name) is working, because the redirect loop only hits that specific page. The question is, why isn't the switch to HTTPS causing the first condition to not match?

EDIT:

I put the exact same .htaccess rules into a test area on another server -- same file and path info. And they worked just fine. There's got to be something wrong with the server configuration elsewhere.

I'm setting up SSL on an Ubuntu server. One of fields it asks for as part of setting up the CSR is a "challenge password". What is that? The default is blank. Do I need to enter one?

I'm trying to debug a problem with sending mail on a Red Hat/Apache server. When I send email from the command line using sendmail -t, I get bounce-back emails with messages like this:

The original message was received at Thu, 7 Apr 2011 10:24:56 -0500
from internal.example.edu [127.0.0.1]

   ----- The following addresses had permanent fatal errors -----
<[email protected]>

   ----- Transcript of session follows -----
<[email protected]>... Deferred: Connection timed out with mail.global.frontbridge.com.
Message could not be delivered for 5 days
Message will be deleted from queue

--p3CFqmnx022071.1302624414/internal.example.edu
Content-Type: message/delivery-status

Reporting-MTA: dns; internal.example.edu
Arrival-Date: Thu, 7 Apr 2011 10:24:56 -0500

Final-Recipient: RFC822; [email protected]
Action: failed
Status: 4.4.7
Remote-MTA: DNS; mail.global.frontbridge.com
Last-Attempt-Date: Tue, 12 Apr 2011 11:06:54 -0500

That timeout talking to mail.global.frontbridge.com is likely the cause of the problem. BUT, it has been suggested that this particular timeout was caused because sending the test email from the command line caused the local MTA to identify itself using the name "internal.example.edu", causing the remote MTA to reject the message because it didn't recognize our interal-only DNS.

The original problem was identified when some PHP scripts on the server started failing to send mail correctly. Those scripts are accessed through a web browser (obviously) using a valid external name for the server.

So basically, I want to check the apache user's local mail spool to see if there are bounce emails there that would take the internal/external name thing out of the equation and possibly shed some more light on the matter. Is that possible? Does the apache user even HAVE a mail spool?

I tried sending some mails with the Return-Path header set to an email address I control ... but since the whole problem is that the server can't send email right, that didn't work.

I often use the mysql command-line client on an Ubuntu box. It works fine, except that the key bindings are wrong. Specifically, pressing "Home", "End", or "Delete" makes it output a ~ (tilde) character, and beep.

I've determined that Home/End can be done by pressing Control+A and Control+E respectively, as on the Mac terminal. I'm on a Windows box, using PuTTY to talk to Ubuntu, but the home/end/delete keys work fine on the regular command line. I'm pretty sure it's an issue specific to the mysql client.

I checked the man file but didn't find anything useful. Is there any way to bind the Home/End/Delete keys to their actual functions? Maybe a config file in my home directory or something?

Every time I move back a few spaces to edit a query and press delete, I just wind up having to go forward again so I can use the Backspace key to delete not only the incorrect query but also the ~ that the delete key puts in.

It's driving me slowly but surely to madness. Madness, I tell you, MADNESS!