Questions[amazon-linux-2](server)

Forgive the cross post from Stack Overflow but I realized this was probably the better location to ask.

I recently rebuilt my server on AWS, upgrading from Amazon Linux to Amazon Linux 2, which like Linux 1 appears to be a fork of CentOS, now CentOS 7. My vhost config remains the same as the previous build while all the other conf files are the newer default files. (Just FYI, not sure it has any bearing).

Since the rebuild I can no longer do systemctl restart httpd. I get the error The Apache HTTP Server httpd pid (NNNN) already running.

I have done lots of googling and see that quite a few people have this error but it always turns out to be something different that they have done. I cannot think of anything "different" I did.

I have learned that the PID that I see as "already running", owned by root, is the "master process" and all the other PIDs, owned by apache, are the "worker processes". CentOS httpd running as both root and apache user

root      4461     1  0 07:18 ?        00:00:00 /usr/sbin/httpd -k restart
apache    4467  4461  0 07:18 ?        00:00:09 /usr/sbin/httpd -k restart
apache    4468  4461  0 07:18 ?        00:00:08 /usr/sbin/httpd -k restart
apache    4471  4461  0 07:18 ?        00:00:08 /usr/sbin/httpd -k restart
apache    4477  4461  0 07:18 ?        00:00:08 /usr/sbin/httpd -k restart
apache    4498  4461  0 07:18 ?        00:00:07 /usr/sbin/httpd -k restart
apache    5236  4461  0 07:19 ?        00:00:08 /usr/sbin/httpd -k restart
apache    5248  4461  0 07:19 ?        00:00:09 /usr/sbin/httpd -k restart
apache    5987  4461  0 07:20 ?        00:00:09 /usr/sbin/httpd -k restart
apache    5993  4461  0 07:20 ?        00:00:08 /usr/sbin/httpd -k restart
apache    5994  4461  0 07:20 ?        00:00:09 /usr/sbin/httpd -k restart
# systemctl restart httpd;systemctl status httpd;
Job for httpd.service failed. See "systemctl status httpd.service" and "journalctl -xe" for details.
● httpd.service - The Apache HTTP Server
   Loaded: loaded (/usr/lib/systemd/system/httpd.service; disabled; vendor preset: disabled)
   Active: failed (Result: protocol) since Thu 2021-12-23 19:56:05 UTC; 4ms ago
     Docs: man:httpd.service(8)
  Process: 31799 ExecStart=/usr/sbin/httpd $OPTIONS -DFOREGROUND (code=exited, status=0/SUCCESS)
 Main PID: 31799 (code=exited, status=0/SUCCESS)
   Status: "Reading configuration..."

Dec 23 19:56:04 ip-10-5-11-55.ec2.internal systemd[1]: Starting The Apache HTTP Server...
Dec 23 19:56:05 ip-10-5-11-55.ec2.internal httpd[31799]: httpd (pid 4461) already running
Dec 23 19:56:05 ip-10-5-11-55.ec2.internal systemd[1]: Failed to start The Apache HTTP Server.
Dec 23 19:56:05 ip-10-5-11-55.ec2.internal systemd[1]: Unit httpd.service entered failed state.
Dec 23 19:56:05 ip-10-5-11-55.ec2.internal systemd[1]: httpd.service failed.

Nothing unusual in the Apache error log:

[Thu Dec 23 07:18:58.777745 2021] [suexec:notice] [pid 4460] AH01232: suEXEC mechanism enabled (wrapper: /usr/sbin/suexec)
[Thu Dec 23 07:18:58.802753 2021] [lbmethod_heartbeat:notice] [pid 4461] AH02282: No slotmem from mod_heartmonitor
[Thu Dec 23 07:18:58.802850 2021] [http2:warn] [pid 4461] AH10034: The mpm module (prefork.c) is not supported by mod_http2. The mpm determines how things are processed in your server. HTTP/2 has more
[Thu Dec 23 07:18:58.847386 2021] [mpm_prefork:notice] [pid 4461] AH00163: Apache/2.4.51 () PHP/7.4.21 configured -- resuming normal operations
[Thu Dec 23 07:18:58.847414 2021] [core:notice] [pid 4461] AH00094: Command line: '/usr/sbin/httpd'

If I kill this "master" PID, then I can do restarts no problem.

# kill 4461
# systemctl restart httpd;systemctl status httpd;
● httpd.service - The Apache HTTP Server
   Loaded: loaded (/usr/lib/systemd/system/httpd.service; disabled; vendor preset: disabled)
   Active: active (running) since Thu 2021-12-23 20:01:31 UTC; 6ms ago
     Docs: man:httpd.service(8)
 Main PID: 3091 (httpd)
   Status: "Configuration loaded."
   CGroup: /system.slice/httpd.service
           └─3091 /usr/sbin/httpd -DFOREGROUND

Dec 23 20:01:31 ip-10-5-11-55.ec2.internal systemd[1]: Starting The Apache HTTP Server...
Dec 23 20:01:31 ip-10-5-11-55.ec2.internal systemd[1]: Started The Apache HTTP Server.
# ps -Af | grep httpd
root      3145     1  1 20:01 ?        00:00:00 /usr/sbin/httpd -DFOREGROUND
apache    3147  3145  0 20:01 ?        00:00:00 /usr/sbin/httpd -DFOREGROUND
apache    3153  3145  0 20:01 ?        00:00:00 /usr/sbin/httpd -DFOREGROUND
apache    3171  3145  0 20:01 ?        00:00:00 /usr/sbin/httpd -DFOREGROUND
apache    3177  3145  0 20:01 ?        00:00:00 /usr/sbin/httpd -DFOREGROUND
apache    3183  3145  0 20:01 ?        00:00:00 /usr/sbin/httpd -DFOREGROUND

Most people's advice is "just kill the process", Apache failes to start, already running but not maching pid file? but I need this to work unattended. What else could I be looking at?

TDLR: I have a Catch 22 where, depending on permissions on the user's home directory, I can get the SSH authentication to work, or the user directory constraints, but not both.

BTW, I really want to roll my own SFTP server. Please don't recommend I try AWS Transfer service or something alternative. Thanks.

Here is relevant (changed from default) content in /etc/ssh/sshd_config:

Subsystem sftp internal-sftp
Port 22
Port 2299
Match Group sftpusers LocalPort 2299
  ChrootDirectory /sftp-data/%u
  ForceCommand internal-sftp
Match Group sftpusers LocalPort 22
  DenyGroups sftpusers
Match LocalPort 2299  Group *,!sftpusers
  DenyUsers *

I want port 22 functioning as ssh typically does, but only for non-sftp users. For sftp users, in group "sftpusers", I want port 2299 to function as sftp only, and not ssh. For non-sftpusers, I want access to port 2299 denied.

Ok, so, I created a user "user1" with home directory /sftp-data/user1 and shell /sbin/nologin. I created /sftp-data/user1/.ssh/authorized_keys and populated that with the public ssh key. /sftp-data is owned by root with 700 permissions. /sftp-data/user1/.ssh and below are owned by user1, and /sftp-data/user1/.ssh/authorized_keys has permission 600. The ownership/permissions of /sftp-data/user1 are under question here. More below.

I created the user group sftpusers and added user1 to that group. The built-in ec2-user you get with AWS is not a member of that group, however. Testing with ec2-user worked great: access via ssh, port 22 worked as it always does, but no access to port 2299.

So, testing with user1 is where it got interesting. User1 has no access to port 22 - that's great! With user1 owning /sftp-data/user1, the ssh public key authentication succeeds on port 2299, but the user is immediately logged out, with this message saved in /var/log/secure:

Sep  2 19:21:38 ip-192-168-0-25 sshd[10369]: Accepted publickey for user1 from <ip-address redacted> port 61110 ssh2: ECDSA SHA256:<sha redacted>
Sep  2 19:13:23 ip-192-168-0-25 sshd[9803]: pam_unix(sshd:session): session opened for user user1 by (uid=0)
Sep  2 19:13:23 ip-192-168-0-25 sshd[9803]: fatal: bad ownership or modes for chroot directory "/sftp-data/user1" [postauth]
Sep  2 19:13:23 ip-192-168-0-25 sshd[9803]: pam_unix(sshd:session): session closed for user user1

Sure, that makes sense. Chroot requires /sftp-data/user1 to be owned by root, permissions 700. So, make that so, and now the sftp (ssh key) authentication fails.

Sep  2 19:41:00 ip-192-168-0-25 sshd[11693]: error: AuthorizedKeysCommand /opt/aws/bin/eic_run_authorized_keys user1 SHA256:<sha redacted> failed, status 22

BTW, eic_run_authorized_keys is a wrapper AWS puts around standard ssh authentication to enable AWS Instance Connect.

For extra credit... if the above problem is not challenging enough, can you come up with a scheme where I can give particular sftp users access to particular project directories, and only those, without creating a group for every project? Link from each user's home directory to a project directory would be awesome.

Additional info requested by @anx:

# getent passwd user1
user1:x:1001:1001::/sftp-data/user1:/sbin/nologin
# namei -l /sftp-data/user1/.ssh/authorized_keys
f: /sftp-data/user1/.ssh/authorized_keys
dr-xr-xr-x root  root      /
drwxr-xr-x root  root      sftp-data
drwx------ root  root      user1
drwx------ user1 sftpusers .ssh
-rw------- user1 sftpusers authorized_keys

I turned on DEBUG logging for sshd. Using the ChrootDirectory directive, with /sftp-data/user1 owned by root, and SSH authentication failing, I see this in /var/log/secure:

debug1: Could not open authorized keys '/sftp-data/user1/.ssh/authorized_keys': Permission denied

ps clearly shows me that root is running the sshd process.

I'm messing around with some timeout settings, and am trying to figure out the correct way to set things for systemd/system daemons. Specifically, this is an underpowered server, and I keep timing out on starting clamd, so I'm trying to figure out how high I need to set the TimeoutStartSec parameter. I tried

systemctl edit --full [email protected]

and edited it to say

TimeoutStartSec=20min

then did systemctl daemon-reload

But it kept mysteriously timing out after 7 minutes. Eventually, I found that /usr/lib/systemd/system/[email protected] had TimeoutStartSec=420.

Probably against all Right Way to do things, I edited the /usr/lib version of the file, and commented out that line. Sure enough, that changed things, but it still doesn't seem to be reading the /etc/systemd version. Once I commented it out, it start timing out after 90 seconds, which I guess must be the default.

As a temporary workaround, I've edited the /usr/lib version to be my desired 20 minutes, but ... this doesn't seem like the right way to do things.

Is systemctl edit supposed to be editing the version in /etc/systemd/system? Is this the putatively correct way to do things? Do I have to do something more than systemctl daemon-reload?

Open to any and all suggestions on The Right Way to do this.

TIA

I'm running Amazon Linux 2, on EC2 instances in AWS. I want to be able to add my own iptables rules, and have them survive reboots.

What is the correct way (or a correct way) to do this?

We're trying to migrate our Java applications from from the current Elastic Beanstalk JDK 8 platforms to the new ones running Corretto 11 on Amazon Linux 2. The app works well, but the way logs are handled changed. The output from the web process is now stored in /var/log/web.stdout.log and every line is prefixed with the timestamp and process name, ie:

May 20 17:00:00 ip-10-48-41-129 web: {"timestamp":"2020-05-20T17:00:00.035Z","message":"...","logger":"...","thread":"MessageBroker-2","level":"INFO"}

How can we get rid of the prefix? These logs are streamed into CloudWatch and we're outputting them to stdout in JSON so that we can later query them with Logs Insights. But with the prefix, Insights doesn't "see" the JSON and just treats the whole line as a text blob.

I cannot find any documentation for that on AWS. Almost all of Elastic Beanstalk's documentation refers to the first version of Amazon Linux.