Egyas's questions

I have a simply Splunk set-up.  about 120 or so Linux servers (that are all basically appliances) w/ universal forwarder installed, and a single Linux server running Splunk Enterprise acting as the indexer, search head, etc.

The problem I have is that the forwarders must feed the server's audit log into Splunk.  That feed is actually working fine, but it's flooding the server, and causing me to go over my license limit.  

Specifically, the appliance app has an event in cron that runs very often, and it's flooding the audit log with file access, file mod, etc events, which is ballooning the amount of data I send to Splunk Enterprise.  Data that Is simply do not need.

What I want to do is filter out these specific events, but ONLY for this specific user.  I believe this can be done using transforms.conf and props.conf  on the indexer, but I'm having trouble getting the syntax and fields right.

Can anyone assist with this?

Here's the data I need to remove... sourcetype=auditd acct=appuser exe=/usr/sbin/crond exe=/usr/bin/crontab

So basically ANY events in the audit log for user "appuser" that reference either "/usr/bin/crontab" or "usr/bin/crontab" need to be dropped.

Here's an example of the events I want to drop.

type=USER_END msg=audit(03/04/2024 15:58:02.701:5726) : pid=26919 uid=root auid=appuser ses=184 msg='op=PAM:session_close grantors=pam_loginuid,pam_keyinit,pam_limits,pam_systemd acct=appuser exe=/usr/sbin/crond hostname=? addr=? terminal=cron res=success'

type=USER_ACCT msg=audit(03/04/2024 15:58:02.488:5723) : pid=26947 uid=appuser auid=appuser ses=184 msg='op=PAM:accounting grantors=pam_access,pam_unix,pam_localuser acct=appuser exe=/usr/bin/crontab hostname=? addr=? terminal=cron res=success'

Can this be done?

I have some Linux servers that are getting errors like the below in the logs...

auditd[1074]: Error receiving audit netlink packet (No buffer space available)

I know HOW to resolve the issue (tweak the audit buffer setting in audit.rules), but I'm wondering WHAT is the impact of this?

Am I actually losing auditd events in the blog? Is it failing to write the events when it runs out of buffer space?

I have been Googling, but I haven't found a concrete answer.

I'm wanting to implement some rate-limiting onto our named servers and am looking for some help on making sure the values are "sane". This is what I'm thinking...

rate-limit { errors-per-second 2;responses-per-second 15; window 60; };

Even after reading the docs, I'm still not 100% sure on how "window" is working in the case, so I just wanted to get an outside opinion. Do these values look 'sane' for general-purpose DDOS protection? Thoughts?

I just built a linux bastion, let's call it "bastion1" (IP: 66.66.66.6) on RHEL 8 to replace an older RHEL 6 bastion "bastion0" (IP: 77.77.77.7) that is doing the exact same function. The two servers are configured the same (we use salt to push configurations, etc), The IPtables setup is also fine (all necessary entries have been duplicated for the new IP, etc). For this issue, let's assume my VPN IP is 55.55.55.5, and my username is "user1".

I can successfully ssh from my linux laptop to "bastion1", then ssh from "bastion1" to other servers on our network (in this example, let's call it "host1.ournetwork.com"). So far so good.

We use a config locally (ie: on my laptop) to make ssh "jump" through the bastion to get to another host. This is what's not working. When I say "ssh host1.ournetwork.com" it goes to the bastion, asks for my login, accept it successfully, then tries to get to "host1", and fails. It throws this error...

channel 0: open failed: connect failed: open failed
stdio forwarding failed
kex_exchange_identification: Connection closed by remote host

Looking at the logs, "host1" shows nothing at all in the logs. "bastion1" shows this in the secure log...

Dec 29 17:25:23 bastion1 sshd[607500]: Accepted password for user1 from 55.55.55.5 port 39028 ssh2
Dec 29 17:25:23 bastion1 sshd[607500]: pam_unix(sshd:session): session opened for user user1 by (uid=0)
Dec 29 17:25:23 bastion1 sshd[607505]: error: connect to host1.ournetwork.com port 22 failed: Permission denied
Dec 29 17:25:23 bastion1 sshd[607500]: pam_unix(sshd:session): session closed for user user1

Obviously I have anonymized the specific info.

My local ssh config file has these entries in it....

# US2 bastion.
Host bastion1
 HostName 66.66.66.6
 User user1
 port 22
 ForwardAgent yes
 Pubkeyauthentication yes
 CertificateFile ~/.ssh/id_rsa-cert.pub

Host *.ournetwork.com
 ProxyCommand ssh -A -W %h:%p bastion1
 port 22
 User user1
 Pubkeyauthentication yes
 CertificateFile ~/.ssh/id_rsa-cert.pub

So when I type locally "ssh host1.ournetwork.com" it tries to ssh to "bastion1" (66.66.66.6) and asks for password. When it authenticated successfully, it then jumps to "host1.ournetwork.com", where it asks for my password again. This setup has worked successfully for a long time w/ our current rhel6 bastion. Let's assume that it's IP was "77.77.77.7". So all I did locally once "bastion1" came online was change the IP in my local ssh config from 77.77.77.7 to 66.66.66.6

Here's what I get when I try to ssh now...

→ ssh host1.ournetwork.com

                       WARNING!
========================================================
 All access to this machine is monitored. The following
 actions are criminal offences and it is our company
 policy to prosecute against:
 ** Unauthorised access to this computer
 ** Unauthorised viewing, copying or deleting data
 ** Unauthorised tampering of data
 ** Unauthorised use of this computer to access other computers.

========================================================

[email protected]'s password: 
channel 0: open failed: connect failed: open failed
stdio forwarding failed
kex_exchange_identification: Connection closed by remote host

Here's what I should be seeing, and what I see using the old bastion "bastion0"...

→ ssh host1.ournetwork.com

                       WARNING!
========================================================
 All access to this machine is monitored. The following
 actions are criminal offences and it is our company
 policy to prosecute against:
 ** Unauthorised access to this computer
 ** Unauthorised viewing, copying or deleting data
 ** Unauthorised tampering of data
 ** Unauthorised use of this computer to access other computers.

========================================================

[email protected]'s password: 

                       WARNING!
========================================================
 All access to this machine is monitored. The following
 actions are criminal offences and it is our company
 policy to prosecute against:
 ** Unauthorised access to this computer
 ** Unauthorised viewing, copying or deleting data
 ** Unauthorised tampering of data
 ** Unauthorised use of this computer to access other computers.

========================================================

[email protected]'s password: 
Last login: Tue Dec 29 17:01:29 2020 from 66.66.66.6

I'm guessing that I'm just missing something simple, but I'm not great w/ ssh tunnels, etc so I can't figure out what I missed. Thoughts?

Edited to add...

Figured that someone would ask for a "-v" output, so here it is.

Here's what I see using the new "bastion1"...

→ ssh -v host1.ournetwork.com
OpenSSH_8.2p1 Ubuntu-4ubuntu0.1, OpenSSL 1.1.1f  31 Mar 2020
debug1: Reading configuration data /home/user1/.ssh/config
debug1: /home/user1/.ssh/config line 30: Applying options for *.ournetwork.com
debug1: /home/user1/.ssh/config line 51: Applying options for *
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: /etc/ssh/ssh_config line 19: include /etc/ssh/ssh_config.d/*.conf matched no files
debug1: /etc/ssh/ssh_config line 21: Applying options for *
debug1: Executing proxy command: exec ssh -A -W host1.ournetwork.com:22 bastion1
debug1: identity file /home/user1/.ssh/id_rsa type -1
debug1: identity file /home/user1/.ssh/id_dsa type -1
debug1: identity file /home/user1/.ssh/id_ecdsa type -1
debug1: identity file /home/user1/.ssh/id_ecdsa_sk type -1
debug1: identity file /home/user1/.ssh/id_ed25519 type -1
debug1: identity file /home/user1/.ssh/id_ed25519_sk type -1
debug1: identity file /home/user1/.ssh/id_xmss type -1
debug1: certificate file /home/user1/.ssh/id_rsa-cert.pub type 4
debug1: Local version string SSH-2.0-OpenSSH_8.2p1 Ubuntu-4ubuntu0.1

                       WARNING!
========================================================
 All access to this machine is monitored. The following
 actions are criminal offences and it is our company
 policy to prosecute against:
 ** Unauthorised access to this computer
 ** Unauthorised viewing, copying or deleting data
 ** Unauthorised tampering of data
 ** Unauthorised use of this computer to access other computers.

========================================================

[email protected]'s password: 
channel 0: open failed: connect failed: open failed
stdio forwarding failed
kex_exchange_identification: Connection closed by remote host

Here's what I see using "bastion0" which actually works...

→ ssh -v host1.ournetwork.com
OpenSSH_8.2p1 Ubuntu-4ubuntu0.1, OpenSSL 1.1.1f  31 Mar 2020
debug1: Reading configuration data /home/user1/.ssh/config
debug1: /home/user1/.ssh/config line 30: Applying options for *.ournetwork.com
debug1: /home/user1/.ssh/config line 51: Applying options for *
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: /etc/ssh/ssh_config line 19: include /etc/ssh/ssh_config.d/*.conf matched no files
debug1: /etc/ssh/ssh_config line 21: Applying options for *
debug1: Executing proxy command: exec ssh -A -W host1.ournetwork.com:22 bastion1
debug1: identity file /home/user1/.ssh/id_rsa type -1
debug1: identity file /home/user1/.ssh/id_dsa type -1
debug1: identity file /home/user1/.ssh/id_ecdsa type -1
debug1: identity file /home/user1/.ssh/id_ecdsa_sk type -1
debug1: identity file /home/user1/.ssh/id_ed25519 type -1
debug1: identity file /home/user1/.ssh/id_ed25519_sk type -1
debug1: identity file /home/user1/.ssh/id_xmss type -1
debug1: certificate file /home/user1/.ssh/id_rsa-cert.pub type 4
debug1: Local version string SSH-2.0-OpenSSH_8.2p1 Ubuntu-4ubuntu0.1

                       WARNING!
========================================================
 All access to this machine is monitored. The following
 actions are criminal offences and it is our company
 policy to prosecute against:
 ** Unauthorised access to this computer
 ** Unauthorised viewing, copying or deleting data
 ** Unauthorised tampering of data
 ** Unauthorised use of this computer to access other computers.

========================================================

[email protected]'s password: 
debug1: Remote protocol version 2.0, remote software version OpenSSH_5.3
debug1: match: OpenSSH_5.3 pat OpenSSH_5* compat 0x0c000002
debug1: Authenticating to host1.ournetwork.com:22 as 'user1'
debug1: SSH2_MSG_KEXINIT sent
debug1: SSH2_MSG_KEXINIT received
debug1: kex: algorithm: diffie-hellman-group-exchange-sha256
debug1: kex: host key algorithm: ssh-rsa
debug1: kex: server->client cipher: aes128-ctr MAC: hmac-sha2-256 compression: none
debug1: kex: client->server cipher: aes128-ctr MAC: hmac-sha2-256 compression: none
debug1: SSH2_MSG_KEX_DH_GEX_REQUEST(2048<8192<8192) sent
debug1: got SSH2_MSG_KEX_DH_GEX_GROUP
debug1: SSH2_MSG_KEX_DH_GEX_INIT sent
debug1: got SSH2_MSG_KEX_DH_GEX_REPLY
debug1: Server host key: ssh-rsa SHA256:12Twz9Tp+BLbi91KWZ1gIyA3kNKns64hIK6BXkZcsls
debug1: Host 'host1.ournetwork.com' is known and matches the RSA host key.
debug1: Found key in /home/user1/.ssh/known_hosts:37
debug1: rekey out after 4294967296 blocks
debug1: SSH2_MSG_NEWKEYS sent
debug1: expecting SSH2_MSG_NEWKEYS
debug1: SSH2_MSG_NEWKEYS received
debug1: rekey in after 4294967296 blocks
debug1: Will attempt key: /home/user1/.ssh/id_rsa-cert.pub RSA-CERT SHA256:ABJwputoncHL/SXD48hdFTH7gomP59BQEJxW/gGNa28 explicit
debug1: Will attempt key: /home/user1/.ssh/id_rsa 
debug1: Will attempt key: /home/user1/.ssh/id_dsa 
debug1: Will attempt key: /home/user1/.ssh/id_ecdsa 
debug1: Will attempt key: /home/user1/.ssh/id_ecdsa_sk 
debug1: Will attempt key: /home/user1/.ssh/id_ed25519 
debug1: Will attempt key: /home/user1/.ssh/id_ed25519_sk 
debug1: Will attempt key: /home/user1/.ssh/id_xmss 
debug1: SSH2_MSG_SERVICE_ACCEPT received

                       WARNING!
========================================================
 All access to this machine is monitored. The following
 actions are criminal offences and it is our company
 policy to prosecute against:
 ** Unauthorised access to this computer
 ** Unauthorised viewing, copying or deleting data
 ** Unauthorised tampering of data
 ** Unauthorised use of this computer to access other computers.

========================================================

debug1: Authentications that can continue: publickey,password
debug1: Next authentication method: publickey
debug1: Offering public key: /home/user1/.ssh/id_rsa-cert.pub RSA-CERT SHA256:ABJwputoncHL/SXD48hdFTH7gomP59BQEJxW/gGNa28 explicit
debug1: Server accepts key: /home/user1/.ssh/id_rsa-cert.pub RSA-CERT SHA256:ABJwputoncHL/SXD48hdFTH7gomP59BQEJxW/gGNa28 explicit
debug1: Trying private key: /home/user1/.ssh/id_rsa
debug1: Trying private key: /home/user1/.ssh/id_dsa
debug1: Trying private key: /home/user1/.ssh/id_ecdsa
debug1: Trying private key: /home/user1/.ssh/id_ecdsa_sk
debug1: Trying private key: /home/user1/.ssh/id_ed25519
debug1: Trying private key: /home/user1/.ssh/id_ed25519_sk
debug1: Trying private key: /home/user1/.ssh/id_xmss
debug1: Next authentication method: password
[email protected]'s password: 
debug1: Authentication succeeded (password).
Authenticated to host1.ournetwork.com (via proxy).
debug1: channel 0: new [client-session]
debug1: Requesting [email protected]
debug1: Entering interactive session.
debug1: pledge: proc
debug1: Sending environment.
debug1: Sending env LANG = en_US.UTF-8
Last login: Tue Dec 29 18:25:58 2020 from 77.77.77.7

I'm replacing a RHEL6 server that runs, among other things, bind/named, with a RHEL8 server. As part of this, the bind version updated from 9.8 to 9.11. I'm not a bind wiz in any way, but I foolishly thought this should be a simple update. Unfortunately I ran into a problem I barely even understand, being only slightly familiar w/ bind. Apparently you can no longer have multiple views pointing to the same writable file? It's preventing named from starting.

I did some Googling, and found out that this is indeed the "proper" reaction ever since bind 9.10, and that I am supposed to incorporate in-view rewferences to get around this.

Here's a snip of the errors I get...

Dec 03 16:45:51 server-1a bash[97204]: /etc/named/slave.zones:4: writeable file 'data/test.exampledomain.com.zone': already in use: /etc/named/slave.zones:4
Dec 03 16:45:51 server-1a bash[97204]: /etc/named/slave.zones:10: writeable file 'data/test2.exampledomain.com.zone': already in use: /etc/named/slave.zones:10
Dec 03 16:45:51 server-1a bash[97204]: /etc/named/slave.zones:16: writeable file 'data/test3.exampledomain.com.zone': already in use: /etc/named/slave.zones:16

I'm hoping that someone has the talent and willingness to help me update my config appropriately, since I'm lost.

Here's the (anonymised) portion of my named.conf file that needs to be updated I believe. Thoughts?

view "localhost_resolver" {
    match-clients { localhost; };
    allow-query { localhost; };
    allow-recursion { localhost; };
    recursion yes;
    include "/etc/named/root.hints";
    include "/etc/named.rfc1912.zones";
        include "/etc/named/slave.zones";
        include "/etc/named/forwarder.zones";

};

view "abc_clients" {
  response-policy { zone "abc"; };
  match-clients { 1.2.3.4/24; };
  allow-query { 1.2.3.4/24; };
  allow-recursion { 1.2.3.4/24; };
  recursion yes;
  include "/etc/named/abc.conf.local";
  include "/etc/named/root.hints";
  include "/etc/named/slave.zones";
};
view "trusted_resolver" {
    match-clients { 1.9.0.0/21;1.8.0.0/21;1.7.0.0/24;1.6.0.0/21; };
    allow-query { trusted; };
    allow-transfer { "none"; };
    allow-recursion { trusted; };
        recursion yes;
        include "/etc/named/root.hints";
        include "/etc/named/slave.zones";
        include "/etc/named/forwarder.zones";
};
view "default" {
       match-clients { any; };
    allow-transfer { "none"; };
       recursion no;
};

Currently, we use Snoopy to monitor all commands issued by users on some externally accessible servers. We're in the process of updating everything to RHEL8 to ensure supportability and compliance, and discovered that my beloved Snoopy is no longer maintained. So it won't pass the compliance audit and needs to be replaced.

I looked into using auditd to do it, by enabling "pam_tty_audit.so" in system-auth and password-auth. This did the trick, but the output is, well let's just say it's less then desirable. Not to mention basically unreadable.

I tried setting-up /etc/profile to log by adding this...

function log2syslog
{
   declare COMMAND
   COMMAND=$(fc -ln -0)
   logger -p local1.notice -t bash -i -- "${USER}:${COMMAND}"
}
trap log2syslog DEBUG

And adding this to /etc/rsyslog.conf

local1.* -/var/log/cmdline

It works GREAT! But the solution was declined because it can be overridden by users.

I even tried using rootsh as a shell for users and logging that. Logs well, but there's no time/date stamps on it. So not acceptable.

So back to the question at hand. I need a replacement for Snoopy, that logs EVERY command executed, in a readable format with time/stamps, that users cannot override.

Any thoughts?

I had to recently rebuild my laptop. In the process, I switched from Fedora31 to Kubuntu 20.04 LTS. Everything in the switch went without a hitch, except for one thing. Where I work we use 2FA for all logins, and utilize a yubi key for this purpose. I thought I had everything set-up correctly, but whenever I try to ssh to a server now (and use PIV) I get this error...

Enter passphrase for PKCS#11:  Could not add card
"/usr/lib/x86_64-linux-gnu/libykcs11.so": agent refused operation

Now, every time I reboot the system, etc I have to re-add the card as normal. This shows that it was properly added already.

ssh-add -s /usr/lib/x86_64-linux-gnu/libykcs11.so 
Enter passphrase for PKCS#11: 
Card added: /usr/lib/x86_64-linux-gnu/libykcs11.so

Despite this, it's still throwing that annoying error at me. Now I CAN just manually enter my PW and hit the Yubi and log in. So it's not a show-stopper. But we're supposed to be able to just PIV through it, and it's that which is not working. Annoying.

Thought I had everything set-up correctly, but I guess not. On the old build (prior to rebuild) I did a complete export of all private and public keys, and trusts. On the new system I imported those private & public keys, and the trusts file. I also copied over my ssh configs, etc.

After a TON of Googling, I tried all the remedies I could find, including verifying ownership and permissions on the cert file itself. To my knowledge, this is all correct.

-r--------  1 REDACTED_USER REDACTED_USER  1537 Jan 20  2020 id_rsa-cert.pub

If I do a "ssh-add -l" I do see the proper signature there.

ssh-add -l
2048 SHA256:<<REDACTED>> Public key for Digital Signature (RSA)
2048 SHA256:<<REDACTED>> Public key for PIV Attestation (RSA)

While I redacted it here, I did verify that the sha256 value for the key does match with the servers in question.

So obviously, the problem is a user-induced config issue on my laptop.

Anyone have any thoughts on what the issue could be?

First, I have to admit that I'm a total n00b to NGINX. I have only done very basic work with it.

Now the situation. We have a reverse-proxy box in the DMZ that takes incoming connections/requests and "sends them along" to their destinations. Currently, one of these is allowing only connections from a specific subnet to be forwarded to a specific web server. This is to allow VPN user to reset a specific application password. Unfortunately, it forwards any requests for that web server's URL along.

What I want: I need to lock this down. Instead of passing "https://webserver.com/whatever_they_type" to the server, I want to block everything except for a single, specific URL. ex: "https://webserver.com/this-url/only" Everything else would get blocked.

Anyone have any thoughts on how to modify that location in NGINX to accomplish this?

Wondering if something like this would work?

    location = /good_page/reset_password.html {
            proxy_pass https://1.2.3.4:443;
            #### Set headers ####
            proxy_set_header Host webserver.com;
            proxy_set_header X-Real-IP $remote_addr;
            proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
            #### Only allow OpenVPN networks ###
            allow 5.6.7.8/24;
            deny all;
    }


    location ^~ /good_page {
            deny all;
    }