olivierg's questions

we are sending emails using sendmail on the command line, with a command such as :

cat <email_contents> | sendmail -t

the <email_contents> contains fields such as sender, recipient etc.:

To: [email protected]
Subject: Testing
From: [email protected]

as well as a huge file, attachments etc.

we now need to use Authenticated SMTP (pass a login & password argument to authenticate against the SMTP server).

is there a way to do that with this method ? (using sendmail arguments or modifying the email_contents taking in consideration the following elements:

1. NOT using mailx or any other mail client. we need to keep using sendmail from the commandline
2. NOT amending the sendmail configuration files (sendmail.cf)

thanks again regards,

we have a syslog-ng server acting as an intermediary logger.

the server receives data from all devices on the network (from many different sources, on both UDP & TCP ports), filters them a bit, and forwards the data to a SIEM (splunk).

when looking at ifconfig stats, i can see this :

[root@xxxxxxxx ~]# ifconfig eth0
eth0      Link encap:Ethernet  HWaddr 00:50:56:00:00:00
          inet addr:1.1.1.1    Bcast:1.1.1.1    Mask:255.255.255.0
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:60451021996 errors:0 dropped:0 overruns:0 frame:0
          TX packets:158501574 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:5000
          RX bytes:12508573129969 (11.3 TiB)  TX bytes:220148785267 (205.0 GiB)

this part is quite shocking to me :

          RX bytes:12508573129969 (11.3 TiB)  TX bytes:220148785267 (205.0 GiB)

obviously, the server received around 11 Terrabytes of data since the last boot (15 days uptime) but "only" transferred a little part of it ? (250 Gbs).

is there a way for me to check how is this possible ? i know there's a bunch of UDP dropped messages, but is it taken in consideration ? (dropped is displayed 0).

i expect a little less TX (due to the fact that we filter some messages), but not that much (10% max).

For information, Netstat on UDP shows :

Udp:
 10903564401 packets received
 8401685 packets to unknown port received.
 49356622070 packet receive errors
 43665773 packets sent
 RcvbufErrors: 34287641

any tips on how to investigate this ? thanks !

we've just changed our logging application on our AS/400 partitions and are now using a product called PowerTech SIEM agent for IBM I

The tool used to format messages using the old syslog convention and is apparently now capable of sending IETF messages (RFC 5424), however for some reason our Syslog-NG server is not able to process them, as if the format was not correct.

here is an example error we receive :

Feb  8 15:53:49 hostname syslog-ng[1489874]: Error processing log message: <38>1 2021-02-08T15:53:34.918160+1:00 HOSTNAME.fqn.here - - - - CEF:0!Powertech!SIEM Agent!4.2!AUDIT!TCD0003!6!src=0.0.0.0 dst=1.2.3.4 reason=Command executed msg=A command (WRKPTSA) was run by 12345/T555500/PC01AFB0

the event itself is :

<38>1 2021-02-08T15:53:34.918160+1:00 HOSTNAME.fqn.here - - - - CEF:0!Powertech!SIEM Agent!4.2!AUDIT!TCD0003!6!src=0.0.0.0 dst=1.2.3.4 reason=Command executed msg=A command (WRKPTSA) was run by 12345/T555500/PC01AFB0

Looking at the RFC, it looks like the contents are OK. they should respect :

<priority>VERSION ISOTIMESTAMP HOSTNAME APPLICATION PID MESSAGEID STRUCTURED-DATA MSG

in my case, i have :

<priority> is "38"
VERSION is "1"
ISOTIMESTAMP is "2021-02-08T15:53:34.918160+1:00"
HOSTNAME is "HOSTNAME.fqn.here"
APPLICATION is unknown (-)
PID  is unknown (-)
MESSAGEID  is unknown (-)
STRUCTURED-DATA is unknown (-)
MSG is "CEF:0!Powertech!SIEM Agent!4.2!AUDIT!TCD0003!6!src=0.0.0.0 dst=1.2.3.4 reason=Command executed msg=A command (WRKPTSA) was run by 12345/T555500/PC01AFB0"

we run syslog-NG 3.8

note that i did replace the hostname & IP on purpose, but kept the case.

i have tried multiple things in the syslog-ng configuration (in the source mainly), such as switching from the network() to the syslog() driver (also specified flags(syslog-protocol)), etc. but nothing changed), few examples of what i tried in the source :

source s_ISERIES        { syslog(transport("tcp") ip(0.0.0.0) port(51513)); };
source s_ISERIES        { network(transport("tcp") ip(0.0.0.0) port(51513) flags(syslog-protocol)); };

if i do not use syslog() or the syslog-protocol and use the basic network() driver without flags, the message gets "rewritten" (syslog-NG adds its own header and adds a timestamp and hostname at the beginning of the message, which i do not want).

is there a way to fix that ? do you spot any mistake in the event format ? and if not, is there a way to tell syslog-NG to process the "raw" message (take it and send it where i want without doing further modification of it).

thanks !

edit

after restarting and doing multiple checks, now we just don't have the error anymore. but logs aren't processed at all.

the whole "logging" configuration is quite simple, we've got :

the source defined like this :

source s_ISERIES        { syslog(transport("tcp") ip(0.0.0.0) port(51513)); }; 

the logging config defined like this :

log {
  source(s_ISERIES);
  destination(d_LOCALCOPY_ISERIES);
  destination(d_SPLUNK_ISERIES);
};

destinations defined like this :

destination d_LOCALCOPY_ISERIES   { file("/app/syslog-ng/logs/${YEAR}/${MONTH}/${DAY}/iseries/${HOST}.log" create-dirs(yes)); };

destination d_SPLUNK_ISERIES    { syslog("splunk-host.here.net" port(51513) send-time-zone("UTC")); }; 

and i can't see any message leaving the syslog-NG host when tcpdump'ing (i see them incoming, but not leaving the host). also, no local copy is created. so i "believe" syslog-NG is receiving the message but is unable to parse it for some reason. i tried the flags(no-parse) but it won't work either, as if it was not able to recognize the message at all on the "source" defined.

i'm trying to configure my HAProxy to log more information than just saying "proxy backend_xx started" and it looks like i am failing to understand how it works.

my HAProxy is a pure TCP LB (just forwards requests from the frontend to backends, pure L4).

i would like to have logs as mentionned in the HAProxy documentation here https://cbonte.github.io/haproxy-dconv/1.8/configuration.html#8.2.2

here is my configuration & issues at the moment, in the "global/default sections", i have :

#---------------------------------------------------------------------
# Global settings
#---------------------------------------------------------------------
global
    daemon
    user                haproxy
    group               haproxy
    log                 /dev/log local6 debug
    maxconn             50000
    chroot              /var/lib/haproxy
    pidfile             /var/run/haproxy.pid

#---------------------------------------------------------------------
# common defaults that all the 'listen' and 'backend' sections will
# use if not designated in their block
#---------------------------------------------------------------------
defaults
    mode                 tcp
    option               tcplog
    log                  global
    option               dontlognull
    timeout connect      5000
    timeout client       50000
    timeout server       50000

i also have this option under my frontend :

frontend main_https_listen
    bind xxxxxxxxxxxxxx:443
    mode                tcp
    option              tcplog
    xxxxxxx

i configured a file in my rsyslog.d with :

[root@xxxxxxx ~]# cat /etc/rsyslog.d/haproxy.conf
# -----------------------------------------------
# Haproxy specific logging configuration
# -----------------------------------------------
local6.debug             /var/log/haproxy-traffic.log
local6.notice            /var/log/haproxy-admin.log
[root@xxxxxxx ~]#

unfortunately, both files contain the same information, e.g:

[root@xxxxxxx ~]# tail -11 /var/log/haproxy-admin.log
Feb 25 18:05:30 xxxxxxx haproxy[838761]: Proxy stats started.
Feb 25 18:05:30 xxxxxxx haproxy[838761]: Proxy main_https_listen started.
Feb 25 18:05:30 xxxxxxx haproxy[838761]: Proxy backend_SIT_AT35073 started.
Feb 25 18:05:30 xxxxxxx haproxy[838761]: Proxy backend_SIT_AT34305 started.
Feb 25 18:05:30 xxxxxxx haproxy[838761]: Proxy backend_SIT_AT28548 started.
Feb 25 18:05:30 xxxxxxx haproxy[838761]: Proxy backend_SIT_AT28756 started.
Feb 25 18:05:30 xxxxxxx haproxy[838761]: Proxy backend_SIT_AT36702 started.
Feb 25 18:05:30 xxxxxxx haproxy[838761]: Proxy backend_UAT_AT28546 started.
Feb 25 18:05:30 xxxxxxx haproxy[838761]: Proxy backend_PRD_AT28547 started.

[root@xxxxxxx ~]# tail -11 /var/log/haproxy-traffic.log
Feb 25 18:05:30 xxxxxxx haproxy[838761]: Proxy stats started.
Feb 25 18:05:30 xxxxxxx haproxy[838761]: Proxy main_https_listen started.
Feb 25 18:05:30 xxxxxxx haproxy[838761]: Proxy backend_SIT_AT35073 started.
Feb 25 18:05:30 xxxxxxx haproxy[838761]: Proxy backend_SIT_AT34305 started.
Feb 25 18:05:30 xxxxxxx haproxy[838761]: Proxy backend_SIT_AT28548 started.
Feb 25 18:05:30 xxxxxxx haproxy[838761]: Proxy backend_SIT_AT28756 started.
Feb 25 18:05:30 xxxxxxx haproxy[838761]: Proxy backend_SIT_AT36702 started.
Feb 25 18:05:30 xxxxxxx haproxy[838761]: Proxy backend_UAT_AT28546 started.
Feb 25 18:05:30 xxxxxxx haproxy[838761]: Proxy backend_PRD_AT28547 started.

while i would expect having the traffic log to generate things like in the documentation, e.g:

Feb  6 12:12:56 localhost \
  haproxy[14387]: 10.0.1.2:33313 [06/Feb/2009:12:12:51.443] fnt \
  bck/srv1 0/0/5007 212 -- 0/0/0/0/3 0/0

any idea on what is wrong in my configuration ?

thanks again

regards

i have zones in my BIND server that are updated dynamically by some windows DHCP servers, quite frequently. however, i need to add records "manually" in these zones. can i still manually update these zones by simply editing them (using vi on my bind server like i do for the others not supporting updates), adding the record, updating the serial and reloading the thing ? (rndc reload) what happens if, during my serial update, an update is done at the same time by any dynamic system (DHCP in my case). thanks

I've set up a zone as a master with allow-updates from a windows DHCP Server (authorized on IP only) when I do a release/renew on a Windows workstation, I can see the PTR record updated/created successfully in a bind in the zone.

if I do a :

nslookup IP MYDNSSRVIP

I can see the PTR updated/created successfully.

however, when looking at the zone file (e.g., 1.1.1.in-addr.arpa), the zone file is not updated immediately (if I open it, I can't see the record yet)

does the "datafile" itself take some time to be refreshed? (as I can see the record successfully created when doing a lookup) how is that possible?

or does it mean that something is wrong somewhere in my config and the lookup comes from somewhere else (e.g., a delegation, etc.)

i have a setup with :

HAProxy acting as a layer 4 LB (just forwarding requests, passthrough, TCP mode for all frontend/backend) to multiple Apache frontends. Each of these apache frontends is sending the traffic to a tomcat backend where an application is running.

Now, if one of these tomcats/applications is crashed, the corresponding apache frontend reports an error 503 (which is normal), others are OK.

problem is : as HAProxy is just load balancing to each Apache randomly (based on source IP in my case), some users get the 503 errors (pointing to a tomcat not working), while others are working (pointing to a tomcat that is OK)

is there a way to detect that in any way ? (and ensure users are only redirected to a working frontend/backend)

i suppose i can't do anything on the HAproxy side as HAproxy is forwarding to the "Apache" backends (and reports them as healthy as port 443 is OK) and does not see the real backends (the tomcat servers). i suppose i have to do something on the apache frontends (such as "if there is a 503, stop apache" or something).

any idea on how to handle this ?

thanks !

ps : let me know if you want to see the config

i am having some trouble setting up HAProxy as a TCP load balancer (layer 4) and i would like to have your advice about it.

i've been following many guides on the web and i came up with this configuration (not showing any errors in the logs, it starts well) :

note : real domain names are masked

#---------------------------------------------------------------------
# Global settings
#---------------------------------------------------------------------
global
 daemon
 user                haproxy
 group               haproxy
 log                 /dev/log local6 debug
 maxconn             50000
 chroot              /var/lib/haproxy
 pidfile             /var/run/haproxy.pid

#---------------------------------------------------------------------
# common defaults 
#---------------------------------------------------------------------
defaults
 mode                 tcp
 log                  global
 option               dontlognull
 timeout connect      5000
 timeout client       50000
 timeout server       50000

#---------------------------------------------------------------------
# dedicated stats page
#---------------------------------------------------------------------
listen stats
 mode http
 bind :22222
 stats enable
 stats uri            /haproxy?stats
 stats realm          Haproxy\ Statistics
 stats auth           xxxxxx:xxxxxxxx
 stats refresh        30s

#---------------------------------------------------------------------
# main frontend which proxys to the backends
#---------------------------------------------------------------------
frontend main_https_listen
 bind *:443
 mode                tcp
 option              tcplog

# -------------------------------
# ACLs - SIT
# -------------------------------

acl acl_SIT_CI5      req.ssl_sni -i url1.domain.net
acl acl_SIT_HR8      req.ssl_sni -i url2.domain.net

# -------------------------------
# Conditions - SIT
# -------------------------------

use_backend backend_SIT_CI5 if acl_SIT_CI5
use_backend backend_SIT_HR8 if acl_SIT_HR8

#---------------------------------------------------------------------
# Backends
#---------------------------------------------------------------------

backend backend_SIT_CI5
 mode tcp
 balance source
 option ssl-hello-chk
 server server_SIT_CI5_1 host1.domain.net:443 check
 server server_SIT_CI5_2 host2.domain.net:443 check

backend backend_SIT_HR8
 mode tcp
 balance source
 option ssl-hello-chk
 server server_SIT_HR8_1 host1.domain.net:443 check
 server server_SIT_HR8_2 host2.domain.net:443 check

i've pointed host1.domain.net to my haproxy vIP (it has a keepalived configuration behind with a Virtual IP).

now when accessing https://url1.domain.net (or even https://loadbalancerURL but i assume this is normal on that one) i have an error This page can't be displayed. Turn on TLS 1.0, TLS 1.1, and TLS 1.2 in Advanced settings and try connecting to https://host1.domain.net again.

a single openssl s_client gives a ssl handshake failure (no certificates blabla).

do you have any idea of what i did wrong ? also, do i need to setup some certificates as i'm listening on 443 ? (even if i don't want these certs to be decrypted or whatever as i only want my HAProxy to act as a proxy).

i also tried to activate debug mode for logging but it didnt show any errors (nor new logs)

note : the backends are located behind firewalls, the communication between backends to HAProxy is not opened on 443 (only FROM Haproxy to the backends), does it need to be directional ? and why ?

note2: in haproxy stats, i can see all backends UP

also, is there a way to know/check if a redirection based on hostname (SNI) is working fine or not ? (i have the impression that the connection stays at the load balancer and is not redirected to the backend, and this is why i have an error)

a haproxy -vv gives :

HA-Proxy version 1.5.18 2016/05/10
Copyright 2000-2016 Willy Tarreau <[email protected]>

Build options :
  TARGET  = linux2628
  CPU     = generic
  CC      = gcc
  CFLAGS  = -O2 -g -fno-strict-aliasing -DTCP_USER_TIMEOUT=18
  OPTIONS = USE_LINUX_TPROXY=1 USE_GETADDRINFO=1 USE_ZLIB=1 USE_REGPARM=1     USE_OPENSSL=1 USE_PCRE=1

Default settings :
  maxconn = 2000, bufsize = 16384, maxrewrite = 8192, maxpollevents = 200

Encrypted password support via crypt(3): yes
Built with zlib version : 1.2.7
Compression algorithms supported : identity, deflate, gzip
Built with OpenSSL version : OpenSSL 1.0.2k-fips  26 Jan 2017
Running on OpenSSL version : OpenSSL 1.0.2k-fips  26 Jan 2017
OpenSSL library supports TLS extensions : yes
OpenSSL library supports SNI : yes
OpenSSL library supports prefer-server-ciphers : yes
Built with PCRE version : 8.32 2012-11-30
PCRE library supports JIT : no (USE_PCRE_JIT not set)
Built with transparent proxy support using: IP_TRANSPARENT IPV6_TRANSPARENT     IP_FREEBIND

Available polling systems :
      epoll : pref=300,  test result OK
       poll : pref=200,  test result OK
     select : pref=150,  test result OK
Total: 3 (3 usable), will use epoll.

I am having some trouble trying to configure log rotation for some syslog-ng local log files I'm trying to keep.

for some reason, the size is not respected and logs are filling my filesystem quickly.

I want to keep something like 3 files of 100Mbs or 3 files of 500Mbs for log files.

my current log rotation configuration is the following :

[root@xm1p1034vmo logs]# cat /etc/logrotate.d/syslog-ng
#
/app/syslog-ng/logs/syslog-ng.log {
missingok
notifempty
rotate 3
compress
maxsize 50M
daily
create 0600 root root
postrotate
    /bin/kill -HUP `cat /app/syslog-ng/var/syslog-ng.pid 2> /dev/null` 2> /dev/null || true
endscript
}
#
/app/syslog-ng/logs/outgoing*log {
missingok
notifempty
rotate 3
compress
daily
maxsize 100M
create 0600 root root
postrotate
    /bin/kill -HUP `cat /app/syslog-ng/var/syslog-ng.pid 2> /dev/null` 2> /dev/null || true
endscript
}
#
/app/syslog-ng/logs/incoming*log {
missingok
notifempty
rotate 3
compress
daily
maxsize 500M
create 0600 root root
postrotate
    /bin/kill -HUP `cat /app/syslog-ng/var/syslog-ng.pid 2> /dev/null` 2> /dev/null || true
endscript
}

the crontab is every 3 hours (could put it more often if required) :

# Logrotate
00 03,06,09,12,15,18,21,23 * * * /usr/sbin/logrotate /etc/logrotate.conf >/dev/null 2>&1

but as you can see, the log files are much bigger and grow quickly :

[root@xm1p1034vmo logs]# ls -al
total 8065908
drwxr-xr-x  2 root root       4096 May  2 09:07 .
drwxr-xr-x 11 root root       4096 Dec 20 16:08 ..
-rw-------  1 root root    5330231 May  2 17:12 cid.log
-rw-------  1 root root 4284556769 May  2 17:12 incoming_all.log
-rw-------  1 root root   18081021 May  2 09:07 incoming_all.log-20170502.gz
-rw-------  1 root root   22753799 May  2 17:12 outgoing_esx.log
-rw-------  1 root root      93372 May  2 09:07 outgoing_esx.log-20170502.gz
-rw-------  1 root root    1718167 May  2 17:12 outgoing_fireeye.log
-rw-------  1 root root       8633 May  2 09:07 outgoing_fireeye.log-20170502.gz
-rw-------  1 root root     362817 May  2 17:09 outgoing_iseries.log
-rw-------  1 root root       2706 May  2 09:07 outgoing_iseries.log-20170502.gz
-rw-------  1 root root      15838 May  2 17:10 outgoing_mssql.log
-rw-------  1 root root        345 May  2 09:07 outgoing_mssql.log-20170502.gz
-rw-------  1 root root      57358 May  2 17:11 outgoing_nas.log
-rw-------  1 root root        722 May  2 09:07 outgoing_nas.log-20170502.gz
-rw-------  1 root root      13932 May  2 16:42 outgoing_network.log
-rw-------  1 root root        197 May  2 09:07 outgoing_network.log-20170502.gz
-rw-------  1 root root   11991510 May  2 17:12 outgoing_oracle.log
-rw-------  1 root root      46817 May  2 09:07 outgoing_oracle.log-20170502.gz
-rw-------  1 root root        664 May  2 12:15 outgoing_printers.log
-rw-------  1 root root        174 May  2 09:07 outgoing_printers.log-20170502.gz
-rw-------  1 root root     407191 May  2 17:12 outgoing_san.log
-rw-------  1 root root       1968 May  2 09:07 outgoing_san.log-20170502.gz
-rw-------  1 root root 3896682829 May  2 17:12 outgoing_unix.log
-rw-------  1 root root   16055682 May  2 09:07 outgoing_unix.log-20170502.gz
-rw-------  1 root root    1179031 May  2 17:12 outgoing_win.log
-rw-------  1 root root       2845 May  2 09:07 outgoing_win.log-20170502.gz
-rw-------  1 root root       4003 May  2 16:47 syslog-ng.log
-rw-------  1 root root        386 May  2 09:07 syslog-ng.log-20170502.gz
[root@xm1p1034vmo logs]#

what did I do wrong in the logrotate/crontab config please ? I want to keep 3 logs of 100Mb for every outgoing* log file, and 3 logs of 500Mb for every outgoing* log file

thanks regards,

I just reclaimed unused space in our HP 3Par using a command on the ESX side such as :

esxcli storage vmfs unmap --l DATASTORENAME

it works nicely, and from the 3Par console, I can now see on my datastore/LUN :

• Used User Space : 26% (it was 46% before the reclaim)
• Reserved User Space : 4.7Tb/10Tb (did not change unfortunately)

However, in VMware, the space didn't change. I suppose this is due to this "reservation" (the space is kept reserved on the 3Par).

is there a way to remove this allocation on the SAN (I suppose the data is Zero'd), by running some kind of tuning command or compact command ?

I saw they were command such as compactcpg, but it doesn't seem to be exactly what I need, as I suppose that it will not balance the data (chunklets) correctly across the disks ? (but used fewer disks as possible and therefore decreasing the performance ?)

thanks, regards,

I am contacting you regarding an issue I have with syslog-NG. some of our devices (mainly HP switches and SANs) are sending syslog messages that do not respect the syslog RFC 5424 it seems.

to give you a few examples :

if I sniff the network interface of the server, we can see these "wrong" messages like that (as you can see, after the PRI, we can see the PROGRAM, then timezone and fields separated with commas. in short, mixed fields, missing fields etc. not a standard syslog message) :

<190>raslogd: 2017/03/08-16:03:20, [SEC-1203], 53642, WWN 10:00:50:eb:1a:6c:21:38 | FID 128, INFO, cswc-mo8x-SAN01, Login information: Login successful via TELNET/SSH/RSH. IP Addr: 1.2.3.4

therefore, I am currently unable to filter these messages. I tried to define a regular filter and try to print fields such as MSGHDR, MSG, PRI etc.. but couldn't see anything.

the ONLY WAY of having this message filtered is by just defining the external interface, and a file as output, without any template.

e.g if I do a filter like this :

destination d_INCOMING_ALL   { file("/app/syslog-ng/logs/incoming_all.log"); };

log {
  source(s_EXTERNAL);
  destination(d_INCOMING_ALL);
};

I can see messages in the log file, but formatted, somehow (I suppose syslog-NG reformats them) :

[root@xxxxxxxxxxxx logs]# grep -i cswc incoming_all.log
Mar  9 09:44:20 cswc-mo8x-hpsan01 raslogd: 2017/03/09-08:34:50, [SEC-1203], 53647, WWN 10:00:50:eb:1a:6c:21:38 | FID 128, INFO, cswc-mo8x-SAN01, Login information: Login successful via TELNET/SSH/RSH. IP Addr: 1.2.3.4
[root@xm1p1034vmo logs]#

the problem is that I cannot filter these messages like that (we receive logs from more than 1000 devices) there, i need to filter messages coming from these devices ! and the only way I can do it is on the hostname (cswc-) or program name (raslogd)

so I tried to display the fields by adding a template to that file, example :

destination d_test { 
    file ("/app/syslog-ng/logs/test_olivier.log" 
    template("pri=${priority} date=${ISODATE} host=${HOST} program=${PROGRAM} message=${MSG}\n") 
    ); 
};

but nothing works, nothing is displayed. I tried all fields, MSG, MESSAGE, MSGHDR etc.. can't manage do display ANYTHING. the only working thing is the parsing without filters or templates.

naturally, if I tred all kind of filters, like these below, it does not work (as fields are mixed) :

filter f_is_SAN     {
    host("cswc.*" flags(ignore-case));
};

same for :

filter f_is_SAN     {
    match(".*cswc.*" flags(ignore-case));
};

any hints on how to create filters for these messages coming from these devices (on hostname or programname) ?

thanks regards,