Questions[syslog](server)

How do I switch on PAM debugging in Debian Squeeze at the admin level?

I have checked every resource I was able to find. Google, manpages, whatever. The only thing I haven't tried yet (I simply not dare to, did I mention that I hate PAM?) is digging into the PAM's library source.

I tried to google for a solution, nothing. What I found so far:

http://www.bitbull.ch/wiki/index.php/Pam_debugging_funktion (/etc/pam_debug) and http://nixdoc.net/man-pages/HP-UX/man4/pam.conf.4.html (debug option on PAM entries in /etc/pam.d/).

Nope, does not work. No PAM output, nothing, absolute silence.

While searching for a solution I even followed links to Pam, that are gas stations here in Germany. Well, yes, perhaps in all those billion of hits might hiding a clue, but shoot me I'd be dead before I discover.

Rest is FYI:

What problem did I have?

After upgrading to Debian Squeeze something got weird (well, hey, it once was, uh, what was right over the Etch .. ah, yes, Woody). So it's probably not Debian's fault, just a long lived screwed up setup. I immediately had the impression it has to do something with PAM, but I really did not know what's going on. I was completely in the dark, left alone, helpless as a baby, YKWIM. Some ssh logins worked, some not. It was kind of funny. No clues in ssh -v, no clues in /var/log/*, nothing. Just "auth succeeded" or "auth fail", sometimes the same user logging in parallely succeeded with one session and failed with the other, at the same time. And nothing you really can get hold of.

After digging trainloads of other options I was able to find out. There is nullok and nullok_secure, a Debian special. Something screwed with /etc/securetty and depending on the tty (which is somewhat random) a login was rejected or not. REALLY NICE, phew!

The fix was easy and everything's now fine again.

However this left me with the question, how to debug such a mess in future. It's not the first time PAM drives me nuts. So I would like to see a final solution. Final as in "solved", not final as in "armageddon". Thanks.

Ah, BTW, this again strengthened my belief in that it's good to hate PAM since it came up. Did I mention that I do?

Possible Duplicate:
Alternatives to Splunk?

This has been discussed, but it has been several months, so it may be time to revisit it:

Earlier discussion RE Splunk alternatives

For the record, Splunk rocks. But the pricing is simply beyond what we can consider (When I spoke with Splunk today, the cost for a system to index 5gb/day of data is over $30,000.)

That is more than we spend on SQL Server (by a large multiple), more than we spend on a rack of servers (by a multiple), etc. etc.

The splunk sales team is correct (that for $30K we get more value and functionality than if we spend the same building our own system), but it doesn't matter. The splunk cost is simply too high (by a multiple).

Soooooo, we are looking around!

Is anyone out there building a splunk like system?

Our basic need:

• Able to listen for syslog messages on multiple udp ports
• Able to index the incoming data in an async way
• Some kind of search engine
• Some kind of UI
• An API to the search engine (to embed in our console)

We currently need to index 3-5gb/day, but need to be able to scale to 10gb/day or more. We do not need a lot of history (30 days is fine).

We use Windows 2008 and 2003 servers.

Thanks for your thoughts!

UPDATE: We spent two weeks researching commercial and open source options. Our conclusion: Write our own (we are a software company... we know how to write things). We built a great system built on mongodb and .NET that gives us the functions we needed from MongoDB in about one engineering week. We have now completed our implementation. We use two Mongodb servers (master and slave), and are able to log and index any amount of log data (5gb/day, 15gb/day, etc), limited only by disk space.

UPDATE TO THE UPDATE (December, 2012): We continue to use our mongodb solution, and it works great! If we were building it today, we would strongly consider building it on top of elasticsearch.

OBSERVATIONS: This space needs a solid solution that is $1000-3000 flat rate. The licensing models used by the commercial firms are based on a "milk the data center ops guys" models. That is their right (of course!), but it leaves a HUGE space open for someone to come in underneath them. My guess is that in another year or two there will be a good open source solution that will be really usable.

Thank you all for your input (even if it was self promotion).

I am trying to write some custom messages in my dmesg output. I tried:

logger "Hello"

but this does not work. It exits without error, but no "Hello" appears int the output of:

dmesg

I am using a Fedora 9, and it seems that there is no syslogd/klogd daemon running. However, all my kernel messages are succesfully written in the dmesg buffer.

Any idea?

I am looking for a better way to log cronjobs. Most cronjobs tend to spam email or the console, get ignored, or create yet another logfile.

In this case, I have a Nagios NSCA script which sends data to a central Nagios sever. This send_nsca script also prints a single status line to STDOUT, indicating success or failure.

0 * * * * root    /usr/local/nagios/sbin/nsca_check_disk

This emails the following message to root@localhost, which is then forwarded to my team of sysadmins. Spam.

forwarded nsca_check_disk: 1 data packet(s) sent to host successfully.

I'm looking for a logging method which:

1. Doesn't spam the messages to email or the console
2. Don't create yet another krufty logfile which requires cleanup months or years later.
3. Capture the log information somewhere, so it can be viewed later if desired.
4. Works on most unixes
5. Fits into an existing log infrastructure.
6. Uses common syslog conventions like 'facility' and 'priority'
7. Can work with third party scripts which don't always do logging internally.

I have a script which needs to be executed each minute. The problem is that cron is logging to /var/log/syslog each time it executes. I end up seeing something like this repeated over and over in /var/log/syslog:

Jun 25 00:56:01 myhostname /USR/SBIN/CRON[1144]: (root) CMD (php /path/to/script.php > /dev/null)

I'm using Debian.

My questions is: Is there any way I can tell cron not write this information to syslog every time?