lsh's questions

I'm stumped. I have two log files being watched by the AWS CloudWatch agent. The first one, /var/log/nginx/access.log, works perfectly fine. The second, /var/log/otherserver/access.log, is not having any changes picked up. Not eventually, not ever UNLESS I restart the agent, whereupon it picks up the changes and sends them to CloudWatch as expected.

/var/log/otherserver/access.log is a log file rsync'ed periodically from another server that can't have the agent installed on it. Command looks like:

rsync -av user@host:/var/log/access.log /var/log/otherserver/access.log

The agent definitely can read it, because it reads the changes in after being restarted.

The position of the entry within the config file doesn't appear to matter.

The dates in the log entries for the rsync'ed log file are the same as the server (everything is UTC).

If I move the log file the agent starts complaining with:

2015-12-14 16:02:26,158 - cwlogs.push.stream - WARNING - 3344 - Thread-1 - No file is found with given path '/var/log/otherserver/access.log'.

The configuration of this second log file is almost identical to the first (below).

[website.access.log]
#datetime_format = 09/Dec/2015:14:15:02 +0000
datetime_format = %d/%b/%Y:%H:%M:%S %z
file = /var/log/otherserver/access.log
log_stream_name = master-platform.sh
initial_position = start_of_file
log_group_name = web-access

[app.access.log]
#datetime_format = 09/Dec/2015:14:15:02 +0000
datetime_format = %d/%b/%Y:%H:%M:%S %z
file = /var/log/nginx/lax.access.log
log_stream_name = {hostname}
initial_position = start_of_file
log_group_name = web-access

Does anyone have any clue what is going on here? Or can suggest an alternative agent that is less ... screwy?

I'm attempting to differentiate between a Ubuntu box and an Arch box within my top.sls file within a Virtualbox 5.0.4 virtual machine managed by vagrant 1.7.4 using salt 2015.8.1 (Beryllium)

This doesn't work:

base:
    'os:Arch':
        - base.arch

This does work:

base:
    {% if grains['os'] == 'Arch' %}
    '*':
        - base.arch
    {% endif %}

Excerpt from salt-call grains.items:

os:
    Arch
os_family:
    Arch
osarch:
    x86_64
oscodename:
osfullname:
    Arch Linux
osrelease:

Does anyone have any insight?

I'm creating a dashboard for our developers that helps visualise and interact with our CloudFormation stacks.

Because a CloudFormation stack can have many resources the number of calls I have to make using boto is getting quite large so the dashboard is quite slow.

I've introduced caching of boto responses but is there a better way of detecting changes than just polling the width and breadth of our infrastructure? Does AWS have some sort of change notification feed my app could subscribe to?

I'm setting up central logging for our servers using syslog-ng + patterndb, however the logs the logging server is receiving from the client are prepended with the date, host and other data. This of course breaks all the patterns in patterndb so nothing matches.

Is there a way to do some preprocessing on the source log file before attempting to classify it or some other way to overcome this problem?

Cheers.

Relevant client conf:

source s_src {
   system();
   internal();
};

destination d_central_logging {
    syslog(192.168.1.1 transport("tcp") port("12345")); 
};

log {
    source(s_src);
    destination(d_central_logging);
};

Relevant server conf:

parser p_patterndb {
    db-parser(file("/var/lib/syslog-ng/patterndb.xml"));
};

source s_network {
    tcp(port(12345) flags(syslog-protocol));
};

filter f_class_unknown {
    match("unknown"
        value(".classifier.class")
        type("string")
    );
};

destination d_all {
    file("/tmp/all");
};

destination d_unknown {
    file("/tmp/unknown");
};

log {
    source(s_network);
    parser(p_patterndb);
    log {
        filter(f_class_unknown);
        destination(d_unknown);
    };
    log {
        destination(d_all);
    };
};

EDIT:

original log line: 10.0.2.2 - - [23/Dec/2014:13:42:49 +0000] "GET /assets/favicon.ico HTTP/1.1" 304 0 "-" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/39.0.2171.71 Safari/537.36"

modified log line: Dec 23 13:59:08 192.168.33.44 264 <13>1 2014-12-23T13:42:50+00:00 devhost 10.0.2.2 - - [meta sequenceId="8"] - - [23/Dec/2014:13:42:49 +0000] "GET /assets/favicon.ico HTTP/1.1" 304 0 "-" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/39.0.2171.71 Safari/537.36"

I need to run a query on a postgresql database in a module I'm writing for Salt. It won't do much, just query a table for a value to determine what version of the software is installed.

The docs have a psql_query function but you can't specify the database, so I'm guessing the query is run against the maintenance db.

To confound things, the postgresql module docs say I have to specify connection params in the minion config, which means having to manually create the postgresql user or duplicating configuration unless the minion config can read from pillar data (pretty sure it can't).

Now, the mysql module for salt does have the exact interface I need in the query function.

The inability to query a psql db seems like such an egregious oversight that its probably me misunderstanding something. Ordinarily I would dive into the code and figure out what is going, but there is so much magic I don't know where to begin. Any help?

I need to more finely tune the Apache vhost file being generated by the example42/kibana Puppet module. Is it possible in Puppet to 'reach through' the example42/kibana plugin and configure the puppetlabs/apache module directly?

For example, I need to modify the apache::vhost.vhost_name setting to "*" somehow (https://forge.puppetlabs.com/puppetlabs/apache#defined-type-apachevhost). Below is my current configuration for Kibana.

class { 'kibana':
    install_url => 'https://download.elasticsearch.org/kibana/kibana/kibana-3.1.0.zip',
    elasticsearch_url => "http://elastic.${::domain}:9200",

    webserver   => 'apache',
    virtualhost => "logs.${::domain}",
}