mc0e's questions

I want to access a few gigabytes of old mail that I have in an archived copy of a dovecot mail server's virtual mailbox files. I think it's an old version of dovecot's mdbox format? The server was built in 2008, and the archive taken in 2011. I'm guessing it might be mdbox 1.1.

I'm looking for suggestions of how to get this mail to the point where I can browse it with thunderbird.

It would be awesome if I could fire up an old version of dovecot to serve this mail via IMAP, possibly in a docker container. OR perhaps newer versions of dovecot can recognise the old format? Maybe a conversion tool of some sort?

Any help with identifying the format/version would be much appreciated. My files look like:

domain/
  username/
    dovecot.index
    dovecot.index.cache
    dovecot.index.log
    dovecot.index.log.2
    dovecot-keywords
    dovecot-uidlist
    maildirsize
    subscriptions
    cur/
      1314076009.V55I369181M295175.mail.exmaple.com:2,STa
    new/
    tmp/
    .folder/
      dovecot.index
      dovecot.index.cache
      dovecot.index.log
      dovecot-keywords
      dovecot-uidlist
      cur
      new
      tmp
    .folder.subfolder/
      dovecot.index
      dovecot.index.cache
      dovecot.index.log
      dovecot-uidlist
      cur
      new
      tmp

There's of course more folders, and more mail files, with flags in the file names as in the one that's there for an example.

In my puppet configuration, I've got a defined resource type as follows. Everything was working fine under puppet 2.7, but after an upgrade to puppet 3.8, I'm having trouble with it. When I create a user, the exec runs without the user and group running first, so the exec resource's command fails to execute properly.

Can anyone explain what is going wrong?

define my_user (
  $uid,
  $fullname,
  $email,
  $group = $name,
  $ingroups = [],
  $shell = '/bin/bash',
  $home = "/home/${name}",
  $ensure = 'present',
  $sudo = 'no',
  $system_user = false,
  $ssh_key = undef,
  $ssh_key_type = undef,
) {

  group{ $name:
    ensure => present,
    gid    => $uid,
  }

  user { $name:
    ensure     => $ensure,
    comment    => $fullname,
    uid        => $uid,
    gid        => $group,
    groups     => $ingroups,
    shell      => $shell,
    home       => $home,
    managehome => true,
    require    => Group[$group],
    system     => $system_user,
  }

  exec { "${name} homedir":
    command => "/bin/cp -R /etc/skel ${home}; /bin/chown -R ${name}:${group} ${home}",
    creates => $home,
    require => User[$name],
  }

  file { "${home}/.ssh":
    ensure  => directory,
    require => [ User[$name], Exec["${name} homedir"] ],
    owner   => $name,
    group   => $name,
    mode    => '0755',
  }

  if $ssh_key {
    ssh_authorized_key{ $email:
      ensure => $ensure,
      user   => $name,
      key    => $ssh_key,
      type   => $ssh_key_type,
    }
  }
}

[Any suggestions on a better title are welcome]

For better or worse, I currently have a data structure like so (yaml)

servers:
  server1.example.com:
    private_ip: 10.0.0.1
    public_ip: 22.33.44.1
  server2.example.com:
    private_ip: 10.0.0.2
    public_ip: 22.33.44.2
global_roles:
  nagios: 
    - server1
    - server2

I can pull that into puppet and do things like:

$nagios_monitor = $global_roles['nagios'][0]
$private_ip["${nagios_monitor}.example.com"]['private_ip']

What I want is a way to get an array of the private IPs for all listed nagios servers. Is there a good way to do this with my existing data structure?

If the data structure needs to be changed, for this to work nicely, suggestions on a good way to present this data to puppet are also welcome. I suspect such a change would make life generally easier, but there's a fair bit of existing code that would need to changed, and I really wouldn't want any mis-steps.

I'm in the process of upgrading an old puppet 2.7 installation to puppet 3.8, and I've also been shuffling the location of some of hiera's data. I'm not sure which is to blame here.

puppet agent --test, run on the same system running the puppetserver is giving me an error:

Error: Could not retrieve catalog from remote server: Error 400 on SERVER: Could not find data item nagios::htpasswd.users in any Hiera data file and no default supplied at /etc/puppet/modules/nagios/manifests/master.pp:77 on node xxx.example.com

This particular hiera key comes from a file via the hiera-file back-end.

The cited puppet config line is in a file resource which specifies:

content => hiera('nagios::htpasswd.users')

However, I can run the following successfully:

puppet apply -e 'notice hiera("nagios::htpasswd.users")'

So, I'm thinking that all the data is in the right place, but that the puppetserver is for some reason not using hiera-file as expected, despite it being configured in /etc/puppet/hiera.yaml

Can anyone suggest what is going on here?

I've got a bunch of client's virtual machines running by various providers who use Xen paravirtualisation. Almost all of these are Debian or Debian based systems.

I rely on APT pretty heavily to track what security updates need doing, but with a paravirtualised system, the kernel that's actually running is typically not the one that APT installs, even after a reboot. Actually updating the kernel generally involves messing about with a control panel or API. Unfortunately the providers tend not to do very well at informing me when I'm using an old kernel.

Is there a tool that I can use within a VPS to monitor what kernel version is actually running (e.g. uname -a), and some online source of security info, and let me know when there's a security issue I need to address?

I'm not sure quite how this should work, given that a huge proportion of the kernel security issues tend to be in drivers, many of which would not be applicable, but maybe someone has done the thinking here to come up with a good strategy?

I have a container that is specified to docker-compose like so:

postfix:
    build: postfix
    ports:
        - "25:25"
    volumes_from:
        - mailspool
    links:
        - dovecot:dovecot
    external_links:
        - mariadb_single_container:mariadb
    env_file: environment

It was working fine, but after a system reboot, the environment variables for linking to the mariadb container are not being presented to the postfix container.

I've tried stopping and starting the postfix container in case it was something to do with the order things came up in. I'm wary of taking down mariadb, which other containers are also depending on, but mariadb looks normal, and other containers are using it successfully (those containers don't use docker-compose).

Any ideas?

I've got a server running dovecot with the dovecot-antispam plugin. These are sourced from the Standard Ubuntu 14.04 repositories.

I know mail is not supposed to be moved into a mailbox of type 'unsure'. The server should refuse this action. What actually happens though is that the imap process throws an error and dies:

Aug 12 17:53:07 255c7fc2d2f0 imap: antispam: mailbox_is_unsure(Review-Classification): 1
Aug 12 17:53:07 255c7fc2d2f0 dovecot: imap([email protected]): Panic: file mail-storage.c: line 2141 (mailbox_copy): assertion failed: (!ctx->unfinished)
Aug 12 17:53:07 255c7fc2d2f0 dovecot: imap([email protected]): Error: Raw backtrace: /usr/lib/dovecot/libdovecot.so.0(+0x5e271) [0x7f718e7dd271] -> /usr/lib/dovecot/libdovecot.so.0(+0x5e34e) [0x7f718e7dd34e] -> /usr/lib/dovecot/libdovecot.so.0(i_fatal+0) [0x7f718e798a9e] -> /usr/lib/dovecot/libdovecot-storage.so.0(+0x7196f) [0x7f718eab896f] -> /usr/lib/dovecot/libdovecot-storage.so.0(mailbox_move+0x15) [0x7f718eab8985] -> dovecot/imap(+0xded5) [0x7f718ef84ed5] -> dovecot/imap(command_exec+0x3c) [0x7f718ef8f6dc] -> dovecot/imap(+0x176cf) [0x7f718ef8e6cf] -> dovecot/imap(+0x17785) [0x7f718ef8e785] -> dovecot/imap(client_handle_input+0x125) [0x7f718ef8eaa5] -> dovecot/imap(client_input+0x75) [0x7f718ef8ee35] -> /usr/lib/dovecot/libdovecot.so.0(io_loop_call_io+0x27) [0x7f718e7ed247] -> /usr/lib/dovecot/libdovecot.so.0(io_loop_handler_run+0xd7) [0x7f718e7edfd7] -> /usr/lib/dovecot/libdovecot.so.0(io_loop_run+0x38) [0x7f718e7ecde8] -> /usr/lib/dovecot/libdovecot.so.0(master_service_run+0x13) [0x7f718e79dc93] -> dovecot/imap(main+0x2c4) [0x7f718ef832f4] -> /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf5) [0x7f718e3dbec5] -> dovecot/imap(+0xc460) [0x7f718ef83460]
Aug 12 17:53:07 255c7fc2d2f0 dovecot: imap([email protected]): Fatal: master: service(imap): child 58 killed with signal 6 (core dumped)

So, the primary issue is that I want to figure out how to get my server to handle this more gracefully, and refuse the move properly without just having the process die. I'm not sure how to approach this. I can provide configuration as required, but presumably this isn't just a configuration issue. I've installed the dovecot-dbg package, but that doesn't seem to improve the logged stacktrace much.

A secondary issue is that Thunderbird's handling of this is pretty nasty also. The message disappears from display and the mail folder it's being moved from, and doesn't appear in the target folder. Thunderbird continually tries and fails to move the message on the server. The only way I know to break this pattern is to temporarily remove the target folder from the antispam_unsure directive in dovecot, which is cumbersome for working out the issue. Any suggestions here are welcome, but this is a secondary concern.

Edit:

After some help from IRC and http://www.dovecot.org/bugreport.html I figured out how to get a stacktrace (Have docker set ulimit -c unlimited before starting dovecot, and set /proc/sys/kernel/core_pattern from outside the container ... would be nice if that was container specific). So, I now have a core file and a stacktrace:

#0  0x00007f5dd5394cc9 in __GI_raise (sig=sig@entry=6) at ../nptl/sysdeps/unix/sysv/linux/raise.c:56
        resultvar = 0
        pid = 61
        selftid = 61
#1  0x00007f5dd53980d8 in __GI_abort () at abort.c:89
        save_stage = 2
        act = {__sigaction_handler = {sa_handler = 0x800, sa_sigaction = 0x800}, sa_mask = {__val = {140041026109520, 178, 140041026077952, 512, 
              140040990095251, 140041026077952, 140041026077952, 512, 140040990087565, 140734823456136, 140040990231826, 140041026077952, 140734823456040, 0, 
              140040990231929, 140041026077952}}, sa_flags = -713567062, sa_restorer = 0x7fff6128dd01}
        sigs = {__val = {32, 0 <repeats 15 times>}}
#2  0x00007f5dd5781298 in default_fatal_finish (type=<optimized out>, status=status@entry=0) at failures.c:192
        backtrace = 0x7f5dd79d0538 "/usr/lib/dovecot/libdovecot.so.0(+0x5e271) [0x7f5dd5781271] -> /usr/lib/dovecot/libdovecot.so.0(+0x5e34e) [0x7f5dd578134e] -> /usr/lib/dovecot/libdovecot.so.0(i_fatal+0) [0x7f5dd573ca9e] -> /usr/lib/d"...
#3  0x00007f5dd578134e in i_internal_fatal_handler (ctx=0x7fff6128dd70, format=<optimized out>, args=<optimized out>) at failures.c:653
        status = 0
#4  0x00007f5dd573ca9e in i_panic (format=format@entry=0x7f5dd5ab1e68 "file %s: line %d (%s): assertion failed: (%s)") at failures.c:264
        ctx = {type = LOG_TYPE_PANIC, exit_status = 0, timestamp = 0x0}
        args = {{gp_offset = 40, fp_offset = 48, overflow_arg_area = 0x7fff6128de60, reg_save_area = 0x7fff6128dda0}}
#5  0x00007f5dd5a5c96f in mailbox_copy (_ctx=<optimized out>, mail=mail@entry=0x7f5dd79ffbc0) at mail-storage.c:2141
        ctx = 0x7f5dd7a043e0
        t = 0x7f5dd79fdf00
        keywords = 0x0
        pvt_flags = 0
        real_mail = <optimized out>
        ret = -1
        __FUNCTION__ = "mailbox_copy"
#6  0x00007f5dd5a5c985 in mailbox_move (_ctx=<optimized out>, mail=0x7f5dd79ffbc0) at mail-storage.c:2153
        ctx = 0x7f5dd7a043e0
#7  0x00007f5dd5f28ed5 in fetch_and_copy (copy_count_r=<synthetic pointer>, src_uidset_r=<synthetic pointer>, search_args=<optimized out>, 
    src_trans_r=0x7fff6128df18, t=<optimized out>, move=true, client=0x7f5dd79ee440) at cmd-copy.c:67
        search_ctx = 0x7f5dd79ff920
        src_trans = 0x7f5dd79fec10
        srcset_ctx = {str = 0x7f5dd79d0350, first_uid = 0, last_uid = 4294967295}
        ret = 1
        save_ctx = 0x0
        mail = 0x7f5dd79ffbc0
        copy_count = 1
        src_uidset = 0x7f5dd79d0350
#8  cmd_copy_full (cmd=0x7f5dd79eee70, move=<optimized out>) at cmd-copy.c:123
        client = 0x7f5dd79ee440
        dest_storage = <optimized out>
        destbox = 0x7f5dd79f8d10
        t = 0x7f5dd79fdf00
        src_trans = 0x7f5dd573d029 <t_pop_check+9>
        search_args = 0x7f5dd79f73a0
        messageset = 0x7f5dd79dbc10 "76"
        mailbox = 0x7f5dd79dbc18 "Review-Classification"
        src_uidset = <optimized out>
        sync_flags = (unknown: 0)
        imap_flags = (unknown: 0)
        changes = {pool = 0x7f5dd79d0350, uid_validity = 0, saved_uids = {arr = {buffer = 0x1, element_size = 140040986301132}, v = 0x1, v_modifiable = 0x1}, 
          ignored_modseq_changes = 3617438728, changed = 93, no_read_perm = 127}
        msg = <optimized out>
        ret = <optimized out>
        __FUNCTION__ = "cmd_copy_full"
#9  0x00007f5dd5f336dc in command_exec (cmd=0x7f5dd79eee70) at imap-commands.c:158
        hook = 0x7f5dd79d90d0
        ret = <optimized out>
#10 0x00007f5dd5f326cf in client_command_input (cmd=0x7f5dd79eee70) at imap-client.c:780
        client = 0x7f5dd79ee440
        command = <optimized out>
        __FUNCTION__ = "client_command_input"
#11 0x00007f5dd5f32785 in client_command_input (cmd=0x7f5dd79eee70) at imap-client.c:841
        client = 0x7f5dd79ee440
        command = <optimized out>
        __FUNCTION__ = "client_command_input"
#12 0x00007f5dd5f32aa5 in client_handle_next_command (remove_io_r=<synthetic pointer>, client=0x7f5dd79ee440) at imap-client.c:879
No locals.
#13 client_handle_input (client=client@entry=0x7f5dd79ee440) at imap-client.c:891
        _data_stack_cur_id = 3
        remove_io = false
        handled_commands = false
        __FUNCTION__ = "client_handle_input"
#14 0x00007f5dd5f32e35 in client_input (client=0x7f5dd79ee440) at imap-client.c:933
        cmd = 0x7f5dd79e5ae0
        output = 0x7f5dd79eeca0
        bytes = 39
        __FUNCTION__ = "client_input"
#15 0x00007f5dd5791247 in io_loop_call_io (io=0x7f5dd79eed90) at ioloop.c:388
        ioloop = 0x7f5dd79d8720
        t_id = 2
#16 0x00007f5dd5791fd7 in io_loop_handler_run (ioloop=ioloop@entry=0x7f5dd79d8720) at ioloop-epoll.c:220
        ctx = 0x7f5dd79d93b0
        list = 0x7f5dd79dade0
        io = <optimized out>
        tv = {tv_sec = 1799, tv_usec = 999178}
        events_count = <optimized out>
        msecs = <optimized out>
        ret = 1
        i = 0
        call = <optimized out>
        __FUNCTION__ = "io_loop_handler_run"
#17 0x00007f5dd5790de8 in io_loop_run (ioloop=0x7f5dd79d8720) at ioloop.c:412
        __FUNCTION__ = "io_loop_run"
#18 0x00007f5dd5741c93 in master_service_run (service=0x7f5dd79d85b0, callback=<optimized out>) at master-service.c:566
No locals.
#19 0x00007f5dd5f272f4 in main (argc=1, argv=0x7f5dd79d8390) at main.c:400
        set_roots = {0x7f5dd614aa00 <imap_setting_parser_info>, 0x0}
        login_set = {auth_socket_path = 0x7f5dd79d0048 "\001", postlogin_socket_path = 0x0, postlogin_timeout_secs = 60, 
          callback = 0x7f5dd5f3c0f0 <login_client_connected>, failure_callback = 0x7f5dd5f3be20 <login_client_failed>, request_auth_token = 1}
        service_flags = <optimized out>
        storage_service_flags = <optimized out>
        username = 0x0
        c = <optimized out>

• Ubuntu 14.04.3
• dovecot-imapd 1:2.2.9-1ubuntu2
• dovecot-antispam 2.0+20130822-2build1

I need to instantiate a firewall related class which takes an array of the hosts allowed to access a database port on a db server. What's in that list depends on which of various web apps are turned on, on which hosts.

I'd like to put something amongst the logic for the app that says 'make sure the ip of this server is in the list of which ones can access the db', but without being able to re-declare variables, I'm not sure how to accumulate that info.

This is sort of like what concat does, though that would at best get me a file I could @include in the firewall rules. I could rewrite the firewall module so I can do it that way, but it's pretty messy, and I'm hoping to avoid this.

Defining virtual resources in the app logic and then collecting them seems like a nicer way to go, but I'm not sure though how I would get from a collector instantiating a bunch of resources of a given type to having an array of some value from each of those resources?

WebApp::FirewallHole <| dbhost == $fqdn |>

$app_hosts = ???

Firewall::Hole {'mysql':
  proto => 'tcp',
  dport => '3306',
  saddr=> $app_hosts,
}

In this code, WebApp::FirewallHole existst purely for pulling this information in, and can be defined in any way that helps. Firewall::Hole has for current purposes a fixed interface.

The above glosses over some of the details of the firewall logic. I can't just instantiate the Firewall::Hole class repeatedly here, and changing the interface to allow that would just push the problem further down into the firewall logic, so the question is about how I can construct an array of the values from a set of collected resources.

I want postfix to log the 'From' Header, and maybe other headers from the email content (not the Envelope header) in order to assess DMARC domain alignment in outgoing email.

I know about using header_checks with a WARN action, but I have other header_checks rules, and it's hard to avoid conflicts this way.

What other options are there?

What's a good way to test the current date/time in puppet in order to implement a temporary change?

e.g. we've just had a security issue with NTP, and it took a while for Debian to come out with an update. It seemed simplest to just turn ntp off for a while.

service { 'ntp':
  # remember to re-enable this after ntp is secured.  CVE-XXXXXXXXXX
  ensure => "stopped",
  ...
}

So that requires remembering to come back and change it back. What I'd like to have done would be to have made the value of the ensure attribute dependent on a selector which tested the date at run time.

Can anyone suggest a nice way to do that?

I can imagine making a function available through the puppet parser to do it, and it'd be nice to know if someone has written that already, but bonus points if you have a way to do it without an extra module.

service { 'ntp':
  ensure => date_is_after("23 Dec 2014"): {
     true => "running",
     false => "stopped"
  }
  ...
}

It occurs to me also that a test based on the current version of the ntp package would be quite elegant, but I'm also not sure how you'd do that in puppet.

I've read the question and answers at How to setup MySQL replication with minimal downtime. It looks like I can get a slave DB into sync without down time if I am using only Innodb tables.

My primary tables use the Innodb engine, but I also have a set of tables that records the history of the primary tables. A set of triggers intercept updates, and take a copy of the old data before it gets modified, along with a time-stamp. This data is stored in archive tables, which are only ever appended to.

So can I use a similar technique to get replication going without downtime?

I don't mind if the archive tables are temporarily out of sync, and can live with it if they are not a perfect record of changes that happened during the sync process. My first concern is that the availability of the innodb tables should have little or no interruption.

The number of tables involved is quite large, and somewhat dynamic. Solutions that don't involve manually making lists of tables would be much better.

I'm moving from mariadb 5.5. I'm currently planning moving to the same version on a new server, and then treat that update separately, but if upgrading the target version would make a significant difference, please say so.

puppetlabs-postgresql's documentation recommends setting values in postgresql::globals like so:

class { 'postgresql::globals':
  encoding => 'UTF8',
  locale   => 'en_NG',
}->
class { 'postgresql::server':
}

However, this doesn't work:

class { 'postgresql::client: }
class { 'postgresql::globals':
  encoding => 'UTF8',
  locale   => 'en_NG',
}->
class { 'postgresql::server':
}

Because postgresql::client inherits from postgresql::parameters, which inherits from postgresql::globals. So when it gets to the explicit instantiation of the postgresql::globals class, it complains about a duplicate definition.

changing the order, it does work:

class { 'postgresql::globals':
  encoding => 'UTF8',
  locale   => 'en_NG',
}->
class { 'postgresql::server':
}
class { 'postgresql::client: }

However, in real use, my instantiation of postgresql::client and postgresql::server are in different classes associated with server classes I use, and I really don't want to be creating a required ordering for those classes. Also, I want the globals definitions to apply to postgresql::client even on servers which do not run postgresql::server.

If we were using puppet 3.x, I'd define the required globals values in hiera, but we're still on puppet 2.7.

Is there a way to fix this without modifying the puppetlabs distribution? I'm currently thinking this may warrant a bug report to puppetlabs, but still wonder if I'm just missing a simple solution.

EDIT: having read @FelixFrank's input, I've created a bug report at https://tickets.puppetlabs.com/browse/MODULES-1466.

I recently noticed that opendkim on my mail server is objecting to DKIM signatures from a client, saying their key is insecure. It may be that that's due to lack of secure DNS (confirmation?) but I also noticed that the signing algorithm is shown as a=rsa-sha1, and with the move to Deprecate SHA-1 in other contexts, I'm wondering how urgent this is for DKIM?

It is now feasible to brute-force SHA-1 hashes. Am I correct in thinking that this would need to be done for each message an attacker wished to forge the signature for, without being able to re-use that work for the next message?

I'm planning a Mailman server, and looking for a good way to enable list moderators train a spam filter (likely to be either spamassassin or dspam). Has anyone come up with a good way to run training based on list moderator's decisions? Currently I don't have any better strategies than asking list moderators to forward spams one by one to a training address, which seems laborious and most likely to be inconsistently applied.

Any ideas?

I am aware of https://bugs.launchpad.net/mailman/+bug/558292 . I'm hoping someone has a better approach.

Using the Ingo web gui which is part of horde, I can set up sieve rules that get written to file like so:

# Junk-Review
if header :comparator "i;ascii-casemap" :regex "X-DSPAM-confidence" "^(0\.[0-8][0-9]{3})$"  {
    fileinto "Junk-Review";
    stop;
}
# Junk
if header :comparator "i;ascii-casemap" :contains "X-DSPAM-Result" "Spam"  {
    fileinto "Junk";
    stop;
}

I'd like to provide these rules such that they are available to all users, and default to running at the proper point, after things like whitelist and blacklist, but before custom rules.

Is there a way to get ingo to do this? By default it's very much oriented towards the way Spamassassin writes its headers, and it only caters for a 2 way split between Spam and non-Spam, which leaves the user still needing to review all their spam in order to catch the few false positives.

The primary mx server for my domain is down, and is likely to be down for a day or so. Meanwhile pressure is on to get sme high priority mail to people who need it.

What I want to do is set up a secondary MX, which will act as a relay for some specific email addresses, and relay the rest for one domain, while another domain should only use the policy in the virtual alias maps table.

What I've currently got looks like so:

main.cf:

virtual_alias_domains = dom1.example.org, dom2.example.org
virtual_alias_maps = hash:/etc/postfix/virtual
transport_maps = hash:/etc/postfix/transport

transport:

 [email protected]       :
 [email protected]       :
 dom1.example.org             smtp:primarymx.example.org
 dom2.example.org             :

virtual:

 [email protected]  [email protected]
 [email protected]  [email protected]
 [email protected]  [email protected]

The problem is that mail to [email protected] gets rejected:

Remote host said: 550 5.1.1 <[email protected]>: Recipient address rejected: 
User unknown in virtual mailbox table

What I want to happen is that that mail gets directed to smtp:primarymx.example.org and queued until that server is available.

What is going wrong? How can I get this to work?