Adam Ryczkowski's questions

I would like to immediately process certain types of packets if/when they arrive with a custom (bash) script.

As NetFilter is kernel space, I understand I need to look for a userspace logging tools so I set my eyes on ulogd (v.2). Unfortunately I can not find functionality that would let me handle incoming packets.

I could let ulogd output into a file, and set a separate process that monitors that file, but it seems to me too much convoluted and complex.

Is there an option to nft or ulogd that I missed, that would let me fire a script any time a certain packet was encountered?

This question is similar to No tun device in lxc guest for openvpn . The LXC has evolved and the unprivileged LXC containers were introduced recently that offer another layer of security against breaking the jail.

I need to create an OpenVPN server inside one of the unprivileged container. I don't know how to let the container create a private tun network device.

I did append lxc.cgroup.devices.allow = c 10:200 rwm to the ~/.local/share/lxc/mylxc/config.

After starting the container, mknod /dev/net/tun c 10 200 returns mknod: '/dev/net/tun': Operation not permitted inside the container.

I use a vanilla Ubuntu 14.04 64bit as host and a container created with

lxc-create -t download -n mylxc  -- -d ubuntu -r trusty -a amd64

Did anyone managed to get /dev/tun device running under unprivileged LXC?

I am trying to deploy the Redmine server via Puppet. The installation procedure uses Bundle to install its dependencies. When being installed interactively, sudo prompts user for password when Bundle needs to install some system-wide libraries. Installing Redmine as root is not recommended.

I use Ubuntu 14.04 64 bit. The puppet is run from inside the LXC unprivileged container.

I used to have a walkaround that involved touching the appropriate /var/lib/sudo files prior the Bundle's installation. That is hacky, and ceased to work after recent sudo upgrade.

Is there any other way to do it? Or maybe sudo does allow granting grace period non-interactively and I simply missed it?

My sudoers file already contains the Defaults !tty_tickets entry. I would prefer not to have it, but without this line I think there is even less hope for solution.

Here is my forcesudo resource:

define forcesudo ($user = $name, $notify = $notify) {
  file { "/etc/sudoers.d/fix-${user}": content => "${user} ALL = (root) NOPASSWD: ALL\n" }

  file { "/etc/sudoers.d/tty_tickets": content => "Defaults         !tty_tickets\n" }

  file { "/var/lib/sudo/${user}":
    ensure => directory,
    owner  => 'root',
    mode   => 700,
    group  => $user
  }

  file { "/var/lib/sudo/${user}/0":
    content => '',
    owner   => 'root',
    mode    => 0600,
    group   => $user
  }

  touch { "/var/lib/sudo/${user}":
    notify  => $notify,
    require => File["/var/lib/sudo/${user}"]
  }

  touch { "/var/lib/sudo/${user}/0":
    notify  => $notify,
    require => File["/var/lib/sudo/${user}"]
  }

}

And here is the touch resource:

define touch ($user = "root", $group = "root", $notify = $notify) {
  exec { "touch ${name}":
    command     => "/usr/bin/touch ${name}",
    user        => $user,
    group       => $group,
    notify      => $notify,
    refreshonly => true,
  }
}

Edit:

The problem can by bypassed by running the Bundle as root, although this is discouraged by the Bundle: Don't run Bundler as root. Bundler can ask for sudo if it is needed, and installing your bundle as root will break this application for all non-root users on this machine.

This is how it is done by the https://github.com/johanek/johanek-redmine puppet module.

Since this is a workaround rather than fix, the question stands.

Suppose I want hostB to have a distinct file for each puppet node, but with owner that is only known to the hostB.

Theoretically it can be done, if we allow hostB to write a custom fact (my_special_owner), and then read that fact via query_facts function in the my_resource definition by hostA. But that seems too dirty: it pollutes the facts namespace with meaningless stuff, possibly causing name clashes. It is also a lot of coding for such a simple thing.

Here is a mockup of what I have in mind:

define my_resource() {
  $owner=${my_collector::owner} #This attempt fails
  file{$name: ensure=>exists, owner=>$owner}
}

node hostA {
  @@my_resource{"/tmp/file1.tmp"}
}


class my_collector($owner) {
  # How to pass $owner to the collector below?
  My_resource <<| |>> 
}

node hostB {
  class {'my_collector': owner=>bob}
}

Suppose I have written a define, and was able to trap the illegal argument combination. How to fail the manifest with a message that comprises at least a resource that included our definition (or even better: a full calling stack)?

Something similar to the notice{'message': withpath => true }

Background:

I am upgrading our lxc puppet module to support the emerging unprivileged LXC technology.

The code that creates the container is executed unless the following test if the container exists succeeds: lxc-info -n ${lxcname} >/dev/null. If executed as root, the lxc-info checks for the name among the root containers. If invoked by a normal user, it checks against the list of unprivileged containers stored for this particular user.

I want to create a resource along the lines of (code not tested yet):

#Creation of the container ${name}
  exec { "lxc-create of ${hostname}":
    command => "lxc-create -t download -n ${name} -- -d ubuntu -r ${release} -a amd64",
    user    => $user,
    group   => $group,
    timeout => 0,
    unless  => "lxc-info -n ${name}" #Should be executed as user ${user}, not root!
  }

I can fix this problem if I change the last parameter to

unless  => "su -l ${user} -c 'lxc-info -n ${name}'"

but it just don't look elegant. Is there any built-in solution in puppet to make a check as other user?

I've just setup OpenVPN server with the following config:

daemon
server-bridge
push "route 0.0.0.0 255.255.255.255 net_gateway"
proto udp
port 1194
dev tap21
comp-lzo adaptive
keepalive 15 60
verb 3
plugin /usr/lib/openvpn-plugin-auth-pam.so openvpn
client-cert-not-required
username-as-common-name
duplicate-cn
ca ca.crt
dh dh.pem
cert server.crt
key server.key
status-version 2
status status

When I connect to it with client:

client
dev tap
proto udp
remote <remote IP> 1194
float
comp-lzo adaptive
keepalive 15 60
auth-user-pass
ns-cert-type server
resolv-retry infinite
nobind

After connection, when I do ping 192.168.3.12 to the inside network, I've got very erratic latency, like this:

64 bytes from 192.168.3.12: icmp_seq=239 ttl=64 time=730 ms
64 bytes from 192.168.3.12: icmp_seq=240 ttl=64 time=950 ms
64 bytes from 192.168.3.12: icmp_seq=241 ttl=64 time=290 ms
64 bytes from 192.168.3.12: icmp_seq=242 ttl=64 time=236 ms
64 bytes from 192.168.3.12: icmp_seq=243 ttl=64 time=239 ms
64 bytes from 192.168.3.12: icmp_seq=244 ttl=64 time=41.1 ms
64 bytes from 192.168.3.12: icmp_seq=245 ttl=64 time=1585 ms
64 bytes from 192.168.3.12: icmp_seq=246 ttl=64 time=1804 ms
64 bytes from 192.168.3.12: icmp_seq=247 ttl=64 time=1087 ms
64 bytes from 192.168.3.12: icmp_seq=248 ttl=64 time=483 ms
64 bytes from 192.168.3.12: icmp_seq=249 ttl=64 time=864 ms
64 bytes from 192.168.3.12: icmp_seq=250 ttl=64 time=1614 ms
64 bytes from 192.168.3.12: icmp_seq=251 ttl=64 time=1173 ms
64 bytes from 192.168.3.12: icmp_seq=252 ttl=64 time=837 ms
64 bytes from 192.168.3.12: icmp_seq=253 ttl=64 time=834 ms
64 bytes from 192.168.3.12: icmp_seq=254 ttl=64 time=516 ms
64 bytes from 192.168.3.12: icmp_seq=255 ttl=64 time=235 ms
64 bytes from 192.168.3.12: icmp_seq=256 ttl=64 time=94.1 ms
64 bytes from 192.168.3.12: icmp_seq=257 ttl=64 time=573 ms
64 bytes from 192.168.3.12: icmp_seq=258 ttl=64 time=619 ms
64 bytes from 192.168.3.12: icmp_seq=259 ttl=64 time=842 ms
64 bytes from 192.168.3.12: icmp_seq=260 ttl=64 time=943 ms
64 bytes from 192.168.3.12: icmp_seq=261 ttl=64 time=1144 ms
64 bytes from 192.168.3.12: icmp_seq=262 ttl=64 time=711 ms
64 bytes from 192.168.3.12: icmp_seq=263 ttl=64 time=450 ms
64 bytes from 192.168.3.12: icmp_seq=264 ttl=64 time=182 ms
64 bytes from 192.168.3.12: icmp_seq=265 ttl=64 time=314 ms
64 bytes from 192.168.3.12: icmp_seq=266 ttl=64 time=125 ms
64 bytes from 192.168.3.12: icmp_seq=267 ttl=64 time=1519 ms
64 bytes from 192.168.3.12: icmp_seq=268 ttl=64 time=899 ms
64 bytes from 192.168.3.12: icmp_seq=269 ttl=64 time=818 ms
64 bytes from 192.168.3.12: icmp_seq=270 ttl=64 time=991 ms
64 bytes from 192.168.3.12: icmp_seq=271 ttl=64 time=811 ms
64 bytes from 192.168.3.12: icmp_seq=272 ttl=64 time=1082 ms
64 bytes from 192.168.3.12: icmp_seq=273 ttl=64 time=647 ms
64 bytes from 192.168.3.12: icmp_seq=274 ttl=64 time=204 ms
64 bytes from 192.168.3.12: icmp_seq=275 ttl=64 time=361 ms
64 bytes from 192.168.3.12: icmp_seq=276 ttl=64 time=193 ms
64 bytes from 192.168.3.12: icmp_seq=277 ttl=64 time=93.8 ms
64 bytes from 192.168.3.12: icmp_seq=278 ttl=64 time=682 ms
64 bytes from 192.168.3.12: icmp_seq=279 ttl=64 time=1089 ms
64 bytes from 192.168.3.12: icmp_seq=280 ttl=64 time=1212 ms
64 bytes from 192.168.3.12: icmp_seq=281 ttl=64 time=807 ms
64 bytes from 192.168.3.12: icmp_seq=282 ttl=64 time=1191 ms
64 bytes from 192.168.3.12: icmp_seq=284 ttl=64 time=143 ms
64 bytes from 192.168.3.12: icmp_seq=283 ttl=64 time=1523 ms
64 bytes from 192.168.3.12: icmp_seq=285 ttl=64 time=283 ms
64 bytes from 192.168.3.12: icmp_seq=286 ttl=64 time=342 ms
64 bytes from 192.168.3.12: icmp_seq=287 ttl=64 time=1501 ms
64 bytes from 192.168.3.12: icmp_seq=288 ttl=64 time=1181 ms
64 bytes from 192.168.3.12: icmp_seq=289 ttl=64 time=1234 ms
64 bytes from 192.168.3.12: icmp_seq=290 ttl=64 time=940 ms
64 bytes from 192.168.3.12: icmp_seq=291 ttl=64 time=1172 ms
64 bytes from 192.168.3.12: icmp_seq=292 ttl=64 time=1378 ms
64 bytes from 192.168.3.12: icmp_seq=294 ttl=64 time=308 ms
64 bytes from 192.168.3.12: icmp_seq=295 ttl=64 time=456 ms
64 bytes from 192.168.3.12: icmp_seq=293 ttl=64 time=2710 ms
64 bytes from 192.168.3.12: icmp_seq=296 ttl=64 time=1398 ms
64 bytes from 192.168.3.12: icmp_seq=297 ttl=64 time=571 ms
64 bytes from 192.168.3.12: icmp_seq=298 ttl=64 time=864 ms
64 bytes from 192.168.3.12: icmp_seq=299 ttl=64 time=601 ms
64 bytes from 192.168.3.12: icmp_seq=300 ttl=64 time=1515 ms
64 bytes from 192.168.3.12: icmp_seq=301 ttl=64 time=1181 ms
64 bytes from 192.168.3.12: icmp_seq=302 ttl=64 time=1451 ms
64 bytes from 192.168.3.12: icmp_seq=304 ttl=64 time=268 ms
64 bytes from 192.168.3.12: icmp_seq=305 ttl=64 time=479 ms
64 bytes from 192.168.3.12: icmp_seq=306 ttl=64 time=229 ms
64 bytes from 192.168.3.12: icmp_seq=307 ttl=64 time=10.0 ms
64 bytes from 192.168.3.12: icmp_seq=308 ttl=64 time=355 ms
64 bytes from 192.168.3.12: icmp_seq=309 ttl=64 time=175 ms
64 bytes from 192.168.3.12: icmp_seq=310 ttl=64 time=214 ms
64 bytes from 192.168.3.12: icmp_seq=311 ttl=64 time=53.4 ms
64 bytes from 192.168.3.12: icmp_seq=312 ttl=64 time=748 ms
64 bytes from 192.168.3.12: icmp_seq=313 ttl=64 time=2025 ms
64 bytes from 192.168.3.12: icmp_seq=314 ttl=64 time=1581 ms
64 bytes from 192.168.3.12: icmp_seq=315 ttl=64 time=1668 ms
64 bytes from 192.168.3.12: icmp_seq=316 ttl=64 time=1143 ms
64 bytes from 192.168.3.12: icmp_seq=317 ttl=64 time=723 ms
64 bytes from 192.168.3.12: icmp_seq=318 ttl=64 time=673 ms
64 bytes from 192.168.3.12: icmp_seq=319 ttl=64 time=932 ms
64 bytes from 192.168.3.12: icmp_seq=320 ttl=64 time=1152 ms
64 bytes from 192.168.3.12: icmp_seq=321 ttl=64 time=1355 ms
64 bytes from 192.168.3.12: icmp_seq=322 ttl=64 time=1025 ms
64 bytes from 192.168.3.12: icmp_seq=323 ttl=64 time=993 ms
64 bytes from 192.168.3.12: icmp_seq=324 ttl=64 time=69.7 ms
64 bytes from 192.168.3.12: icmp_seq=325 ttl=64 time=73.8 ms
64 bytes from 192.168.3.12: icmp_seq=326 ttl=64 time=1140 ms
64 bytes from 192.168.3.12: icmp_seq=327 ttl=64 time=595 ms

I have a ping over 500ms on the link with latency 40ms. Did I miss something important with the setup, or is it a natural limitation of the OpenVPN?

On both machines I use OpenVPN on Linux. Client has kernel 3.13.0 with OpenVPN 2.3.2 x86_64-pc-linux-gnu [SSL (OpenSSL)] [LZO] [EPOLL] [PKCS11] [eurephia] [MH] [IPv6] and server has kernel 2.6.22.19 with OpenVPN 2.3.2 mipsel-unknown-linux-gnu [SSL (OpenSSL)] [LZO] [EPOLL] [eurephia] [MH] [IPv6]

Sometimes there are complicated interactions between packages. To avoid class interface with countless, seemingly unrelated parameters, I though about using tags, defined(<resource>) function to let the resources slightly change their behaviour based on the rest of the configuration (not facts).

Unfortunately I can't figure out a way to let the Puppet first parse the class that can generate conditions, and only then parse the class which behaviour implicitly (via e.g. tags) depends on the way the first class got executed.

Suppose here is a class, that manages a place, where a specific user stores his sensitive personal data. The class can e.g. create the partition (possibly encrypted), or just maintain an external folder (on e.g. /mnt/user). Or it can keep the folders where they are by default, i.e. in the /home/user (spurcetype=>'none')

class userdocs ( $sourcetype) {
  case $sourcetype {
    'cryptpartition' : {
      tag 'dedicated_user_docs_partition'
      ...
    }
    'partition' : {
      tag 'dedicated_user_docs_partition'
      ...
    }
    'none' : {
      ...
    }
  }
  ...
}

Apart from fairly standard set of folders, that a user might want to forward to his partition, there are folders which existence depends on the particular software running, like thunderbird or virtualbox. Creation of the link (instead of folder) for e.g. virtualbox should be done only, if two conditions are satisfied: a) user manages separate partition (tagged('dedicated_user_docs_partition')) and user installs VirtualBox.

There must be a place, where I can put that extra code, that checks for those two conditions. I can put it into the class, that manages the virtualbox:

class vbox {
  if tagged('dedicated_user_docs_partition') {
    file { "${userhome}/.VirtualBox": ensure=>link, require=>Class['userdocs'] ...
    }
  } else {
    file { "${userhome}/.VirtualBox": ensure=>directory, require=>Class['userdocs'] ...
    }
  }
  ...
}

The problem is, that although I explicitly require the file resource to be executed before the userdocs class, the file resource is evaluated after that, and puppet wrongly assumes, that user doesn't have the dedicated_user_docs_partition and executes wrong branch of the if statement.

Is there any way to solve this problem?

I've cross-posted this question on ask.puppet.com.

I followed some "best practices" from Puppet 2.7 Cookbook and created agent - puppet master framework, with git cvs. Unfortunately, the book doesn't cover Gepetto, so I am forced to improvise that part.

I have one main git repository that contains manifests and modules folders. I want to add a custom module in the modules, but when doing so I encountered problems.

First of all, I cannot find anything like "master configuration" project type, which by default would contain the manifests and modules folders. So I created a bare "Puppet project" project, and created the folders manually.

Because - as I understand it - in this configuration Gepetto doesn't really know, that this project will actually hold all the modules, it doesn't provide an option to create "Puppet module project" subproject, which would be nested inside the "Puppet project"'s modules folder.

When I nest it manually (by specifying a custom location in the New Puppet Module Project wizard), I get an 'Auto share git projects' has encountered a problem error:

Unable to ignore resources Attempted to beginRule: F/puppet@puppetmaster/modules/unnamed, does not match outer scope rule: MultiRule[P/mymodule]

Did anyone of you managed to get Gepetto working with modules when everything is covered by something like "master project" that encapsulates all configuration?

I've accidentally overwritten an important file on BTRFS.

Knowing that this filesystem is COW, it should be in theory possible to recover an old version of the file.

Unfortunately googling didn't reveal relevant information. Can someone please advice me, what steps can I do (if any)?

I use Linux and I am in the process of migrating from one Linux Mint 15 into Linux Mint 16. Each OS is installed on the same partition as a different subvolume on the same btrfs partition.

As I use both systems at the moment, I'd like to share all LXC Containers across both distributions.

I deep-copied the whole /var/lib/lxc/<container> path to separate btrfs subvolume.

Surprisingly it is not as easy as it seems. Each time when I make any form of linking of the container source into /var/lib/lxc/ on the host, I got an error (lxc-start: command get_init_pid failed to receive response). Unfortunately, any form of delegation (symlinks, mount --bind) doesn't work.

Copying the container with rsync -aAX works well, but I don't want to maintain two separate instances of the container, but to share one instance between two host OSes.

I believe my server has just got compromised interactively.

As this server will go to thrash anyway, I wonder if I can get a better insight of what the villain is doing by forcing logging of all tty.

Years ago I've seen a kernel patch, that does just that, but I can't find it anymore. I wonder, if such patch exist (I don't know how to write one).

The best would be, if the output wouldn't be stored locally, but forwarded to my other host (e.g. by http post or even as netcat).

My host uses 3.8 mainline kernel.

That should be a quick one:

I'm running a small server, and one day discovered, that one of the LXC containers inside is down. I started it again, but I want to find out, for how long it was down.

I guess, that the most detailed information will be in dmesg logs, but to read timestamps in them I need to know exact moment the system was started during that session.

I routinely use Unison to sync users' home directories between workstations where the user is expected to work. Unfortunately, as the firm grows, the Unison become slower and slower in determining what files have changed. The time taken by actual transfer is negligible in comparison.

The synchronization is done in the star topology, with RAID-6 unison server in the center. Some workstations use Windows (with NTFS), some Linux with either Ext-4 or BTRFS(!).

At the time of writing, there is one user, whose home directory is 45GB large with 100K files, and the full synchronization time takes about 30 minutes for him. Note, that simple directory traversal with find >null takes less about 2 minutes.

What are the strategies for further speeding up the process? (except reducing the number of files to sync) I believe that in theory the Unison can be sped up, but the fastcheck option isn't enough.

I've read that "It is not recommended that RAID types be mixed in a pool", which implies that it is possible. Why is it not recommended?

It would be very desirable in my setup, which consists of a mix of very-important, important and not important files, to dynamically assign different RAID-Z levels for different file systems inside ONE pool.

The second question is: what would happen, if disks begin failing? I understand, that I wouldn't be able to access not sufficiently raided file systems. But what if the raided ones (with sufficient redundancy to handle the failure) are nested inside file system that failed? Would they by available? How?

I'm planning to use ZOL, version 0.6.0.80, which compatible with zpool version 28 and zfs version 5.

Welcome.

I try to recover from linux software raid failure. I am in a process of recreating the superblocks based on a syslog from the system when it was healthy. The log speaks about 2 md devices, each based on 5 500GB partitions. The RAID-5 /dev/md5 is composed from /dev/sd[b-f]6 and RAID-6 /dev/md6 is composed from /dev/sd[b-f]5. The relevant part dmesg log looks like this:

[    5.964928] raid5: allocated 5265kB for md5
[    5.979771] 4: w=1 pa=0 pr=5 m=1 a=2 r=5 op1=0 op2=0
[    5.987064] 1: w=2 pa=0 pr=5 m=1 a=2 r=5 op1=0 op2=0
[    5.994256] 3: w=3 pa=0 pr=5 m=1 a=2 r=5 op1=0 op2=0
[    6.001441] 2: w=4 pa=0 pr=5 m=1 a=2 r=5 op1=0 op2=0
[    6.008665] 0: w=5 pa=0 pr=5 m=1 a=2 r=5 op1=0 op2=0
[    6.015883] raid5: raid level 5 set md5 active with 5 out of 5 devices, algorithm 2
[    6.031441] RAID5 conf printout:
[    6.039266]  --- rd:5 wd:5
[    6.046856]  disk 0, o:1, dev:sdb6
[    6.054308]  disk 1, o:1, dev:sdf6
[    6.061524]  disk 2, o:1, dev:sdc6
[    6.068667]  disk 3, o:1, dev:sde6
[    6.075774]  disk 4, o:1, dev:sdd6
[    6.089504] md5: bitmap initialized from disk: read 1/1 pages, set 0 bits
[    6.096768] created bitmap (15 pages) for device md5
[    6.141010] md5: detected capacity change from 0 to 1978409418752
[    6.153794]  md5: unknown partition table

[    5.594614] raid5: device sde5 operational as raid disk 3
[    5.599701] raid5: device sdd5 operational as raid disk 4
[    5.604750] raid5: device sdf5 operational as raid disk 1
[    5.609879] raid5: device sdc5 operational as raid disk 2
[    5.615032] raid5: device sdb5 operational as raid disk 0
[    5.621672] raid5: allocated 5265kB for md6
[    5.627243] 3: w=1 pa=0 pr=5 m=2 a=2 r=5 op1=0 op2=0
[    5.632846] 4: w=2 pa=0 pr=5 m=2 a=2 r=5 op1=0 op2=0
[    5.638502] 1: w=3 pa=0 pr=5 m=2 a=2 r=5 op1=0 op2=0
[    5.644155] 2: w=4 pa=0 pr=5 m=2 a=2 r=5 op1=0 op2=0
[    5.649844] 0: w=5 pa=0 pr=5 m=2 a=2 r=5 op1=0 op2=0
[    5.655566] raid5: raid level 6 set md6 active with 5 out of 5 devices, algorithm 2
[    5.667653] RAID5 conf printout:
[    5.673791]  --- rd:5 wd:5
[    5.679837]  disk 0, o:1, dev:sdb5
[    5.685721]  disk 1, o:1, dev:sdf5
[    5.691570]  disk 2, o:1, dev:sdc5
[    5.697348]  disk 3, o:1, dev:sde5
[    5.703122]  disk 4, o:1, dev:sdd5
[    5.710762] md6: bitmap initialized from disk: read 1/1 pages, set 0 bits
[    5.716915] created bitmap (15 pages) for device md6

• What do the numbers on the line [ 5.979771] 4: w=1 pa=0 pr=5 m=1 a=2 r=5 op1=0 op2=0 mean, including the first 4:?
• What do the numbers on the line [ 6.039266] --- rd:5 wd:5] mean?
• What do the numbers on the line [ 6.046856] disk 0, o:1, dev:sdb6 mean?
• Does the line [ 6.096768] created bitmap (15 pages) for device md5 tell anything about the place, where the bitmap is located and what is its size?

I try to recover from mdadm raid disaster, which happened when moving from ubuntu server 10.04 to 12.04. I know the correct order of devices from dmesg log, but given this information, I still cannot access the data.

The superblocks look messy; the mdadm --examine for each disk is on this question on askubuntu

By inspecting the raw contents of backing storage, I found the beginning of my data (the LUKS container in my case) at position 0x22000 relative to the beginning of the first partition in the raid.

Question: What is the combination of options issued to "mdadm --create" to re-create mdadm that starts with the given offset? Bitmap size?

PS. The relevant information from syslog when the system was healthy are pasted here.

I'm attempting to recover from a failure of my raid volume after upgrade from ubuntu 10.04 to 12.04.

I've tried re-creating the array in any combination of 5 elementary volumes with one replaced with "missing" to ensure that the array wakes as degraded.

Next, with help of dd if=/dev/md1, I made a backup of the first 256kB of each version of my reassembled raid for inspection.

To my astonishment I see only 5 distinct version of the first 256kB chunk out of possible 120 permutations on a 5 disk set or even 24 on a 4 disk set. I assume the 4 disk set number should be correct, because 1 volume must be set as missing and henceforth shouldn't be accounted for.

How can this occur?