Our Nomad agents sometimes fails jobs because they cannot pull images from ECR. /var/log/docker will contain messages like:
Not continuing with pull after error: error pulling image configuration: Get https://prod-eu-west-1-starport-layer-bucket.s3.eu-west-1.amazonaws.com/uuid/uuid?...: dial tcp IP:443: i/o timeout
The jobs are not really idempotent, so I would prefer not to have Nomad retry them.
Is there any way I can instruct Docker to retry these operations?
(Followup on GKE pod connecting via VPN?)
I am trying to connect a GKE cluster to a remote network using a GCE VPN to a Cisco ASA 5510. Ping from GKE pod 10.248.0.26 -> remote node 10.99.193.115 arrives at 10.99.193.115 and the ASA says that the echo reply goes back through the tunnel to GKE. However, tcpdump on 10.248.0.26 shows no replies coming in.
Firewall and routing as reported by Google Cloud Console:
Name Source tag / IP range Allowed protocols / ports Target tags
default-allow-icmp 0.0.0.0/0 icmp Apply to all targets
default-allow-internal 10.240.0.0/16 tcp:1-65535; udp:1-65535; icmp Apply to all targets
default-allow-ssh 0.0.0.0/0 tcp:22 Apply to all targets
gke-zecluster-d6cc7a55-all 10.248.0.0/14 tcp; udp; icmp; Apply to all targets
gke-zecluster-d6cc7a55-ssh <public_ip>/32 tcp:22 gke-zecluster-d6cc7a55-node
gke-zecluster-d6cc7a55-vms 10.240.0.0/16 tcp:1-65535; udp:1-65535; icmp gke-zecluster-d6cc7a55-node
k8s-fw-a1a92183fb18e11e5be3442010af0001 0.0.0.0/0 tcp:80,443 gke-zecluster-d6cc7a55-node
k8s-fw-a1aa3fe95b18e11e5be3442010af0001 0.0.0.0/0 tcp:2003 gke-zecluster-d6cc7a55-node
Name Destination IP ranges Priority Instance tags Next hop
default-route-3eed071cad0670e8 0.0.0.0/0 1000 None Default internet gateway
default-route-7a9ddc4457c714a0 10.240.0.0/16 1000 None Virtual network
gke-zecluster-d6cc7a55-7b61213c-b187-11e5-be34-42010af00015 10.248.0.0/24 1000 None gke-zecluster-d6cc7a55-node-j4jx (Zone ze-zone-1)
gke-zecluster-d6cc7a55-7ec5f7a9-b187-11e5-be34-42010af00015 10.248.1.0/24 1000 None gke-zecluster-d6cc7a55-node-rluf (Zone ze-zone-1)
vpn-1-tunnel-1-route-1 10.99.0.0/16 1000 None
Is there some logging I can turn on to see what goes on? As far as I can see, the VPN says nothing pertinent about this traffic, only:
15:24:51.058 sending DPD request
15:24:51.058 generating INFORMATIONAL_V1 request 3069408857 [ HASH N(DPD) ]
15:24:51.058 sending packet: from <gce-vpn-ip>[500] to <asa-ip>[500] (92 bytes)
15:24:51.092 received packet: from <asa-ip>[500] to <gce-vpn-ip>[500] (92 bytes)
15:24:51.092 parsed INFORMATIONAL_V1 request 146600869 [ HASH N(DPD_ACK) ]
If I modify the VPN tunnel (GCE VPN, ASA) to have the default net 10.240.0.0/16 at the GCE end traffic passes correctly in both directions.
I assume this is a routing issue, but what? Should not the route 10.248.0.0/24 send the traffic back to the GKE node? Or do I have to somehow declare the GKE network as a network?
I have a GKE cluster with a handful of nodes and I would like pods in this cluster to be able to connect to remote hosts on a private network that can be reached via a site-to-site VPN provided by GCE. As far as I can tell, there is no simple way to assign an address to a pod for outbound connections? (It does not seem feasible to add each pod-cidr to VPN configuration each time a node is added or replaced.) Do I have to set up a NAT bradge external to the cluster, or is there some Kubernetes way to control the outbound address of a pod?
It appears the reply for pool.ntp.org changed recently. This is making my CentosOS 6 ntp servers unhappy.
$ host pool.ntp.org
pool.ntp.org has address 0.0.0.2
pool.ntp.org has address 83.209.8.142
pool.ntp.org has address 130.236.254.17
pool.ntp.org has address 195.178.181.98
$ /usr/lib64/nagios/plugins/check_ntp_time -H pool.ntp.org
can't create socket connection#
$ strace -f /usr/lib64/nagios/plugins/check_ntp_time -H pool.ntp.org
...
connect(3, {sa_family=AF_INET, sin_port=htons(123), sin_addr=inet_addr("83.209.8.142")}, 16) = 0
socket(PF_INET, SOCK_DGRAM, IPPROTO_UDP) = 3
connect(3, {sa_family=AF_INET, sin_port=htons(123), sin_addr=inet_addr("130.236.254.17")}, 16) = 0
socket(PF_INET, SOCK_DGRAM, IPPROTO_UDP) = 4
connect(4, {sa_family=AF_INET, sin_port=htons(123), sin_addr=inet_addr("195.178.181.98")}, 16) = 0
socket(PF_INET, SOCK_DGRAM, IPPROTO_UDP) = 5
connect(5, {sa_family=AF_INET, sin_port=htons(123), sin_addr=inet_addr("83.209.8.142")}, 16) = 0
socket(PF_INET, SOCK_DGRAM, IPPROTO_UDP) = 6
connect(6, {sa_family=AF_INET, sin_port=htons(123), sin_addr=inet_addr("0.0.0.2")}, 16) = -1 EINVAL (Invalid argument)
fstat(1, {st_mode=S_IFCHR|0620, st_rdev=makedev(136, 1), ...}) = 0
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f6571c5b000
write(1, "can't create socket connection", 30can't create socket connection) = 30
exit_group(3) = ?
+++ exited with 3 +++
It appears that there is general agreement on this, as modern Ubuntu says:
$ ping 0.0.0.2
connect: Invalid argument
I thought 0.0.0.2 is a valid IP?
UPDATE
The issue is indeed periodic and has been going on for several days (since 2015-12-20 according to my nagios):
[12:40] host pool.ntp.org
pool.ntp.org has address 194.71.144.71
pool.ntp.org has address 79.136.86.176
pool.ntp.org has address 83.209.8.142
pool.ntp.org has address 178.73.198.130
[13:09] host pool.ntp.org
pool.ntp.org has address 194.71.144.71
pool.ntp.org has address 0.0.0.2
pool.ntp.org has address 178.73.198.130
pool.ntp.org has address 192.36.143.130
I suppose there is a ranking battle of some sort going on.
UPDATE 2
For those interested, this issue occurs when DNS RRs for pool.ntp.org is queried from Sweden.
I need to rsync a file tree to a specific pod in a kubernetes cluster. It seems it should be possible if only one can convince rsync that kubectl acts sort of like rsh. Something like:
rsync --rsh='kubectl exec -i podname -- ' -r foo x:/tmp
... except that this runs into problems with x since rsync assumes a hostname is needed:
exec: "x": executable file not found in $PATH
I can not seem to find a method to help rsync construct the rsh command. Is there a way to do this? Or some other method by which relatively efficient file transfer can be achieved over a pipe?
(I am aware of gcloud compute copy-files, but it can only be used onto the node?)
I have a Linux guest that uses an LVM volume directly as root file system (that is, there is no partition table). libvirt config looks thus:
<os>
<type arch='x86_64' machine='rhel6.4.0'>hvm</type>
<kernel>/boot/vmlinuz-X.Y.Z.el6.x86_64</kernel>
<initrd>/boot/initramfs-X.Y.Z.el6.x86_64.img</initrd>
<cmdline>console=ttyS0 root=/dev/vda</cmdline>
<boot dev='hd'/>
</os>
<disk type='block' device='disk'>
<driver name='qemu' type='raw' cache='none' io='native'/>
<source dev='/dev/vg/guest'/>
<target dev='vda' bus='virtio'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/>
</disk>
From inside the guest:
$ mount
/dev/vda on / type ext4 (rw)
proc on /proc type proc (rw)
sysfs on /sys type sysfs (rw)
devpts on /dev/pts type devpts (rw,gid=5,mode=620)
tmpfs on /dev/shm type tmpfs (rw)
none on /proc/sys/fs/binfmt_misc type binfmt_misc (rw)
Is it possible to resize the guest's root partition without rebooting the guest? Just doing lvextend on the host and resize2fs from the guest does not seem to be enough.
I have several boxen running Debian Wheezy. They appear to have the highly annoying problem that pressing ctrl+c in a shell that has been su:d to root will kill su, not whatever is running in the root shell. This makes working with e.g. ping or tcpdump near impossible. As I understand it, the issue is debated at length in Debian bug #628843 but there seems to be no consensus.
[20:38] alias
zsu='su root -p -c $SHELL'
[20:38] zsu
Password:
zsh compinit: insecure directories and files, run compaudit for list.
Ignore insecure directories and files and continue [y] or abort compinit [n]? y
[20:38] <- I simply pressed ctrl+c
Session terminated, terminating shell...
[20:38] ...killed.
I need to find a fix for this. Can I execute su differently or is there a replacement I can use instead? (Preferably no sudo and I would prefer not to allow ssh root login.)
I have an saslauthd setup to authenticate against PAM. It seems to do its stuff:
root@sasltest:~# testsaslauthd -u quest -p #### -s smtp
0: OK "Success."
I have libsasl 2.1.23, postfix 2.7.1.
I have a postfix configured thus:
smtpd_sasl_type = cyrus
smtpd_sasl_path = /var/spool/postfix/private/saslauthd/mux
smtpd_sasl_auth_enable = yes
broken_sasl_auth_clients = yes
smtpd_sasl_security_options = noanonymous
With a master.cf thus:
submission inet n - - - - smtpd
-o smtpd_tls_security_level=encrypt
-o smtpd_sasl_auth_enable=yes
-o smtpd_client_restrictions=permit_sasl_authenticated,reject
However, trying to authenticate in this postfix gives the following error message:
Jan 23 22:13:14 sasltest postfix/smtpd[1252]: warning: SASL authentication problem: unable to open Berkeley db /etc/sasldb2: No such file or directory
Jan 23 22:13:14 sasltest postfix/smtpd[1252]: warning: SASL authentication problem: unable to open Berkeley db /etc/sasldb2: No such file or directory
Jan 23 22:13:14 sasltest postfix/smtpd[1252]: warning: X[A.B.C.D]: SASL LOGIN
authentication failed: authentication failure
Meanwhile, there is no output from my debug-logging saslauthd.
I interpret this as meaning that libsasl2 tries to uses sasldb auth rather than try to talk to saslauthd. What I can't figure out how to tell libsasl that I want it to talk to saslauthd.
Various instructions inform you to create a file /etc/sasl2/smtpd.conf or /etc/postfix/sasl/smtpd.conf. I have tried creating these files containing:
pwcheck_method: saslauthd
mech_list: LOGIN PLAIN
But to no effect.
How do I instruct libsasl to use saslauthd authentication?
(I can of course create /var/spool/postfix/etc/sasldb2, but this will still not result in connections to saslauthd.)
I am in the process of switching from Xen to qemu-kvm. My current Xen installation has a number of LVM volumes each with a partition (rather than a full disk image). In Xen, I start each VM with a configuration like so:
kernel = "/boot/vmlinuz-2.6.18-128.4.1.el5xen"
ramdisk = "/boot/initrd-xen-2.6.18-128.4.1.el5xen"
disk = [ 'phy:/dev/vmdata/vm-standalone,sda1,w' ]
I now want to reuse these VMs but start them under kvm instead. It seems that virt-install --import would let me do this, using --boot kernel=<kernel>,initrd=<initrd>, but I don't see how I can instruct virt-install to attach the existing volume as boot/root file system. Is this possible?
I'm trying to figure out how to assign IP addresses from an ippool to mobile units. Ideally, I would like a unit to receive the same IP at each request. There is no data source to use to populate static entries for each unit, so I would prefer each unit to get an IP assigned from a pool first time it connects and then keep that address.
Currently, I have the following pool:
ippool dapool {
range-start = X.X.0.5
range-stop = X.X.0.254
netmask = 255.255.255.0
# should have room for all IPs:
cache-size = 260
session-db = ${db_dir}/db.ippool
ip-index = ${db_dir}/db.ipindex
override = yes
maximum-timeout = 0
key = "%{Calling-Station-Id}"
}
I thought that choosing a key that is unique for the requesting unit would do the trick, but apparently not.
1) Is it possible to achieve what I want?
2) Is there any good documentation that I should have read instead of pestering serverfault? I can't seem to find much useful documentation on FreeRadius beyond what is listed in the config files.
I have an Cisco ASA 5510 configured thus:
interface Ethernet0/0
description ### Trunk for inside, wlan ###
speed 1000
no nameif
no security-level
no ip address
interface Ethernet0/0.10
description ### OFFICE ###
vlan 10
nameif inside
security-level 100
ip address 172.18.0.1 255.255.255.0
interface Ethernet0/0.12
description ### WIRELESS ###
vlan 12
nameif wlan
security-level 20
ip address 172.18.2.1 255.255.255.128
interface Ethernet0/3
description ### Upstream ###
nameif outside
security-level 0
ip address X.X.X.X 255.255.255.252
access-group WLAN in interface wlan
global (outside) 10 interface
nat (wlan) 0 access-list NONATWIRELESS
nat (wlan) 10 172.18.2.0 255.255.255.128
nat (inside) 0 access-list NONATINSIDE
nat (inside) 10 172.18.0.0 255.255.255.0
dhcprelay server ZZZ inside
dhcprelay enable wlan
access-list WLAN extended permit object-group DNS object-group WLAN host nic
access-list WLAN extended permit object-group DNS object-group WLAN host idns
access-list NONATWIRELESS extended permit ip any 172.18.0.0 255.255.255.0
access-list NONATWIRELESS extended permit ip any 172.18.3.0 255.255.255.0
access-list NONATINSIDE extended permit ip any 172.18.2.0 255.255.255.0
access-list NONATINSIDE extended permit ip any 172.18.3.0 255.255.255.0
no nat-control
There are no static routes.
On this configuration, hosts on vlan 10 are allowed to access the outside world, but hosts on vlan 12 are not. They provoke like log entries:
Jan 13 14:35:02 172.18.0.1 %ASA-4-106023: Deny tcp src wlan:172.18.2.125/48593 dst outside:Y.Y.Y.Y/80 by access-group "WLAN" [0x0, 0x0]
How come?
EDIT: I suppose this is because there is an access-list on wlan but not on inside, but this seems stupid? security-level should still be evaluated if there are no matching entries in the access-list?
My ASA 5510 has the following configuration for an interface. My Ubuntu box (2.6.35) connected to this network will correctly autoconf an IPv6 address, but it will not set a default route.
interface Ethernet0/0.10
vlan 10
no shutdown
nameif inside
security-level 100
ip address 172.18.0.1 255.255.254.0
ipv6 address REMOVED:1::1/64
ipv6 nd prefix REMOVED:1::/64
ipv6 nd ra-interval 120
ipv6 enable
Thus, ping6 REMOVED:1::1 works fine and if I manually add a default route for IPv6 it works fine.
The resulting router advertisement looks like this:
01:06:42.253895 IP6 (class 0xe0, hlim 255, next-header ICMPv6 (58) payload length: 64) fe80::21c:58ff:fed3:ea36 > ff02::1: [icmp6 sum ok] ICMP6, router advertisement, length 64
hop limit 64, Flags [none], pref medium, router lifetime 1800s, reachable time 0s, retrans time 1000s
source link-address option (1), length 8 (1): 00:1c:58:d3:ea:36
0x0000: 001c 58d3 ea36
mtu option (5), length 8 (1): 1500
0x0000: 0000 0000 05dc
prefix info option (3), length 32 (4): REMOVED:1::/64, Flags [onlink, auto], valid time 2592000s, pref. time 604800s
0x0000: 40c0 0027 8d00 0009 3a80 0000 0000 XXXX
0x0010: XXXX XXXX 0001 0000 0000 0000 0000
How come I do not get a default gateway set?
My tradition is to set all zone serials to the timestamp at modification. Now that Puppet is my new religion, I want to set serial timestamps when building zone files from exported resources. A somewhat trivialized example may look like this:
file { "/tmp/dafile": content = inline_template("<%= Time.now.to_i %>"), }
The problem with this approach is that content will be different all the time, which will (ultimately) provoke rebuilding of zone files on each puppet config poll.
Is there some way I can insert a timestamp without it being included in the data that is compared against previous state?
I have a machine that has a DNAT rule that redirects data to neighbouring server:
iptables -t nat -A PREROUTING -p udp -d 10.99.193.116 --dport 54321 -j DNAT --to 10.99.193.115:54321
As can be seen, this rule is triggered quite frequently:
pkts bytes target prot opt in out source destination
192K 19M DNAT udp -- * * 0.0.0.0/0 10.99.193.116 udp dpt:54321 to:10.99.193.115:54321
However, looking at the interface, the redirected packets are not sent out on the interface.
$ tcpdump -i eth0 -n port 54321
17:03:36.553363 IP 172.21.0.159.54321 > 10.99.193.116.54321: UDP, length 82
17:03:37.514257 IP 172.21.0.20.54321 > 10.99.193.116.54321: UDP, length 39
17:03:37.633556 IP 172.21.0.91.54321 > 10.99.193.116.54321: UDP, length 82
17:03:37.678211 IP 172.21.0.196.54321 > 10.99.193.116.54321: UDP, length 68
17:03:38.542620 IP 172.21.0.185.54321 > 10.99.193.116.54321: UDP, length 39
This problem occurred after the machine rebooted for the first time after adding these rules, so presumably there is some difference between then and now. But what? Or is there some kernel bug that means these packets are dropped even tho the counter is incremented?
The machine is a CentOS4 box, Linux 2.6.9-67.0.1.ELsmp.
I am trying to do Kerberos-auth:d NFS4 on EC2. In order to do this, it seems one wants a kernel at least 2.6.35 in order to get decent encryption algos. The distro I could find that provides this is Ubuntu/Natty, which has 2.6.38. However, the default images are the -virtual flavour, which lacks rpcsec_gss_krb5. Thus I'm trying to make an image that boots a -generic image, but these seem not to be appreciated by the EC2 pv-grub loader:
Xen Minimal OS!
start_info: 0xa01000(VA)
nr_pages: 0x26700
shared_inf: 0xbee66000(MA)
pt_base: 0xa04000(VA)
nr_pt_frames: 0x9
mfn_list: 0x967000(VA)
mod_start: 0x0(VA)
mod_len: 0
flags: 0x0
cmd_line: root=/dev/sda1 ro 4
stack: 0x946780-0x966780
MM: Init
_text: 0x0(VA)
_etext: 0x61e65(VA)
_erodata: 0x76000(VA)
_edata: 0x7b6d4(VA)
stack start: 0x946780(VA)
_end: 0x966d34(VA)
start_pfn: a10
max_pfn: 26700
Mapping memory range 0xc00000 - 0x26700000
setting 0x0-0x76000 readonly
skipped 0x1000
MM: Initialise page allocator for b3e000(b3e000)-0(26700000)
MM: done
Demand map pfns at 26701000-36701000.
Heap resides at 36702000-76702000.
Initialising timer interface
Initialising console ... done.
gnttab_table mapped at 0x26701000.
Initialising scheduler
Thread "Idle": pointer: 0x36702008, stack: 0xbf0000
Initialising xenbus
Thread "xenstore": pointer: 0x36702478, stack: 0x26600000
Dummy main: start_info=0x966880
Thread "main": pointer: 0x367028e8, stack: 0x26610000
"main" "root=/dev/sda1" "ro" "4"
vbd 2049 is hd0
******************* BLKFRONT for device/vbd/2049 **********
backend at /local/domain/0/backend/vbd/1312/2049
Failed to read /local/domain/0/backend/vbd/1312/2049/feature-barrier.
Failed to read /local/domain/0/backend/vbd/1312/2049/feature-flush-cache.
4194304 sectors of 0 bytes
**************************
[H
[J Booting 'Ubuntu Natty (development branch), kernel 2.6.38-11-virtual'
root (hd0)
Filesystem type is ext2fs, using whole disk
kernel /boot/vmlinuz-2.6.38-11-generic root=LABEL=uec-rootfs ro console=hvc0
initrd /boot/initrd.img-2.6.38-11-generic
ERROR Invalid kernel: elf_xen_note_check: ERROR: Will only load images built for the generic loader or Linux images
xc_dom_parse_image returned -1
Error 9: Unknown boot failure
Press any key to continue...
I guess my questions are thus:
Alternatively, what do I need to do to build a pv-grub/EC2 variant of the -generic kernel?
I am trying to build a new CentOS 6 VM from my CentOS 5 host, but it appears that something fundamental has changed between these versions:
$ rpm -ivh --nodeps --nodigest --ignorearch --ignoreos --force --root=/mnt centos-release-6-0.el6.centos.5.i686.rpm
warning: centos-release-6-0.el6.centos.5.i686.rpm: Header V3 RSA/SHA256 signature: NOKEY, key ID c105b9de
Preparing... ########################################### [100%]
1:centos-release ########################################### [100%]
error: unpacking of archive failed: cpio: Bad magic
$ rpm --version
RPM version 4.4.2.3
Did the magic number of RPMs change between these versions? How do I work around this?
EDIT (Clarification):
I am not trying to upgrade an existing machine. I am trying to create a new installation using yum --installroot=/mnt groupinstall core. In order to get a working config, step one is to install the release rpm which contains the yum repo config.
I have /etc/auto.master:
/cpy/users /etc/auto.users --timeout=90
I have /etc/auto.users:
* -fstype=nfs4,rw,proto=tcp,port=2049 server:/users/&
On an "ordinary" CentOS 5 machine, these work splendidly. On my virtual machine which has been built using "yum groupinstall core" and some minor adjustments, I get this when user myself logs in:
Apr 14 09:33:29 testing automount[5240]: handle_packet: type = 3
Apr 14 09:33:29 testing automount[5240]: handle_packet_missing_indirect: token 136, name myself, request pid 5249
Apr 14 09:33:29 testing automount[5240]: attempting to mount entry /cpy/users/myself
Apr 14 09:33:29 testing automount[5240]: lookup_mount: lookup(program): looking up myself
Apr 14 09:33:29 testing automount[5240]: lookup(program): lookup for myself failed
Apr 14 09:33:29 testing automount[5240]: ioctl_send_fail: token = 136
Apr 14 09:33:29 testing automount[5240]: handle_packet: type = 3
Apr 14 09:33:29 testing automount[5240]: handle_packet_missing_indirect: token 137, name myself, request pid 5249
Apr 14 09:33:29 testing automount[5240]: failed to mount /cpy/users/myself
Apr 14 09:33:29 testing automount[5240]: attempting to mount entry /cpy/users/myself
Apr 14 09:33:29 testing automount[5240]: ioctl_send_fail: token = 137
Apr 14 09:33:29 testing automount[5240]: failed to mount /cpy/users/myself
There is no network traffic against the NFS server when this occurs.
I realize that it can be a bit difficult to know exactly what is wrong, but any suggestion is welcome.
I need a cheap setup to store historic snapshots of a dataset in the 100-200GB range. These will see little use, but need to be kept online for some time (shared by Samba). The box housing the dataset is a Dell PowerEdge 750 with a PCI-X CERC 6-channel SATA controller. It has 4 SATA channels free.
I am considering the following setup. Your experience with similar solutions would be greatly appreciated.
I would add a 4-channel bracket in the free slot and connect these with ordinary SATA cables. I would then purchase a STARDOM 1U chassis or similar and use 4 ESATA cables to interconnect these to the bracket and use 4 ordinary 7200 rpm SATA disks in a RAID5 configuration using Linux software RAID because I want to be certain that I can read these disks when the PCI-X controller caves in.
Specifically, my questions are thus:
I want to build a server that exports user home directories over SMB/CIFS and NFS. This server would be joined to a Win2k3 AD domain controller that holds our user database. As I understand it, winbind will invent UIDs for these users on the fly. This username-UID mapping needs to be available to NFS clients that mount the home directories or file ownership will not be presented correctly.
I assume this can be achieved using SFU, but as far as I can tell SFU is discontinued and will not be supported on recent versions of Windows, so I would prefer not to use it.
How do I best provide this mapping to NFS clients?
(You would think this a common use case, but I fail to find a relevant howto. My Google-fu may be weak.)
EDIT: As an aside, would it be possible in this scenario for a user to connect over NFS without first having connected via SMB/CIFS?
I am replacing an old Linux file server serving NFS and CIFS. For the new server (still serving CIFS and NFS), I would like to have software that automatically and efficiently maintains old revisions of files in parallel trees, so that they can be accessed by users without special tools. I am looking for software that is akin to Time Machine or Flyback, but works well on a server.
The dataset is some 10000 files weighing maybe 60 GB. Changes are relatively few, usually less than 100 files changes daily.
Using LVM snapshots will not cut it, as the old revisions must reside on a separate set of disks from the live data.
Edit: To clarify: keeping old revisions is non-vital addition to the solution, so any suggestion will have to stay in the range of some hundred euros.