I purchased a new Western Digital Elements 3TB a couple of weeks ago.

As usual, I did all SMART tests, then badblocks, then SMART tests again.

When I first got the drive, SMART showed this:

 Short self-test routine 
 recommended polling time:   (   2) minutes.
 Extended self-test routine
 recommended polling time:   (   7) minutes.

I then ran a short SMART test and a long SMART test, which came back ok:

 SMART Self-test log structure revision number 1
 Num  Test_Description    Status                  Remaining  LifeTime(hours)  LBA_of_first_error
 # 1  Extended offline    Completed without error       00%         0         -
 # 2  Short offline       Completed without error       00%         0         -

Then, I ran a full badblocks destructive write test which showed no bad blocks.

Then, I ran another SMART short self-test, which also came back ok:

 SMART Self-test log structure revision number 1
 Num  Test_Description    Status                  Remaining  LifeTime(hours)  LBA_of_first_error
 # 1  Short offline       Completed without error       00%       261         -
 # 2  Extended offline    Completed without error       00%         0         -
 # 3  Short offline       Completed without error       00%         0         -

Then I ran a long SMART test again, and it gets stuck on 90%, even after 24 hours!

 Self-test execution status:      ( 249)    Self-test routine in progress...
                    90% of test remaining.

At some point during this, the extended self-test routine time has changed from 7 minutes to 682 minutes!

 Short self-test routine 
 recommended polling time:   (   2) minutes.
 Extended self-test routine
 recommended polling time:   ( 682) minutes.

I returned it thinking it was just a fault with the drive, but the replacement (also new) shows exactly the same error after the same sequence.

I have posted this issue to the WD Community but got no response.

Is this a batch issue (like a faulty EEPROM command) or a known issue (e.g. SMART firmware overrun tripped by a long sequence of 0xff?) with the drive? Should I return and try again or change to a different make/model?

I have a large number of systems (100s) managed by a small group of people which has changed over time. Each system is installed using a base image (which has its own version which is different depending on the age of the installation) which is then customised over time (forked) in various ways according to the needs of the client.

I have a copy of each version of the install image. Over 90% of the install image is the same between versions. Customisations are usually less than 3%.

I need to find out which versions are installed and what customisations have been made since the install.

Due to bandwidth constraints, I can't do a network diff or an rsync --dry-run over the network*.

However, I envisage being able to run a script over each install image, and send that as a database to each system to compare with its own filesystem and report back - like a "fingerprint", if you will.

The "fingerprint" (filesystem tree + checksum for each file & folder) would be limited to the fileset that are modifiable (and not /proc, /sys, /tmp, pipes, sockets, etc.).

The "fingerprint" can't be an MD5 of the filesystem because one change would result in a different fingerprint and we can't be sure which files may have been customised.

I'm looking for a utility that will report 2 things:

1. Suggest which version best matches the filesystem as it currently stands from a database of filesystem "fingerprints" (metadata of tree structure + file & folder checksum), and
2. List which files/folders have changed (customised) from that version, including new files and deleted files.

Additionally, it would be good if I could create new databases from existing ones so that I can take information from customisations to make new versions (e.g. Version 2.0.3-withmodX).

I've considered:

• Backup utilities - they presume that versions have a 1:1 linear progression per client
• Image management systems - tend to presume that images go server->client with only known customisation (e.g. new files, specific config folders), where we want information where client (references database)->server.

I could, perhaps, use git in some way to generate a '.git' database of the filesystem and then send multiple .git databases to compare against, then:

1. Least number of git status lines = version.
2. git status output against version = customisations.

Is there such a "fingerprint"-ing utility for filesystems or is there some utility that will make this easier to build?

*although I'm wondering if rsync can output a database of meta-information which could be used to build such a tool easily.

Here's my network diagram:

        56.56.56.56 192.168.0.1/24
                    MAC:AA:BB:CC:DD:EE:01
               ___________     
            ---| Modem 1 |-------
            |  ___________      |             ___________
Internet ---|               | Switch |--------| Machine |
            |  ___________      |             ___________
            ---| Modem 2 |-------            192.168.0.3/24
               ___________     
        67.67.67.67 192.168.0.2/24
                    MAC:AA:BB:CC:DD:EE:02

1. Both modems forward the same ports from the internet to the machine.
2. The machine behind the modems should respond appropriately to any request coming from the internet. e.g. Modem 1 packets return via Modem 1, Modem 2 packets return via Modem 2.
3. The machine has only one network port and the switch is unmanaged.
4. The machine uses Netplan, iptables, and iproute2 for network configuration.

The "What's new in Bareos 18" document says that in Bareos 17:

TLS is only started AFTER authentication is complete!

and under "Gloals(sic) achieved" (for version 18):

Per default the network communication is encrypted Certificate authority and certificates are not needed

but under "Gloals(sic) still open" it says:

CRAM-MD5 authencation is done inside of TLS Tunnel

Then, on the Bareos "What's new" web page it says:

Bareos 18.2.5 uses TLS encryption right from the start. All daemons (Bareos Director, File Daemon, and Storage Daemon) now support TLS encryption via pre-shared key (PSK) when authenticating.

Is it therefore now safe (using Bareos defaults) to do backups over a public network or do I still need to pipe remote backups over an encrypted tunnel?

I'm investigating using LVMRAID instead of MD+LVM. In my case, I'm looking at a RAID6.

As expected, the docs say you need N+1 devices for RAID5, and a minimum of 3 drives.

The docs say for RAID 6 you need N+2.

However, it then says that the minimum drives required is 5.

In an MD RAID6, only 4 drives are required. So if I built RAID6+LVM I'd only need 4 drives whereas, according to the docs, I'd need 5 drives for LVMRAID6.

Does LVMRAID6 really need 5 drives?

If so, why does it need 5 drives rather than MDRAID6's 4 drives?

Docker stores its containers, volumes, and images in a collection of large files under /var which is not the greatest thing for backing up incrementally.

With Bareos, I understand that backing up MySQL databases has the same issue and can be done with bpipe.

How can I backup Docker containers (and images, although there's another solution to that*) so that they are easily able to be restored to the same machine or another one?

*(There's also docker registry, which will let you push images to a remote repository, which works a bit like a backup also, but without the backup mindset. And it only works for images, not containers or volumes.)

I'm setting up a server, and my predecessor had a habit of setting up multiple RAID 1s with only 2 disks, which eventually resulted in split-brain situations which were difficult to recover from.

Since I want the same speed of access, would adding a third disk to the RAID 1 reduce the likelihood of a split-brain situation?

Or, to put it another way, Do disks "vote" in a RAID 1 when one of the disks contains an error on read?

Is there a better way to reduce the chance of hosing the RAID when one of the disks has a non-obvious failure?

I'm trying to nail down a buildroot (version 2017.02.4) config, and I need to separate kernel-related issues from software environment issues.

So I've tried to build a simple disk image which will work both in a Buildroot environment and a real machine.

I took a generic config, built it into an image, converted it to a vdi, and attached it to a Virtualbox VM.

make pc_x86_64_bios_defconfig
make clean
make
VBoxManage output/images/disk.img output/images/disk.vdi

This boots, but hangs at 'Booting Buildroot'

I did this also with a qemu_x86_64_defconfig (slightly modified to create a grub2 disk image using genimage) and got the same result. I did this before a few years ago, so I don't understand why it wouldn't work now.

What config is causing this to hang?

Do Virtualbox guests need a special hook to boot properly?

UPDATE: These images appear to work in qemu kvm, so it must be a Virtualbox config or specification issue.

I'm trying to build a quick script which listens for one UDP packet, stores it to a file, and exits.

socat to the rescue!

$ socat UDP-RECVFROM:9999 CREATE:/tmp/results_9999
2016/04/08 11:56:59 socat[1955] E read(6, 0x5220d3d0020, 8192): Bad file descriptor

Huh?

The file contains the packet received:

$ cat /tmp/results_9999
SUCCESS on Port 9999

strace seems to show it attempting to read the file it's written to:

recvfrom(5, "SUCCESS on Port 9999\n", 8192, 0, {sa_family=AF_INET, sin_port=htons(34032), sin_addr=inet_addr("192.168.56.102")}, [16]) = 10
recvfrom(3, 0x78ea087b3700, 519, 64, 0, 0) = -1 EAGAIN (Resource temporarily unavailable)
recvfrom(3, 0x78ea087b3d40, 519, 64, 0, 0) = -1 EAGAIN (Resource temporarily unavailable)
write(6, "SUCCESS on Port 9999\n", 10)             = 10
recvfrom(3, 0x78ea087b3d40, 519, 64, 0, 0) = -1 EAGAIN (Resource temporarily unavailable)
recvfrom(3, 0x78ea087b3840, 519, 64, 0, 0) = -1 EAGAIN (Resource temporarily unavailable)
read(6, 0xb675ac8f020, 8192)            = -1 EBADF (Bad file descriptor)
recvfrom(3, 0x78ea087b3840, 519, 64, 0, 0) = -1 EAGAIN (Resource temporarily unavailable)
open("/etc/localtime", O_RDONLY|O_NONBLOCK|O_CLOEXEC) = 7
fstat(7, {st_mode=S_IFREG|0644, st_size=2223, ...}) = 0
mmap(NULL, 2223, PROT_READ, MAP_SHARED, 7, 0) = 0x6c38b6779000
close(7)                                = 0
getpid()                                = 1954
writev(2, [{"", 0}, {"2016/04/08 11:56:41 socat[1954] "..., 84}], 22016/04/08 11:56:41 socat[1954] E read(6, 0xb675ac8f020, 8192): Bad file descriptor
) = 84
exit_group(1)                           = ?
+++ exited with 1 +++

Why would it need to read the file and, if so, why would they open the file write-only?

Is there a switch to fix this or is it a bug?

I'm playing with BareOS. I understand that, when everything is configured correctly, all backups are automatic and purges are managed so that nothing goes missing.

But while I'm getting the configuration right, I have quite a number of test backups that are of no use. In BAT, if I click on "delete" I get the warning:

Are you sure you want to delete?? !!!. (sic)

This delete command is used to delete a Job record and all associated catalog records that were created. This command operates only on the Catalog database and has no effect on the actual data written to a Volume. This command can be dangerous and we strongly recommend that you do not use it unless you know what you are doing. The Job and all its associated records (File and JobMedia) will be deleted from the catalog.Press OK to proceed with the delete operation.? (sic)

There are small variations to this message throughout the BAT software.

Now, I don't know what I'm doing so I've always clicked "cancel", but I have taken a subsequent Full backup. So I'm pretty sure that if I delete this I'll still have all the backups I need.

What is the correct process to delete backups and reclaim disk space manually?

I have a BareOS installation with very little modification to the default config files. There are Full, Incremental, and Differential backups being performed. Most clients appear to be being backed up as expected.

However, one of my clients appears to be repeatedly backing up over 10% of the overall filesystem in every incremental cycle.

How do I find the largest files and folders that are being backed up repeatedly?

BAT does not appear to be very helpful here, since it only lists the size of the file node itself, rather than the entire folder size. I'm effectively looking for a du command that works within the BareOS framework for a specific backup attempt.

I recently bought a set of 6 cables to connect my LSI MegaRAID card to my Norco-4224 backplane. The connector on both ends is SFF-8087.

However, when they arrived, the disks would not be detected. I thought the cable was faulty, but I tried all 6 and none work. So I borrowed an SFF-8087 cable from my Highpoint Tech RAID card and the disk was detected.

I tried poking a multimeter in, but they seemed to be wired correctly. Although the connectors are very fiddly so I can't guarantee my results.

The connectors between them appear to be different slightly. The Highpoint Tech cables have thick copper on the connectors PCB with varying lengths, whereas the ones purchased with the LSI have thin copper with a horizontal groove.

The ones I purchased look like this:

Whereas the Highpoint Tech ones look like this:

This has left me very puzzled! And with lots of questions like:

1. Is this enough to cause a connection fault?
2. Are the cables themselves faulty or just the connectors?
3. Do certain cards/backplanes need certain connector types?
4. Are the different copper lengths to ensure they can be hotplugged?
5. How does one better describe these differences?
6. How does one go about testing these cables?

Question

From the research I've done, the rack rail market seems to be very brand-centric. It's really hard to be sure that the rails I buy will match the servers as well as the rack. I'm in a position where I have rails that match the rack but not the servers, and rails that match the servers but not the rack.

From what I see, the main difference between the brands is the location of the screw holes for the inner part of the sliding mechanism. My question is two-fold:

1. Is there a diagram that shows across all brands what terms are used when specifying rail types and dimensions, options for "slotting", etc., but particularly depth (so that they reach the back post) and screw hole placement to match the servers?
2. Are there generic nU telescopic rail kits that would work on most nU servers regardless of brand?

Back story

I've been handed a bunch of 1RU IBM x3250 M2s, and I have to mount them into a 19" 4-post rack which is 29 inches deep with square holes. Unfortunately, I only have one pair of rails, and they don't seem to extend quite to my rack depth of 29 inches. They're short by maybe 0.75". I've tried inserting "rail extensions" (long bolts) in, but there are other issues...

The rails with the x3250s are basically two tiny shelves (one on each rail) half the length of the server which, when fully extended, seem only barely to support the units and aren't very stable since all the weight causes the rail to rotate. When fully extended, there appear small gaps which, if the angle of the server is not perfect as its pushed in, the server falls through. (Yikes!)

The rails I have from some newer Intel servers are sliding rails, and these I prefer since the servers can't fall out without lifting them or activating the release mechanism. However, the screw holes don't match. Before I go drilling holes in newer rails, I want to check that I'm using the right terminology when searching for rail kits.

I have a point of contention on my linux server. One of a number of processes access a single file and lock the file at a random time for a considerable period (>60 seconds) which, in turn, causes other things to fail.

Is there a way to detect how long a file has been locked and by which process?

I'm trying to automate the process of creating raw disk images. I don't care so much about C/H/S, but that the block size should be a rough standard of 512 bytes. However, I'm having trouble specifying the correct dimensions for partitions so that sfdisk can import them.

I first created a 32MB blank file:

$dd if=/dev/zero of=disk.img bs=1M count=32

Then I partitioned it using cfdisk:

$cfdisk -h 255 -s 63 -c 4 disk.img

• Verify that cfdisk sees the C/H/S switches (at the top of the screen)
• Create new partition 32.5MB, end of disk (to leave space for the boot stuff later)
• Write
• Quit

Then I output an sfdisk dump:

$sfdisk -H 255 -S 63 -C 4 -d disk.img > disk.parts

and got the following (in disk.parts):

# partition table of disk.img
unit: sectors

disk.img1 : start=     1276, size=    64260, Id=83, bootable
disk.img2 : start=        0, size=        0, Id= 0
disk.img3 : start=        0, size=        0, Id= 0
disk.img4 : start=        0, size=        0, Id= 0

However, when I try to reload that back into the image (as a test), sfdisk seems to at first accept the C/H/S switches when reading the original partition table and then throw them away when it tries to calculate the new partition table:

$sfdisk -H 255 -S 63 -C 4 disk.img < disk.parts

Warning: disk.img is not a block device
Disk disk.img: cannot get geometry

Disk disk.img: 4 cylinders, 255 heads, 63 sectors/track
Old situation:
Warning: The partition table looks like it was made
  for C/H/S=*/21/16 (instead of 4/255/63).
For this listing I'll assume that geometry.
Units = cylinders of 172032 bytes, blocks of 1024 bytes, counting from 0

   Device Boot Start     End   #cyls    #blocks   Id  System
disk.img1   *      3+    195-    192-     32130   83  Linux
        start: (c,h,s) expected (3,16,13) found (0,20,17)
        end: (c,h,s) expected (195,0,16) found (4,20,16)
disk.img2          0       -       0          0    0  Empty
disk.img3          0       -       0          0    0  Empty
disk.img4          0       -       0          0    0  Empty
New situation:
Warning: The partition table looks like it was made
  for C/H/S=*/21/16 (instead of 4/255/63).
For this listing I'll assume that geometry.
Units = sectors of 512 bytes, counting from 0

   Device Boot    Start       End   #sectors  Id  System
disk.img1   *      1276     65535      64260  83  Linux
        start: (c,h,s) expected (3,16,13) found (0,20,17)
        end: (c,h,s) expected (195,0,16) found (4,20,16)
disk.img2             0         -          0   0  Empty
disk.img3             0         -          0   0  Empty
disk.img4             0         -          0   0  Empty
Warning: partition 1 does not end at a cylinder boundary
end of partition 1 has impossible value for cylinders: 4 (should be in 0-3)

sfdisk: I don't like these partitions - nothing changed.
(If you really want this, use the --force option.)

It appears that these two sections are in conflict:

 Warning: The partition table looks like it was made
  for C/H/S=*/21/16 (instead of 4/255/63).
 For this listing I'll assume that geometry.
 Units = cylinders of 172032 bytes, blocks of 1024 bytes, counting from 0

and

 For this listing I'll assume that geometry.
 Units = sectors of 512 bytes, counting from 0

And it then reinforces it with:

 end of partition 1 has impossible value for cylinders: 4 (should be in 0-3)

I tried -f (force) and it gives exactly the same output. :-(

Why would sfdisk not process it's own dump format correctly, particularly when I'm already giving it all the information it needs? Why would it process the C/H/S when reading but not when writing? The C/H/S is not in the file, so why would it say that it looks like it was made for */21/16?

More importantly, how do I get around this so I can generate a partition table in a script?

I come across quite a few use cases where it would be very useful to take input from a (usually newline-delimited) stream and summarise it in a top-like fashion (see top, iotop, etc). A kind of on-the-fly pivot table.

e.g. Take the log-esque input:

I heard A from unit 1 and it said "Great!" 56
I heard A from unit 2 and it said "Oh no!" 42
I heard C from unit 1 and it said "Waiting for input." 33
I heard B from unit 3 and it said "Stopped." -1
...

From this, we could run a tool with a regex and group indicators:

topify [lineout] [regex] [name #1] [group #1] [name #2] [group #2] [All other columns name position]
     where:
         lineout is the number of lines before removing it from the display
         regex is a regex of the lines to match, complete with group indicators
         name #n is a string for the title of column n
         group #n is the number of the group in the regex

e.g.

topify '/^I heard ([A-Z]) from unit ([1-9]) and it said "(.*)" ([-0-9]*)$/' Unit 2 Status 1 Message 3 RetVal 4

This would display in an interactive way, such that the columns could be selected/reordered, etc.:

Unit Status Message            Retval
1    C      Waiting for input. 33
2    A      Oh no!             42
3    B      Stopped.           -1

I understand the brittleness of it, but I'd be really surprised if it hadn't been built before and wanted to check before I go into building it. I also appreciate it isn't particularly complicated to write so maybe everyone has just implemented their own solution...

Has anyone seen a tool like this?

(Please excuse the tags I've used here. I know I may be pushing/breaking the rules of some tags, but this is very generalised. Suggestions welcome.)

Is there an application that can link all the mail processing steps together from connection to delivery in a way that shows the whole process and the decision live?

I know it's possible to tail -f /var/log/mail.log, but when there becomes a large amount of connections and internal steps (client authorisation, recipient authorisation, relays, bounces, etc.) this log can become quite time-consuming and prone to missing steps.

I'm thinking something a bit like "top" but for email.

e.g.

1. connect from 1.2.3.4
2. client denied/accepted, mail from, to
3. recipient address verified/bounced
4. policy checked, antispam, antivirus.
5. relayed/delivered to...

This could then be shown in a top-like display like:

CONNECT               FROM           TO               POLICY     DELIVERY RESULT
example.com [1.2.3.4] [email protected] [email protected]  OK         Maildir  2.5.0 OK
spam.com [1.2.4.5]    [email protected]   [email protected]  SBL Fail   Reject   5.7.1 Deny
1.2.4.6                                               NO_FQDN    Reject   4.7.1 Retry

(Excuse the oversimplification)

Additionally, this could be used to build/store statistics on mail acceptance/rejection which could, in turn, be used to spot accidental misconfiguration. Has anyone seen anything like this?

I have two ADSL modem-routers and a server all in the same statically-assigned IP address range (192.168.0.1/24).

Internet 1 -- (1.1.1.1) Modem 1 (192.168.0.1) -- Switch -- (192.168.0.3) Server 
Internet 2 -- (2.2.2.2) Modem 2 (192.168.0.2) -----/

Each of the modems forward ports to the server, e.g. ssh. This works on one modem but not the other. If I do a packet trace, the ssh packets arrive at the server and are returned via the default gateway, which may have a different external IP to the origin. If it does not match the origin, the response is thrown away and the ssh connection times out.

e.g. If the default gateway in the server is 192.168.0.1, then the ssh packets would take the following paths:

Request:  SSH to 1.1.1.1 -> 192.168.0.1 -> 192.168.0.3
Response: 192.168.0.3 -> 192.168.0.1 -> 1.1.1.1
Result: WORKS! :-D

Request:  SSH to 2.2.2.2 -> 192.168.0.2 -> 192.168.0.3
Response: 192.168.0.3 -> 192.168.0.1 -> 1.1.1.1
Result: WRONG RESPONSE IP (2.2.2.2 != 1.1.1.1)

I understand from chatting with the people on IRC ##networking that what I want is "Source-Based Routing", a type of Policy Based Routing.

From what I can tell, a PBR looks something like:

access-list 1 permit 192.168.0.1
access-list 2 permit 192.168.0.2
!
interface async 1
 ip policy route-map equal-access
!
route-map equal-access permit 10
 match ip address 1
 set ip default next-hop 192.168.0.1
route-map equal-access permit 20
 match ip address 2
 set ip default next-hop 192.168.0.2
route-map equal-access permit 30
 set default interface null0

I have spent many hours looking at tutorials and examples on this, but they don't seem to come near my needs. Specifically, I can't seem to understand:

1. How the originating IP is matched to the response IP,
2. The meaning of 'async' in the above example,
3. If the above example is even remotely correct for my needs, and
4. Where I should put this configuration on a standard Ubuntu Server?

I have a number of services in init.d that are simple one-line commands wrapped in a script that repeats every few seconds. The template for them looks like this:

e.g.

#!/bin/sh
autostart() {
    while true
    do
            $@ ; sleep 20
    done
}

case "$1" in
    start)
            echo " started"
            autostart /path/to/checksomethingquickly
            ;;
    stop)
            me=`basename $0`
            killall checksomethingquickly $me
            ;;
esac

## End of File

The crux of this "service" is to call checksomethingquickly every 20 seconds. (Hence it's too quick for cron.) We want it to start when told to, and kill any/all running instances when it's told to stop.

It looks to me like it could be a "Task Job" in upstart but I'm unclear on how to ensure that the Task is repeated indefinitely and separated by 20 seconds. Ensuring it's serialised would be a nice improvement, too.

Is this the kind of script that can be turned into a upstart script? If so, how?

Is there an sshd config command to block keys that have blank or empty passphrases in them?

I'd like to limit ssh access to people with passwords without having passwords stored on servers.