Dennis Nolte's questions

Environment:

• Linux X86 Server with Debian 9.3

• kernel: Debian 4.9.65-3+deb9u1

• smbclient: Version 4.5.12-Debian

• cifs-utils: 2:6.7-1

Situation:

Backupserver (backuppc) has to backup a Windows 10 Home client.

No domain

Alternative:

Cygwin/Rsync used for other Win7 Clients rejected because of manual work involved.

Prerequisites known already:

• DNS entry for Win10Client: done, tested and working

• DHCP entry for Win10Client: done, tested and working

• SMB1 is unsecure so do not go the "easy way".

The smbclient supports SMB2/3 protocol so i should not have to set the smb1 registry hack to enable SMB1 from e.g.

How to check which SMB Version is enabled on Windows Server 2008 R2

Problems: I cannot mount the administrative or any other share, nor do I see them with smbtree -b -N

What I have tried:

• I tried to use the non-administrative share (no $ at the end)

• I renamed the administrative share from C$ to mC$ or mC.

• Added a test share in documents.

• Added a test share in C:\test

None of those are visible via smbtree.

Directly connecting to the shares (or get a list of shares):

tested:

smbclient -U WIN10Username -L //CLIENT/
protocol negotiation failed: NT_STATUS_CONNECTION_RESET

smbclient -L //CLIENT/
protocol negotiation failed: NT_STATUS_CONNECTION_RESET

How do i get this working with either SMB2 or SMB3 protocol?

I have an ancient xen 4.0 host system with LVM and only one Volume Group Inside the VG there are a few logical volumes.

The Logical Volume i am talking about (slave) is a snapshot of another logical volume (origin master).

I did extend The "Slave" logical volume for a few hundred GB, however the guest debian linux is unable to see this increase.. fdisk still shows the same guest partition shema as before the increase.

The Guests (slave and therefor master) has 3 partitions on the Logical volume.. How would i increase the guest virtual harddrive when i can confirm the logical volume is larger.. (normally with 1 logical volume = 1 partition you would just resize the fs..) But how do i resize the partition table of the guest HDD when it is just the snapshot which i did increase?

I can extend or resize master or slave however i want, but i cannot create a new master or make the slave not beeing a snapshot anymore (due to size limitations)

Following situation:

A productive linux debian 7 server with kernel 3.2.0-4-amd64 #1 SMP Debian 3.2.68-1+deb7u2 x86_64 GNU/Linux

Manufacturer: Supermicro Product Name: X10SLL-F Version: 1.02

SATA controller: Intel Corporation Lynx Point 6-port SATA Controller 1 [AHCI mode] (rev 04)

2x SSD, 2x hdd

each drive can do Sata Rev3 (6.0Gb/s)

hdparm -I /dev/sd[a-d]|egrep "Model|speed|Transport"
    Model Number:       TOSHIBA THNSNH128GBST                   
    Transport:          Serial, ATA8-AST, SATA 1.0a, SATA II Extensions, SATA Rev 2.5, SATA Rev 2.6, SATA Rev 3.0
       *    Gen1 signaling speed (1.5Gb/s)
       *    Gen2 signaling speed (3.0Gb/s)
       *    Gen3 signaling speed (6.0Gb/s)
       *    SMART Command Transport (SCT) feature set
    Model Number:       TOSHIBA THNSNH128GBST                   
    Transport:          Serial, ATA8-AST, SATA 1.0a, SATA II Extensions, SATA Rev 2.5, SATA Rev 2.6, SATA Rev 3.0
       *    Gen1 signaling speed (1.5Gb/s)
       *    Gen2 signaling speed (3.0Gb/s)
       *    Gen3 signaling speed (6.0Gb/s)
       *    SMART Command Transport (SCT) feature set
    Model Number:       ST2000VX000-1CU164                      
    Transport:          Serial, SATA Rev 3.0
       *    Gen1 signaling speed (1.5Gb/s)
       *    Gen2 signaling speed (3.0Gb/s)
       *    Gen3 signaling speed (6.0Gb/s)
       *    SMART Command Transport (SCT) feature set
    Model Number:       ST2000VX000-1CU164                      
    Transport:          Serial, SATA Rev 3.0
       *    Gen1 signaling speed (1.5Gb/s)
       *    Gen2 signaling speed (3.0Gb/s)
       *    Gen3 signaling speed (6.0Gb/s)
       *    SMART Command Transport (SCT) feature set

The kernel messages suggest (to me at least) an issue with all 4 drives, which lead's me to believe it's the sata controller who might be at fault.

ata1: exception Emask 0x50 SAct 0x0 SErr 0x4090800 action 0xe frozen
ata1: irq_stat 0x00400040, connection status changed
ata1: SError: { HostInt PHYRdyChg 10B8B DevExch }
ata1: hard resetting link
ata2: exception Emask 0x50 SAct 0x0 SErr 0x4090800 action 0xe frozen
ata2: irq_stat 0x00400040, connection status changed
ata2: SError: { HostInt PHYRdyChg 10B8B DevExch }
ata2: hard resetting link
ata4: exception Emask 0x50 SAct 0x0 SErr 0x4090800 action 0xe frozen
ata4: irq_stat 0x00400040, connection status changed
ata4: SError: { HostInt PHYRdyChg 10B8B DevExch }
ata4: hard resetting link
ata3: exception Emask 0x50 SAct 0x0 SErr 0x4090800 action 0xe frozen
ata3: irq_stat 0x00400040, connection status changed
ata3: SError: { HostInt PHYRdyChg 10B8B DevExch }
ata3: hard resetting link
ata3: SATA link up 1.5 Gbps (SStatus 113 SControl 310)
ata2: SATA link up 1.5 Gbps (SStatus 113 SControl 310)
ata4: SATA link up 1.5 Gbps (SStatus 113 SControl 310)
ata1: SATA link up 1.5 Gbps (SStatus 113 SControl 310)
ata4.00: ACPI cmd f5/00:00:00:00:00:00 (SECURITY FREEZE LOCK) filtered out
ata4.00: ACPI cmd b1/c1:00:00:00:00:00 (DEVICE CONFIGURATION OVERLAY) filtered out
ata2.00: ACPI cmd f5/00:00:00:00:00:00 (SECURITY FREEZE LOCK) filtered out
ata2.00: ACPI cmd b1/c1:00:00:00:00:00 (DEVICE CONFIGURATION OVERLAY) filtered out
ata1.00: ACPI cmd f5/00:00:00:00:00:00 (SECURITY FREEZE LOCK) filtered out
ata1.00: ACPI cmd b1/c1:00:00:00:00:00 (DEVICE CONFIGURATION OVERLAY) filtered out
ata3.00: ACPI cmd f5/00:00:00:00:00:00 (SECURITY FREEZE LOCK) filtered out
ata3.00: ACPI cmd b1/c1:00:00:00:00:00 (DEVICE CONFIGURATION OVERLAY) filtered out
ata2.00: ACPI cmd f5/00:00:00:00:00:00 (SECURITY FREEZE LOCK) filtered out
ata2.00: ACPI cmd b1/c1:00:00:00:00:00 (DEVICE CONFIGURATION OVERLAY) filtered out
ata1.00: ACPI cmd f5/00:00:00:00:00:00 (SECURITY FREEZE LOCK) filtered out
ata1.00: ACPI cmd b1/c1:00:00:00:00:00 (DEVICE CONFIGURATION OVERLAY) filtered out
ata2.00: configured for UDMA/33
ata2: EH complete
ata1.00: configured for UDMA/33
ata1: EH complete
ata3.00: ACPI cmd f5/00:00:00:00:00:00 (SECURITY FREEZE LOCK) filtered out
ata3.00: ACPI cmd b1/c1:00:00:00:00:00 (DEVICE CONFIGURATION OVERLAY) filtered out
ata4.00: ACPI cmd f5/00:00:00:00:00:00 (SECURITY FREEZE LOCK) filtered out
ata4.00: ACPI cmd b1/c1:00:00:00:00:00 (DEVICE CONFIGURATION OVERLAY) filtered out
ata3.00: configured for UDMA/33
ata3: EH complete
ata4.00: configured for UDMA/33
ata4: EH complete

What i did already figure out (or believe to have figured out)

The commands SECURITY FREEZE LOCK and DEVICE CONFIGURATION OVERLAY are not important to the issue.

While reading about 20 bugreports and lot of documentations, a few linked some did suggest to disable NCQ, which i did .

First for one device, after waiting 1 day to check if the error repeats it happend again and i disabled it for all 4 devices

echo "1" >/sys/block/sdc/device/queue_depth

No obvious change in the situation.

https://ata.wiki.kernel.org/index.php/Libata_error_messages

https://wiki.archlinux.org/index.php/Solid_State_Drives#Resolving_NCQ_errors

Others suggest sata cable or even an incompatibility between board + drives.

However as i seem to either have the issue on one drive and this populates to all 4, or having the issue directly on all 4 devices i am unable to pinpoint the issue further.

As this is a production server putting this server down for maintenance (aka bios/kernel param changes) is possible, but i like to prevent that if possible.

According to the hoster this might be power management related:

https://bugzilla.kernel.org/show_bug.cgi?id=74961 https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1318218

echo "medium_power" >/sys/class/scsi_host/host0/link_power_management_policy 

Before the change this was set to max_performance.

This did not help either.

Smart Values of the HDDs/SDDs are OK, nothing too obvious.

Note that the UDMA Value seems to be 33 now only.

On boot of the server this were the sata link speed values:

[    3.161850] ata6: SATA link down (SStatus 0 SControl 300)
[    3.161867] ata2: SATA link up 6.0 Gbps (SStatus 133 SControl 300)
[    3.161882] ata4: SATA link up 3.0 Gbps (SStatus 123 SControl 300)
[    3.161894] ata1: SATA link up 6.0 Gbps (SStatus 133 SControl 300)
[    3.161907] ata5: SATA link down (SStatus 0 SControl 300)

The situation might happen on high load on the HDDs only, i did not test that yet as it would impact the server performance obviously.

There is no load on the SSDs, they are mounted but not used by any of the processes.

The RAM is ECC as far as i can tell.

dmidecode -t 17
# dmidecode 2.11
SMBIOS 2.7 present.

Handle 0x0023, DMI type 17, 34 bytes
Memory Device
    Array Handle: 0x0022
    Error Information Handle: Not Provided
    Total Width: 72 bits
    Data Width: 64 bits
    Size: 8192 MB
    Form Factor: DIMM
    Set: None
    Locator: P1-DIMMA1
    Bank Locator: P0_Node0_Channel0_Dimm0
    Type: DDR3
    Type Detail: Synchronous
    Speed: 1600 MHz
    Manufacturer: Samsung
    Serial Number: 373A6427
    Asset Tag: 9876543210
    Part Number: M391B1G73QH0-CK0  
    Rank: 2
    Configured Clock Speed: 1600 MHz

Please let me know if i can give additional informations as i lack the ideas what to do next.

We have a 4 core CPU production system which does a lot of cronjobs , having constant proc queue and an usual load of ~1.5.

During night time we do some IO intensive stuff with postgres. We generate a graph showing the load/memory usage (rrd-updates.sh) This "fails" sometimes on high IO load situations. It is happening nearly every night, but not on every high IO situation.

My "normal" solution would be to nice and ionice the postgres stuff and increase the prio of the graph generation. However this still fails. The graph generation is semi-thread-proof with flock. I do log the execution times and for the graph generation it is up to 5 min during high IO load, seemingly resulting in a missing graph for up to 4 min.
The timeframe is exactly matching with the postgres activity (this sometimes happens during the day aswell, though not that often) Ionicing up to realtime prio (C1 N6 graph_cron vs C2 N3 postgres), nicing way above the postgres (-5 graph_cron vs 10 postgres) did not solve the issue.

Assuming the data is not collected, the additional issue is the ionice/nice somehow still not working.
Even with 90% IOwait and a load of 100 i was still able to use the data generation command free without more than maybe 5 sec delay (on testing at least).

Sadly i have not been able to reproduce this exactly in testing (having only a virtualized dev system)

Versions:

Kernel 2.6.32-5-686-bigmem
Debian Squeeze rrdtool 1.4.3 Hardware: SAS 15K RPM HDD with LVM in hardware RAID1
mount options: ext3 with rw,errors=remount-ro
Scheduler: CFQ
crontab:

* * * * *               root    flock -n /var/lock/rrd-updates.sh nice -n-1 ionice -c1 -n7 /opt/bin/rrd-updates.sh

There seems to be a somhow possibly related BUG from Mr Oetiker on github for rrdcache:
https://github.com/oetiker/rrdtool-1.x/issues/326

This actually could be my issue (concurrent writes) but it does not explain the cronjob to not fail. In the asumption i actually have 2 concurrent writes flock -n would return exit code 1 (per man page ,confirmed in testing) As i do not get an email with the output either and the observation that the cronjob do actually run fine all the other time i am somehow lost.

Example output:

Based on the comment i added the important source of the update script.

rrdtool update /var/rrd/cpu.rrd $(vmstat 5 2 | tail -n 1 | awk '{print "N:"$14":"$13}')
rrdtool update /var/rrd/mem.rrd $(free | grep Mem: | awk '{print "N:"$2":"$3":"$4}')
rrdtool update /var/rrd/mem_bfcach.rrd $(free | grep buffers/cache: | awk '{print "N:"$3+$4":"$3":"$4}')

What do i miss or where can i check further?

Remember: Productive system so no dev, no stacktrace or similiar available or installable.