I saw a curious thing in IDS logs recently. It was a report of an outgoing connection that showed the parent process as /usr/sbin/sshd -R.
I am aware of the -R argument to ssh (client) for remote port forwarding, but have never seen a -R option to the sshd daemon.
No -R appears in man sshd. What does this argument do?
I recently learned about FreeBSD's accept_filter socket option which can allow a worker process to avoid context switching by, for example, waiting until a full HTTP request is received with accf_http:
This is a filter to be placed on a socket that will be using accept() to receive incoming HTTP connections.
It prevents the application from receiving the connected descriptor via accept() until either a full HTTP/1.0 or HTTP/1.1 HEAD or GET request has been buffered by the kernel.
If something other than a HTTP/1.0 or HTTP/1.1 HEAD or GET request is received the kernel will allow the application to receive the connection descriptor via accept().
The utility of accf_http is such that a server will not have to context switch several times before performing the initial parsing of the request. This effectively reduces the amount of required CPU utilization to handle incoming requests by keeping active processes in preforking servers such as Apache low and reducing the size of the file descriptor set that needs to be managed by interfaces such as select(), poll() or kevent() based servers.
My gut feeling is that on modern hardware, serving traffic to clients on high-speed connections (cable modem/DSL speeds or better) this may be a micro-optimization. Given that accf_http can't be used for HTTPS or HTTP/2 connections and accf_data just waits for the first byte, I don't see much advantage here. Maybe they'd save one or two context switches?
Are there any recent (post-2015 maybe?) benchmarks about how much FreeBSD's accept_filter can actually improve performance or throughput/concurrency?
In VMWare ESXi / vSphere, you can mark a disk or LUN as "Flash", indicating that the disk is an SSD (or the LUN is SSD/flash storage backed)
When you do, you get the following warning:
Marking HDD disks as flash disks could deteriorate the performance of datastores and services that use them. Mark disks as flash disks only if you are certain that those disks are flash disks.
This warning makes me curious: what does marking a LUN as flash actually do? How does it change VMware's behavior / performance? How could it deteriorate performance, I.E. what technically changes at a low-level?
I have a ZFS zpool on linux under kernel 2.6.32-431.11.2.el6.x86_64 which has a single vdev. The vdev is a SAN device. I expanded the size of the SAN, and despite the zpool having autoexpand set to on, even after rebooting the machine, exporting/importing the pool, and using zpool online -e, I was unable to get the pool to expand. I am sure the vdev is larger because fdisk shows it has increased from 215GiB to 250 GiB. Here's a sample of what I did:
[root@timestandstill ~]# zpool list
NAME SIZE ALLOC FREE CAP DEDUP HEALTH ALTROOT
dfbackup 214G 207G 7.49G 96% 1.00x ONLINE -
[root@timestandstill ~]# zpool import -d /dev/disk/by-id/
pool: dfbackup
id: 12129781223864362535
state: ONLINE
action: The pool can be imported using its name or numeric identifier.
config:
dfbackup ONLINE
virtio-sbs-XLPH83 ONLINE
[root@timestandstill ~]# zpool import -d /dev/disk/by-id/ dfbackup
[root@timestandstill ~]# zpool list
NAME SIZE ALLOC FREE CAP DEDUP HEALTH ALTROOT
dfbackup 214G 207G 7.49G 96% 1.00x ONLINE -
venuebackup 248G 244G 3.87G 98% 1.00x ONLINE -
[root@timestandstill ~]# zpool get autoexpand dfbackup
NAME PROPERTY VALUE SOURCE
dfbackup autoexpand on local
[root@timestandstill ~]# zpool set autoexpand=off dfbackup
[root@timestandstill ~]# zpool set autoexpand=on dfbackup
[root@timestandstill ~]# zpool list
NAME SIZE ALLOC FREE CAP DEDUP HEALTH ALTROOT
dfbackup 214G 207G 7.49G 96% 1.00x ONLINE -
venuebackup 248G 244G 3.87G 98% 1.00x ONLINE -
[root@timestandstill ~]# zpool status -v dfbackup
pool: dfbackup
state: ONLINE
scan: none requested
config:
NAME STATE READ WRITE CKSUM
dfbackup ONLINE 0 0 0
virtio-sbs-XLPH83 ONLINE 0 0 0
errors: No known data errors
[root@timestandstill ~]# fdisk /dev/disk/by-id/virtio-sbs-XLPH83
WARNING: GPT (GUID Partition Table) detected on '/dev/disk/by-id/virtio-sbs-XLPH83'! The util fdisk doesn't support GPT. Use GNU Parted.
WARNING: DOS-compatible mode is deprecated. It's strongly recommended to
switch off the mode (command 'c') and change display units to
sectors (command 'u').
Command (m for help): p
Disk /dev/disk/by-id/virtio-sbs-XLPH83: 268.4 GB, 268435456000 bytes
256 heads, 63 sectors/track, 32507 cylinders
Units = cylinders of 16128 * 512 = 8257536 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x00000000
Device Boot Start End Blocks Id System
/dev/disk/by-id/virtio-sbs-XLPH83-part1 1 27957 225443839+ ee GPT
Command (m for help): q
[root@timestandstill ~]# zpool online -e dfbackup /dev/disk/by-id/virtio-sbs-XLPH83
[root@timestandstill ~]# zpool list
NAME SIZE ALLOC FREE CAP DEDUP HEALTH ALTROOT
dfbackup 214G 207G 7.49G 96% 1.00x ONLINE -
venuebackup 248G 244G 3.87G 98% 1.00x ONLINE -
[root@timestandstill ~]# zpool status -v dfbackup
pool: dfbackup
state: ONLINE
scan: none requested
config:
NAME STATE READ WRITE CKSUM
dfbackup ONLINE 0 0 0
virtio-sbs-XLPH83 ONLINE 0 0 0
errors: No known data errors
How can I expand this zpool?
I have read a lot of information about planning RAM requirements forZFS deduplication. I've just upgraded my file server's RAM to support some very limited dedupe on ZFS zvols which I cannot use snapshots and clones on (as they're zvols formatted as a different filesystem) yet will contain much duplicated data.
I want to make sure that the new RAM I added will support the limited deduplication I intend to be doing. In planning, my numbers look good but I want to be sure.
How can I tell the current size of the ZFS dedupe tables (DDTs) on my live system? I read this mailing list thread but I'm unclear on how they're getting to those numbers. (I can post the output of zdb tank if necessary but I'm looking for a generic answer which can help others)
I have two servers which run variants of FreeBSD: One is a pfSense router and one is a FreeNAS 8 server. Both these servers run SNMP, and I am collecting and graphing their information using a third Cacti server.
The SNMP data from both the pfSense server and the FreeNAS server does not include memory usage, CPU usage, nor Load Average.
The traffic graphs for the pfSense server look fine. The disk usage reports from the FreeNAS server look beautiful. I just don't get any data for memory usage, CPU usage, nor Load Average. I know both these servers should be capable of providing this data, because in the pfSense and freeNAS web admins I can view graphs. But I would prefer to have all the graphs in Cacti for ease of management.
How can I get my pfSense server to provide memory usage, CPU usage, and Load Average data via SNMP? How can I get my FreeNAS server to provide memory usage, CPU usage, and Load Average data via SNMP? I assume the same procedure will work for both servers.
I have a ZFS zpool with two disks in a mirror configuration, da0 and da1. da1 failed, and so I replaced it with da2 using
zpool replace BearCow da1 da2
This ran for a few hours, during which zpool status showed that the array was being resilvered. When that finished, zpool status showed that the resilver was completed, but the array was still degraded...
I tried a zpool scrub and a zpool clear, but the array still shows as degraded:
[root@chef] ~# zpool status BearCow
pool: BearCow
state: DEGRADED
scrub: scrub completed after 0h20m with 0 errors on Tue Oct 9 16:13:27 2012
config:
NAME STATE READ WRITE CKSUM
BearCow DEGRADED 0 0 0
mirror DEGRADED 0 0 0
da0 ONLINE 0 0 0
replacing DEGRADED 0 0 0
da1 OFFLINE 0 0 0
da2 ONLINE 0 0 0
errors: No known data errors
I can't zpool replace BearCow da1 da2 anymore because da2 is already a member of BearCow...
This is FreeBSD (FreeNAS) running ZFS pool version 15.
How do I get my array to show as healthy again?
I have a zpool with 4 2TB USB disks in a raidz config:
[root@chef /mnt/Chef]# zpool status farcryz1
pool: farcryz1
state: ONLINE
scrub: none requested
config:
NAME STATE READ WRITE CKSUM
farcryz1 ONLINE 0 0 0
raidz1 ONLINE 0 0 0
da1 ONLINE 0 0 0
da2 ONLINE 0 0 0
da3 ONLINE 0 0 0
da4 ONLINE 0 0 0
In order to test the pool, I simulated a drive failure by pulling the USB cable from one of the drives without taking it offline:
[root@chef /mnt/Chef]# zpool status farcryz1
pool: farcryz1
state: ONLINE
status: One or more devices has experienced an unrecoverable error. An
attempt was made to correct the error. Applications are unaffected.
action: Determine if the device needs to be replaced, and clear the errors
using 'zpool clear' or replace the device with 'zpool replace'.
see: http://www.sun.com/msg/ZFS-8000-9P
scrub: none requested
config:
NAME STATE READ WRITE CKSUM
farcryz1 ONLINE 0 0 0
raidz1 ONLINE 0 0 0
da4 ONLINE 22 4 0
da3 ONLINE 0 0 0
da1 ONLINE 0 0 0
da2 ONLINE 0 0 0
errors: No known data errors
Data's still there, pool still online. Great! Now let's try to restore the pool. I plugged the drive back in, and issued the zpool replace command as I was instructed to above:
[root@chef /mnt/Chef]# zpool replace farcryz1 da4
invalid vdev specification
use '-f' to override the following errors:
/dev/da4 is part of active pool 'farcryz1'
Um.... That's not helpful... So I tried a zpool clear farcryz1, but that didn't help at all. I still couldn't replace da4. So I tried a combination of onlineing, offlineing, clearing, replaceing, and scrubing. Now I am stuck here:
[root@chef /mnt/Chef]# zpool status -v farcryz1
pool: farcryz1
state: DEGRADED
status: One or more devices could not be used because the label is missing or
invalid. Sufficient replicas exist for the pool to continue
functioning in a degraded state.
action: Replace the device using 'zpool replace'.
see: http://www.sun.com/msg/ZFS-8000-4J
scrub: scrub completed after 0h2m with 0 errors on Fri Sep 9 13:43:34 2011
config:
NAME STATE READ WRITE CKSUM
farcryz1 DEGRADED 0 0 0
raidz1 DEGRADED 0 0 0
da4 UNAVAIL 9 0 0 experienced I/O failures
da3 ONLINE 0 0 0
da1 ONLINE 0 0 0
da2 ONLINE 0 0 0
errors: No known data errors
[root@chef /mnt/Chef]# zpool replace farcryz1 da4
cannot replace da4 with da4: da4 is busy
How can I recover from this situation, where one device in my zpool was unexpectedly disconnected (but is not a failed device) and is now back again, ready to be resilvered?
EDIT: As requested, a tail of dmesg:
(ses3:umass-sim4:4:0:1): removing device entry
(da4:umass-sim4:4:0:0): removing device entry
ugen3.2: <Western Digital> at usbus3
umass4: <Western Digital My Book 1140, class 0/0, rev 3.00/10.03, addr 1> on usbus3
da4 at umass-sim4 bus 4 scbus6 target 0 lun 0
da4: <WD My Book 1140 1003> Fixed Direct Access SCSI-6 device
da4: 400.000MB/s transfers
da4: 1907697MB (3906963456 512 byte sectors: 255H 63S/T 243197C)
ses3 at umass-sim4 bus 4 scbus6 target 0 lun 1
ses3: <WD SES Device 1003> Fixed Enclosure Services SCSI-6 device
ses3: 400.000MB/s transfers
ses3: SCSI-3 SES Device
GEOM: da4: partition 1 does not start on a track boundary.
GEOM: da4: partition 1 does not end on a track boundary.
GEOM: da4: partition 1 does not start on a track boundary.
GEOM: da4: partition 1 does not end on a track boundary.
ugen3.2: <Western Digital> at usbus3 (disconnected)
umass4: at uhub3, port 1, addr 1 (disconnected)
(da4:umass-sim4:4:0:0): lost device
(da4:umass-sim4:4:0:0): removing device entry
(ses3:umass-sim4:4:0:1): lost device
(ses3:umass-sim4:4:0:1): removing device entry
ugen3.2: <Western Digital> at usbus3
umass4: <Western Digital My Book 1140, class 0/0, rev 3.00/10.03, addr 1> on usbus3
da4 at umass-sim4 bus 4 scbus6 target 0 lun 0
da4: <WD My Book 1140 1003> Fixed Direct Access SCSI-6 device
da4: 400.000MB/s transfers
da4: 1907697MB (3906963456 512 byte sectors: 255H 63S/T 243197C)
ses3 at umass-sim4 bus 4 scbus6 target 0 lun 1
ses3: <WD SES Device 1003> Fixed Enclosure Services SCSI-6 device
ses3: 400.000MB/s transfers
ses3: SCSI-3 SES Device
I have a server running VMware ESXi v4.1.0 348481. It has a hardware RAID10 and a SATA backup drive. I have a VM running which has it's primary boot vmdk on the RAID10 datastore, and a 600 GB vmdk on the SATA backup drive's datastore. The VM runs Debian linux with the FreeBSD kernel, and uses ZFS for the backup drive.
EDIT: The drive is not directly attached to the VM. It is used as a VMware Datastore, and the VM has a vmdk on the SATA drive's datastore. The datastore is not full (only 65% full)
I logged in to the server using SSH and found that last night backup was hung, and zfs list or zpool list both hung. So I opened the virtual console in ESXi and was sad to see:
(da1:mpt0:0:1:0): READ(10). CDC: 28 0 19 97 3a 50 0 0 2d 0
(da1:mpt0:0:1:0): CAM status: SCSI Status Error
(da1:mpt0:0:1:0): SCSI status: Check Condition
(da1:mpt0:0:1:0): SCSI sense: MEDIUM ERROR info:4862ec asc:11,4 (Unrecovered read error - auto reallocate failed)
(da1:mpt0:0:1:0): READ(10). CDC: 28 0 19 97 3a 50 0 0 2d 0
(da1:mpt0:0:1:0): CAM status: SCSI Status Error
(da1:mpt0:0:1:0): SCSI status: Check Condition
(da1:mpt0:0:1:0): SCSI sense: MEDIUM ERROR info:4862ec asc:11,4 (Unrecovered read error - auto reallocate failed)
I tried to reboot the VM and I received a message that the system was going down for reboot, and then that hung. (^C appears but does not kill shutdown). I cannot interrupt or kill -9 the zpool list zfs list or rsync processes -- Nothing happens when I try.
UPDATE: I just hard rebooted the VM. After it came back online, the backup zpool was online, however:
root@timestandstill:/home/jnet# zpool status -v
pool: backup
state: ONLINE
status: One or more devices has experienced an error resulting in data
corruption. Applications may be affected.
action: Restore the file in question if possible. Otherwise restore the
entire pool from backup.
see: http://www.sun.com/msg/ZFS-8000-8A
scrub: none requested
config:
NAME STATE READ WRITE CKSUM
backup ONLINE 0 0 0
da1 ONLINE 0 0 0
errors: Permanent errors have been detected in the following files:
/backups/someserver/home/someuser/public_html/somedir/calendar/someuser/calendars/somefile.ics
I am leaning heavily towards replacing the drive...
I have the following setup:
(internet) ---> [ pfSense Box ] /-> [ Apache / PHP server ]
[running HAproxy] --+--> [ Apache / PHP server ]
+--> [ Apache / PHP server ]
\-> [ Apache / PHP server ]
For HTTP requests this works great, requests are distributed to my Apache servers just fine. For SSL requests, I had HAproxy distributing the requests using TCP load balancing, and it worked however since HAproxy didn't act as a proxy, it didn't add the X-Forwarded-For HTTP header, and the Apache / PHP servers didn't know the client's real IP address.
So, I added stunnel in front of HAproxy, reading that stunnel could add the X-Forwarded-For HTTP header. However, the package which I could install into pfSense does not add this header... also, this apparently kills my ability to use KeepAlive requests, which I would really like to keep. But the biggest issue which killed that idea was that stunnel converted the HTTPS requests into plain HTTP requests, so PHP didn't know that SSL was enabled and tried to redirect to the SSL site.
How can I use HAproxy to load balance across a number of SSL servers, allowing those servers to both know the client's IP address and know that SSL is in use? And if possible, how can I do it on my pfSense server?
Or should I drop all this and just use nginx?
I have a Debian 6.0.1 machine running GNU/kFreeBSD 8.1-1-amd64 and Apache. I installed apache2-mpm-worker using apt-get, and my apache2.conf file contains:
<IfModule mpm_prefork_module>
StartServers 4
MinSpareServers 2
MaxSpareServers 5
MaxClients 150
MaxRequestsPerChild 0
</IfModule>
<IfModule mpm_worker_module>
StartServers 1
MinSpareThreads 2
MaxSpareThreads 20
ThreadLimit 64
ThreadsPerChild 20
MaxClients 100
MaxRequestsPerChild 400
</IfModule>
So I should see only a couple Apache processes running. However, it looks more like I am running prefork, because instead of many threads i appear to have many processes:
root@garrison:/# ps awwwwx | grep apache
78438 ? S+ 0:00 grep apache
78424 ? S 0:00 /usr/sbin/apache2 -k start
78423 ? S 0:00 /usr/sbin/apache2 -k start
78422 ? S 0:00 /usr/sbin/apache2 -k start
78421 ? S 0:00 /usr/sbin/apache2 -k start
78420 ? S 0:00 /usr/sbin/apache2 -k start
78419 ? S 0:00 /usr/sbin/apache2 -k start
78418 ? S 0:00 /usr/sbin/apache2 -k start
78417 ? S 0:00 /usr/sbin/apache2 -k start
78416 ? S 0:00 /usr/sbin/apache2 -k start
78415 ? S 0:00 /usr/sbin/apache2 -k start
78414 ? S 0:00 /usr/sbin/apache2 -k start
78413 ? S 0:00 /usr/sbin/apache2 -k start
78412 ? S 0:00 /usr/sbin/apache2 -k start
78411 ? S 0:00 /usr/sbin/apache2 -k start
78410 ? S 0:00 /usr/sbin/apache2 -k start
78409 ? S 0:00 /usr/sbin/apache2 -k start
78408 ? S 0:00 /usr/sbin/apache2 -k start
78407 ? S 0:00 /usr/sbin/apache2 -k start
78406 ? S 0:00 /usr/sbin/apache2 -k start
78405 ? S 0:00 /usr/sbin/apache2 -k start
78404 ? S 0:00 /usr/sbin/apache2 -k start
78402 ? S 0:00 /usr/sbin/apache2 -k start
78401 ? S 0:00 /usr/sbin/apache2 -k start
78400 ? S 0:00 /usr/sbin/apache2 -k start
78397 ? Ss 0:00 /usr/sbin/apache2 -k start
Yet Apache clearly has the worker module installed, not the prefork module:
root@garrison:/HostRAID/netboot/IntelNetboot.nbi# /usr/sbin/apache2 -l
Compiled in modules:
core.c
mod_log_config.c
mod_logio.c
worker.c
http_core.c
mod_so.c
What gives? Why do I have so many apache2 processes? This server is under no load. (That is, no http requests)
This should be dead simple but for the life of me, I can't get it working. I must be doing something stupid.
I have a PFsense server with a public IP address. Behind it is three LAN segments:
[ Internet ] <---> [ pfSense]
+----- 192.168.1.1/24 ---> (multiple servers)
+----- 192.168.2.1/24 ---> (multiple servers)
+----- 172.16.1.1/24 ---> (multiple machines)
I have a new server with address 192.168.1.31 behind the pfSense server, and I want to give it a public IP. I assumed I would do this with a 1:1 NAT, but no matter what I ahve tried, it's not working. Here's what I did:
192.168.1.31/32 as the internal subnet.192.168.1.31/32, port range HTTP - HTTP. I did the same for another port in the 4000s where I wanted to run SSH.After doing all of the above, I could not reach the new IP via HTTP, nor could I SSH into the new machine on the alternate SSH port I chose.
I changed the virtual IP to "Proxy ARP" and then to "Other", but neither of those worked either...
Oddly enough, I can SSH to the new IP on port 22, if I run the SSH server on 192.168.1.31:22. But I can't get to any other ports.
What am I doing wrong here?
I have an OpenSolaris server with a zpool backupz comprised of four SCSI drives:
-bash-3.2# zpool status backupz
pool: backupz
state: ONLINE
scrub: none requested
config:
NAME STATE READ WRITE CKSUM
backupz ONLINE 0 0 0
raidz1 ONLINE 0 0 0
c7t0d0 ONLINE 0 0 0
c7t1d0 ONLINE 0 0 0
c7t2d0 ONLINE 0 0 0
c7t3d0 ONLINE 0 0 0
errors: No known data errors
I want to add a fifth drive... but zpool add backupz raidz c7t4d0 isn't working...
-bash-3.2# zpool add backupz raidz c7t4d0
invalid vdev specification: raidz requires at least 2 devices
Can I not have a raidz config with 5 devices? Do I have to add two devices at once? or am I doing something incorrect altogether here?
I know how to create a command which a given user can execute via sudo. I have a given command I want to allow any user to be able to execute via sudo without entering a password. What would I put in my /etc/sudoers file in order to make that happen?
At the time of writing The IPv4 Depletion Site estimates there are 300 days remaining before all IPv4 addresses have been allocated. I've been following the depletion of IPv4 addresses for some time and realize the "crisis" has been going on for many years and IPv4 addresses have lasted longer than expected, however...
As the systems administrator for a small SaaS / website hosting company, what steps should I be taking to prepare for IPv6? We run a handful of CentOS and Ubuntu Linux systems on managed hardware in a remote datacenter. All our servers have IPv6 addresses but they appear to be link local addresses.
Our primary business function is website hosting on a proprietary website CMS system. One of my concerns is SSL certificates; at the moment every customer with an SSL certificate gets a dedicated IPv4 IP address. What else should I be concerned about / what action should I take to be prepared for IPv4 depletion?
I was just poking around in /usr/bin and I found an ELF binary file called [. /usr/bin/[. I have never heard of this file and my first thought was that it was a clever way of hiding a program, possibly a trojan. However it's present on all my CentOS servers and seems to have no manual entry. I can hazard a guess as to what it is but I was looking for a more authoritative answer...
What's the command under Linux to mark a file as "locked"/to prevent any changes from being made to it?
I'm not talking about chmod. There's a property that can be set (the name escapes me at the moment) which prevents even processes running as root from changing a file. What is this called and how do I set it?
I have forgotten what this is called and without remembering the name the search function on the site and on Google is failing me.
I have a nagios server running Ubuntu with a 2.0 GHz Intel Processor, a RAID10 array, and 400 MB of RAM. It monitors a total of 42 services across 8 hosts, most of which are checked using the check_http plugin even 5 minutes, some every minute. Recently the load on the nagios server has been above 4, often as high as 6. The server also runs cacti, gathering statistics every minute for 6 hosts.
I wonder, how many services should hardware like this be able to handle? Is the load so high because I am pushing the limits of the hardware, or should this hardware be able to handle 42 service checks plus cacti? If the hardware is inadequate, should I look to add more RAM, more cores, or faster cores? What hardware / service checks are others running?
Since I use the *nix command screen all day, and I couldn't find anyone starting this question, I figured it should be started. You know the drill: community wiki, one answer per features so we all can vote.
I don't know if this belongs on SF, SU or SO... let's try SF first.
I'm looking for a "live help" system for website which uses Jabber/XMPP as a backend... so that website visitors could click a "Chat Now" button, and using AJAX, chat with operators who are already set up using Jabber accounts. Something very much like J-Livesupport but without a jabber server (I already have one) and preferably OSS/free.
Openfire's Fastpath Webchat is perfect except you must use Openfire's Spark client. No dice there.
Before I build my own, anyone know of anything out there?