tylerl's questions

Glue records are typically unavailable if a domain and its nameserver don't share a TLD, and technically aren't required if they don't share the same second-level domain, which could lead to extra steps to resolve a domain. The resolver must first look up the address of the nameserver before it can find the address for your domain. But theoretically you could add more steps in there than just those two.

The question here is, how long is that chain allowed to be?

if xyz.com uses nameserver ns1.xyz.info,
and xyz.info uses nameserver ns1.xyz.co,
and xyz.co uses nameserver ns1.xyz.cc,
and xyz.cc uses nameserver ns1.xyz.co.uk, ... and so forth

... you could end up with a very long chain for the resolver to untangle before it can resolve the name you originally wanted.

Presumably there's a practical limit -- BIND should only be willing to traverse so many links, otherwise there's the potential for a denial-of-service. But is there an official limit? Some number of steps beyond which the resolver is officially not required to proceed?

Given a kernel stack trace as below, how do you determine the specific line of code where the issue occurred?

kernel:  [<ffffffff80009a14>] __link_path_walk+0x173/0xfb9
kernel:  [<ffffffff8002cbec>] mntput_no_expire+0x19/0x89
kernel:  [<ffffffff8000eb94>] link_path_walk+0xa6/0xb2
kernel:  [<ffffffff80063c4f>] __mutex_lock_slowpath+0x60/0x9b
kernel:  [<ffffffff800238de>] __path_lookup_intent_open+0x56/0x97
kernel:  [<ffffffff80063c99>] .text.lock.mutex+0xf/0x14
kernel:  [<ffffffff8001b222>] open_namei+0xea/0x712
kernel:  [<ffffffff8006723e>] do_page_fault+0x4fe/0x874
kernel:  [<ffffffff80027660>] do_filp_open+0x1c/0x38
kernel:  [<ffffffff8001a061>] do_sys_open+0x44/0xbe
kernel:  [<ffffffff8005d28d>] tracesys+0xd5/0xe0

While I have no trouble finding the function call -- but translating __link_path_walk plus an offset to an actual line number is the difficult part.

Assuming this is for a standard distribution-provided kernel for which I know the exact version and build number, what's the process for fetching the necessary metadata and doing the corresponding lookup?

Btrfs is still in heavy development and still considered "unstable" by Chris Mason, and many important features are still being added, but the dire warnings of data loss are calamity are long since gone, and it has already become the default filesystem for many distributions, and several distributions have declared it "stable" for their purposes.

And while some risk most certainly remains, there's also some risk inherent in disk storage that is mitigated, such that btrfs has been shown to detect and correct data corruption issues that even high-end RAID cards will miss.

So you can expect that there is some point at which Btrfs, even in its in-development status, will be safer for data than traditional "dumb" filesystems like ext4, because the data-preservation features will outweigh any risk of bug-induced data corruption.

So where is that point? Have we passed it already? Or is there some known fault in Btrfs that should be fixed before we trust it?

Or perhaps do you just wait for enough other people to trust it first?

I get this question all the time from people who run Windows-based web servers. FTP, while ubiquitous, has numerous flaws shortcomings, not all of which can be solved simply by slapping SSL/TLS on top of the existing FTP protocol.

So is there a protocol, configuration, or other solution that a Windows user can employ for general-public file-transfer use to take the place of FTP? The typical requirements are simple and intuitive:

• It needs to be something that a relatively unskilled admin can set up and manage
• Access to this protocol needs to not implicitly grant access to other resources as well
• It needs to be safe over the Internet

Here are some no-go solutions which do not satisfy the question's requirements, but are typically suggested anyway. They're included here to urge you to avoid suggesting them as answers

• Cygwin and other SSH solutions, which besides being complicated for a novice to set up, also implicitly grant command-shell access to the visitor allowing him to do more than just upload and download files (i.e. execute arbitrary code)
• RDP, which while it has file-sharing capability, also grants the visitor desktop access, allowing him to do more than just upload and download files
• FTPS, FTPES, and other FTP-based solutions. Encryption isn't the only problem with FTP. Otherwise we wouldn't be asking this question.
• Windows File Sharing, which is not safe for use over the Internet, and is typically blocked by ISPs for that reason
• DAV, unless it can be VERY easily set up and secured (which is typically not the case)

^{Note that this is very explictly and very intentionally not a shopping question. We're not asking which popular product best satisfies a laundry-list of features. Instead, we're looking for potential solutions for a common problem.}

I have two identical drives that I intend to mirror in in the interest of data safety. These are data-only drives, not a primary OS drive.

In such a system, is it better to create single partition (Linux raid auto: type 0xfd) on each drive and raid together the partition from each drive (e.g. /dev/sdb1 and /dev/sdc1)? Or should I instead create a mirrored array of the unpartitioned drives directly (e.g. /dev/sdb and /dev/sdc)?

Ultimately I intend to create an LVM container from the resultant array for storing the actual data. Are there any considerations that might make one or the other option safer or more desireable down the road?

I'm looking at the differences between using a file versus a partition to store a virtual disk image in VM use. The common knowledge is that partition-based images are faster than file-based images because of a decreased overhead. It makes sense, but I've never seen any actual numbers.

My own testing bears out a different result. When I benchmark a direct-to-partition virtual disk, then format that same partition with ext4, create a virtual disk image stored on that ext4 filesystem, and then benchmark that, I see no speedup at all for the direct-to-partition virtual disk. Instead on some systems the file-based image is even faster (possibly due to host OS caching or something like that). This test was repeated many times on many systems, with fairly consistent results.

So perhaps throwing out the performance justification, is it still considered better to use a partition rather than a virtual disk image? Is there some other reason why direct partition access is better than image files?

Or perhaps is there some reason to go the other way around? Perhaps an advantage in one of the virtual disk file formats that you don't get with raw partition images?

I noticed the e attribute on several files/directories on Linux machines installed on ext4 filesystems.

[[email protected] ~]$ lsattr -d /bin
-------------e- /bin

According to chattr(1):

The ’e’ attribute indicates that the file is using extents for mapping the blocks on disk. It may not be removed using chattr(1).

In what way is this different, and more importantly, in what way is this detail significant -- specifically why is this detail important enough to be reported as a file attribute? Under what circumstances should I ever change my behavior based on the knowledge that this file "is using extents for mapping the blocks on disk"? Presumably this is something I need to know, otherwise it wouldn't be made so obvious, right?

I started seeing this odd sort of effect that resembles denial-of-service attack against a Linux server. The effect is that the network becomes at least partially unusable very much the same as what you see with a traditional DOS or DDOS attack.

Here's a trimmed netstat dump from during the "attack" (assuming that's what it is):

Proto Recv-Q Send-Q Local Address  Foreign Address       State       PID
tcp        1      0 1.2.3.1:80     50.128.251.184:1768   CLOSE_WAIT  18482/httpd         
tcp        0      1 1.2.3.4:80     71.75.22.31:52323     LAST_ACK    -                   
tcp        0  18980 1.2.3.4:80     98.180.31.210:60499   ESTABLISHED 18016/nginx: worker 
tcp        0  11709 1.2.3.4:80     98.180.31.210:60498   ESTABLISHED 18016/nginx: worker 
tcp        0  55743 1.2.3.4:80     71.75.22.31:52239     LAST_ACK    -                   
tcp        0      0 1.2.3.5:80     75.190.139.103:58265  ESTABLISHED 16808/httpd         
tcp        0  32814 1.2.3.4:80     71.75.22.31:52279     LAST_ACK    -                   
tcp        0  48029 1.2.3.4:80     71.75.22.31:52284     LAST_ACK    -                   
tcp        1  33581 1.2.3.4:80     71.75.22.31:52285     LAST_ACK    -                   
tcp        0  23582 1.2.3.4:80     71.75.22.31:52283     LAST_ACK    -                   
tcp        0    684 1.2.3.5:80     123.125.71.31:57865   FIN_WAIT1   -                   
tcp        0  37621 1.2.3.4:80     71.75.22.31:52218     LAST_ACK    -                   
tcp        0  18980 1.2.3.4:80     174.106.209.104:39937 ESTABLISHED 18016/nginx: worker 
tcp        0      0 1.2.3.1:80     95.140.125.125:60078  ESTABLISHED 18377/httpd         
tcp        0      0 1.2.3.2:39509  2.2.3.1:3306          ESTABLISHED 18379/httpd         
tcp        0    174 1.2.3.2:33029  2.2.3.1:3306          ESTABLISHED 18482/httpd         
tcp        0  44538 1.2.3.4:80     72.230.205.217:58271  FIN_WAIT1   -                   
tcp        0  64812 1.2.3.2:80     184.35.67.238:49173   ESTABLISHED 1251/httpd          
tcp        1      0 1.2.3.1:80     174.96.155.77:59167   CLOSE_WAIT  18379/httpd         
tcp        0      1 1.2.3.4:80     174.110.137.71:61496  FIN_WAIT1   -                   
tcp        1  31751 1.2.3.4:80     99.25.112.12:55747    CLOSING     -                   
tcp        0  33396 1.2.3.4:80     99.25.112.12:55764    ESTABLISHED 18016/nginx: worker 

Notice primarily the high usage of Send-Q buffer space by connections that are essentially closed, or partially closed. By keeping these connections open, it appears that an attacker can burn though the allowable send queue and bring traffic largely to a halt. This doesn't appear to be a sophisticated attack, but just a few attackers can apparently bring down a server with minimal traffic.

Does anyone recognize this attack pattern and know how to counter it?

I'm setting up a server with and SSD as the primary drive, and would like to duplicate the data onto a cheaper drive as a block-level backup (à la RAID1).

Since I don't want to hobble the performance of the SSD I was considering setting up DRBD with asynchronous mirroring so that changes get queued up for the spindle drive and eventually synced up once IO permits.

But I'd like to know if anyone has tries something like this and if there are any "gotchas" to pay attention to. Alternately, if there's a better technology to solve the same problem, please share.

When examining a "long" directory listing of a /proc/<pid>/ directory under Linux, I've noticed that the pseudo-file entries within the directory have sometimes different times associated with them.

From what I know of how the internals work, there aren't any "arbitrary" or "unpredictable" attributes in the structure, everything is deterministic, everything (including the file times) has to be determined somewhere based on some criteria; whether or not that criteria is useful is the important question.

For example, the timestamp on the directory itself (/proc/<pid>/) seems to reflect the start time of the corresponding process (process ID, at least -- it is not updated by a call to exec), while various entries such as proc/<pid>/fd/ or /proc/<pid>/cwd might be a different value. My best guess is that many of these cache the time that information was first retrieved, but it's difficult to tell.

Is there any useful knowledge to be gained here?

I've come across a situation where a client needs to blacklist a set of just under 1 million individual IP addresses (no subnets), and network performance is a concern. While I would conjecture that IPTables rules would have less of a performance impact than routes, that's just conjecture.

Does anyone have any solid evidence or other justification for favoring either IPTables or null routing as solution for blacklisting long lists of IP addresses? In this case everything is automated, so ease-of-use isn't really a concern.

EDIT 26-Nov-11

After some testing and development, it appears that none of these options are workable. It appears that both route lookups and iptables do linear searches through the ruleset, and take simply too long to process this many rules. On modern hardware, putting 1M items in an iptables blacklist slows the server down to about 2 dozen packets per second. So IPTables and null routes are out.

ipset, as recommended by Jimmy Hedman, would be great, except that it doesn't allow you to track more than 65536 addresses in a set, so I can't even try to use it unless someone has any ideas.

Apparently the only solution for blocking this many IPs is doing an indexed lookup in the application layer. Is that not so?

More Information:

The usage case in this instance is blocking a "known offenders" list of IP addresses from accessing static content on a web server. FWIW, doing blocking through Apache's Deny from is equally slow (if not more so) as it also does a linear scan.

FYI: Final working solution was to use apache's mod_rewrite in conjunction with a berkeley DB map to do lookups against the blacklist. The indexed nature of berkeley DBs allowed the list to scale with O(log N) performance.

We have a policy of logging all SSH session activity for CYA purposes with our clients. With PuTTY under windows this is trivial, as we can log each session to a file using PuTTY's built-in logging facility. But when working from Linux workstations, it's not quite as simple.

Note that server-side logging, while important for other reasons, is not part of what we're doing here. These session logs are for our own reference alone.

So far we've looked at the following options:

• Logging of ALL terminal activity using script triggered either in .bash_profile or as wrapper to the login shell (far from ideal)
• Replacing /usr/bin/ssh with a wrapper that starts a per-session log (better, but might have side-effects)
• Forcing all employees to use PuTTY for Linux instead of ssh from the command line (also very much not ideal)

Logging needs to be totally automatic, and ideally we want to be able to separate out different sessions into individual files.

Related:

• BASH shell session logging on remote host

I would like to show a listing of a single process and its current children. So, given the following process tree:

Imagine the following process listing:

  PID TTY      STAT   TIME COMMAND
    2 ?        S      0:00 [kthreadd]
    3 ?        S      0:06  \_ [ksoftirqd/0]  
  ...snip...
 1292 ?        Ss     0:06 /usr/sbin/gpm -m /dev/input/mice -t exps2
 1426 ?        Ss     0:00 /usr/lib/postfix/master
 9785 ?        S      0:00  \_ qmgr -l -t fifo -u
12301 ?        S      0:00  \_ pickup -l -t fifo -u -c
 1545 ?        Ss     0:05 /usr/sbin/apache2 -k start
 1570 ?        S      0:00  \_ /usr/sbin/apache2 -k start  
  ...snip...

I would like to instead just show process 1426 and its children. Like this:

  PID TTY      STAT   TIME COMMAND
 1426 ?        Ss     0:00 /usr/lib/postfix/master
 9785 ?        S      0:00  \_ qmgr -l -t fifo -u
12301 ?        S      0:00  \_ pickup -l -t fifo -u -c

Is there a simple way to do this?

When a server won't boot but you still have a functioning hard drive, you still should be able to extract the stored SSL certificates and keys, right? The information is all there after all.

The system certificates are stored in the registry here:
HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\SystemCertificates\My\Certificates

And the associated private keys are stored here:
HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\SystemCertificates\My\Keys\

But simply importing those keys into the live registry results in certificates with keys that don't work. How do I extract an SSL certificate off a dead drive?

Many well-known backup utilities for Windows use the Volume Shadow Copy Service built in to newer versions of windows to create a point-in-time snapshot of the disk in order to safely back up open files.

But how well does this mechanism play with SQL Server? I know that an SQL Server VSS Writer exists to help backup systems interoperate with SQL Server, but not all backup systems know about SQL Server and take advantage the backup-related API that it provides.

So my question is, what about the programs that don't know anything about SQL Server?

They presumably get a point-in-time snapshot of the MDF and LDF files as they existed at one single instant. And presumably the SQL Server data format is robust enough that such a snapshot would be guaranteed to be consistent and usable.

Is that the way things work, or am I missing something? Are there "gotchas" about using such a backup mechanism, or would I be safe?

I noticed that a newly imported database on SQLServer 2005 had configured itself (without my knowledge) to perform daily backups; but it's not deleting old files and quickly filling up the disk.

I don't know how the backup job got configured (maybe that's something that gets transferred when you move a database?) but I'm having trouble modifying it.

The backup runs as part of SQL Server Agent job called "Daily Backups". This job runs a package called "(SSIS Packages)\Maintenance Plans\Backup Plan" -- which I can't find. The "Management\Maintenance Plans" area for my server is empty.

I imagine I could delete the existing plan and re-create it manually, but I was hoping to just modify what was already there, since all that's missing is deleting old files.

I'm trying to set up squid as a load-balancing reverse-proxy, and I'm having a bit of trouble with the SSL. One of the site's applications checks to see if a site is loaded over SSL, and behaves differently if that is the case, so I need SSL connections to be proxied over SSL, and non-SSL connections to be proxied without it.

Right now, the squid server accepts connections over HTTP and HTTPS, but it only makes connections over HTTP to the upstream servers.

The interesting part of my squid configuration looks like this:

http_port 80 vhost vport
https_port 443 vhost vport  cert=/etc/cert.pem key=/etc/cert.key
cache_peer 10.0.0.10 parent 80 0 originserver round-robin
cache_peer 10.0.0.11 parent 80 0 originserver round-robin

Is there a simple way to tell it to proxy the SSL connections upstream over SSL, while leaving the others alone?

The "load average" on a *nix machine is the "average length of the run queue", or in other words, the average number of processes that are doing something (or waiting to do something). While the concept is simple enough to understand, troubleshooting the problem can be less straight-forward.

Here's the statistics on a server I worked on today that made me wonder the best way to fix this sort of thing. Here's the statistics:

• 1GB RAM free, 0 swap space usage
• CPU times around 20% user, 30% wait, 50% idle (according to top)
• About 2 to 3 processes in either "R" or "D" state at a time (tested using ps | grep)
• Server logs free of any error messages indicating hardware problems
• Load average around 25.0 (for all 3 averages)
• Server visibly unresponsive for users

I eventually "fixed" the problem by restarting MySQLd... which doesn't make a lot of sense, because according to mysql's "show processlist" command, the server was theoretically idle.

What other tools/metrics should I have used to help diagnose this issue and possibly determine what was causing the server load to run so high?

I want to proxy requests from an SSL site via a non-SSL site. My Apache httpd.conf looks like this:

<VirtualHost 1.2.3.4:80>
    ServerName foo.com
    ProxyPass / https://bar.com/
</VirtualHost>

So, when I visit http://foo.com, I expect apache to make a request to https://bar.com and send me the the page it fetched.

Instead, I get a 500 error, and in the error log, I see:

[error] proxy: HTTPS: failed to enable ssl support for 4.3.2.1:443 (bar.com)

Presumably I'm missing a directive here. Which might it be?

Never mind the security implications. I fully understand the risks.

I would like to automatically run my own command any time a drive is inserted under Linux (including USB drives, for example). Obviously the simplest solution would be to regularly poll and check for changes to /proc/partitions or to /dev/disk, but I was hoping for something a little more elegant than that.

EDIT

I know this is possible, because GNOME does this very thing; any time a new drive is inserted, GNOME runs it's own "autoplay" mechanism to take some action. ("Would you like to view the pictures on this SD card?")

I can come up with two uses for this: the first is in an Amazon EC2 cluster I manage, to detect and initialize newly attached EBS storage volumes automatically.

The other usage is on a Linux-based home file-server device; when I insert an SD card into the device, it should mount the card, move the pictures from the card onto the internal hard drive, unmount the card, and then beep to let me know it's done.