Steve Schnepp's questions

I'd like to automatically shape bandwidth-hungry TCP connections corresponding to the already transmitted data.

A summary would be : "The first Mbit is transferred at full bandwidth, then it gradually drops to 1% of the bandwidth when it reaches 10Mbit, and stays there".

An example is better than 1k words :

 b-w  ^
      |        .         .
100%  | ********         .
      |        . *       .
      |        .    *    .
      |        .       * .
  1%  |        .         .********
      +----------------------------> Data transfered.
         1Mbit .         . 10Mbit

I know that a full traffic shaping would be better, since it would allow bursting and use of leftover bandwidth, but the idea is specifically to automatically limit big data transfers without any further configuration.

How would I implement this on a linux host ?

Update: It isn't immediately obvious, but the data meter counts both ways (upload & download) as I purposefully didn't precise download or upload.

I use Apache 2.2 with mod_proxy to some 3rd party HTTP server.

As it doesn't send any Cache-Control header, I want to add one of 1 day for response code 200, 5 min for 404 and 0 for any other response code.

So I would something like :

SetEnvIf HttpResponseCode "200" rc_200
SetEnvIf HttpResponseCode "404" rc_404

Header set "Cache-Control" "private, max-age=0"
Header set "Cache-Control" "public, max-age=86400" env=rc_404
Header set "Cache-Control" "public, max-age=86400" env=rc_200

To compile something, I needed the zlib1g-dev package to be installed so I launched an apt-get install zlib1g-dev.

apt-get informed me nicely that the package was already auto-installed because of an other package, and that it understands that I want it installed explicitly now :

# apt-get install zlib1g-dev
zlib1g-dev is already the newest version.
zlib1g-dev set to manually installed.

My compilation done, I don't need it any more explicitly, so I want to revert its status to the previous one : auto-installed. This way it will be pruned automatically when it will not be needed any more with a simple apt-get autoremove.

I cannot do an apt-get remove zlib1g-dev since some packages still depends on it.

So how may I revert the package zlib1g-devinstallation state to auto-installed ?

I know that I might edit /var/lib/apt/extended_states by hand from

Package: zlib1g-dev
Auto-Installed: 0

to

Package: zlib1g-dev
Auto-Installed: 1

... but it just doesn't feel right.

SELECT INTO OUTFILE fails for a CIFS-mounted directory (/mnt/backup/test) but works for a normal directory (/tmp/tests).

mysql> select * from mytable into outfile '/tmp/tests/mytable.txt';
Query OK, 3723 rows affected (0.05 sec)

mysql> select * from mytable into outfile '/mnt/backup/test/mytable.txt';
ERROR 1 (HY000): Can't create/write to file '/mnt/backup/test/mytable.txt' (Errcode: 2)

The mysql user has no problem writing to /mnt/backup/test/mytable.txt :

root:~# su - -s /bin/bash mysql
mysql:~$ touch /mnt/backup/test/mytable.txt
mysql:~$ ls -l /mnt/backup/test/mytable.txt
-rw-r--r-- 1 root root    0 2009-11-16 10:48 /mnt/backup/test/mytable.txt

The CIFS directory /mnt/backup is mounted with : rw,noperm,user=****,password=****

Since the -T option of mysqldump makes use of it, what options (Mysql/CIFS) are needed for SELECT INTO OUTFILE to work with my CIFS-mounted directory ?

Update:

Interestingly, an strace of the mysqld process did indeed reveal a access control failure as Michael suggested. But I still don't know why. A simple Perl script launched as mysql calls the same syscall with no problem.

Extract of mysqld strace

open("/mnt/backup/tests/bugs.1.txt", O_WRONLY|O_CREAT|O_EXCL|O_LARGEFILE, 0666) 
        = -1 EACCES (Permission denied)

Extract of perl strace

open("/mnt/backup/tests/bugs.32064.txt", O_WRONLY|O_CREAT|O_EXCL|O_LARGEFILE, 0666) = 3

Note that the directory /mnt/backup/tests was emptied before each run.

I want to delegate SysV init scripts to each user.

Like the SysV init, each item in ${HOME}/rc.d starting with S will be launched on server start-up with the start argument. The same for the server shut-down with the one starting with K and with the stop argument.

I thought about scripting it myself, but maybe there is already some kind of implementation out there¹. In summary it would be a script in /etc/init.d/ that iterates through all the users and launches runparts as the user on the relevant scripts.

The platform here is a Linux (Debian flavour), but I think the solution would be quite portable among various Unix-like platforms.

Update:

The point here is for users to be able to create their own init scripts that should be launch on their behalf when the system boots up. As Dan Carley pointed out, the services won't be able to access any system asset (priviledged ports, system logs, ...).

_{1. This way I don't have to think that much about all the subtle security implications such as script timeouts for example...}

I want to do some performance monitoring on our JVMs.

What would be interesting to monitor besides the obvious things I already have on my list ?

• JVM Free/Total memory
• GC rate (how many per minute)
• Duration of GC

On a Windows Server, I have 2 kinds of Java processes

• java.exe
• javaw.exe

What are the differences ?

With mod_log you can log headers sent back to the client in the access.log file with the %{Set-Cookie}o directive. But if there are multiple occurences of the same header, as is authorized by the HTTP RPC, only one gets logged.

How can we have all of them logged ?

For reference, the RFC states :

Multiple message-header fields with the same field-name MAY be present in a message if and only if the entire field-value for that header field is defined as a comma-separated list [i.e., #(values)].

Set-Cookie is such a field-name since the field-value is comma-separated. I would not mind to have them joined together as suggested in the HTTP RPC.

When I used RAR, I had an option to add some extra recovery data in order to be able to cope with a not-perfect medium (think floppy disks). It saved my day several times, specially when dealing with old CDs (scratched or when the reflective layer was damaged with bubbles).

I also used Parchive to redistribute several CDs with a "parity" one in a manner not unlike RAID.

So, in the era of DVDs, external HDD and Flash storage :

1. Is it still worth it ?
2. Is it only available as a external package like Parchive or only with RAR since various better compression formats exists ?

I don't want a complete solution like the ones asked here, here or here, just ideally something as lightweight as gunzip in usage (stdin/stdout)

I created a sub-domain for static content to be able to serve it more efficiently by several load-balanced web servers. This static content is updated automatically, at a rate of ~1k files/day.

Right now I use rsync to update the servers in a master/slave way but since the content has a growing number of 100k+ files, it takes more and more time and puts an increasing I/O load on both the master and the slaves.

I cannot use the solution I proposed on the Improve rsync performance question since I cannot know which files are modified without stat-ing them all, and that wouldn't solve the increasing I/O cost. I also have to handle file deletion.

I thought about using something like a r/o NFS on the slaves but that could somewhat defeat the load-balancing effect and put a gratuitous SPOF.

Btw, the servers are running AIX, but I'm also interested in a solution in a more generic context.

In VMWare you can use multiple CPU, but only as much as present on the host. Is there a way to use more CPU than physically available ?

It would be to :

• test software designed to run on an expensive 16-way box on commodity 2-way boxes.
• Use an image designed for 2-way on a 1-way host
• test software on not-power-of-2-way (think 3) (Although this could eventually be done on a 4-way box)

In VMWare you can use multiple CPU, but only as much as present on the host. Is there a way to use more CPU than physically available ?

It would be to :

• test software designed to run on an expensive 16-way box on commodity 2-way boxes.
• Use an image designed for 2-way on a 1-way host
• test software on not-power-of-2-way (think 3) (Although this could eventually be done on a 4-way box)

Update:

I specifically want be able to run a 16-way configured OS on a 2-way hardware, not basic over-subscribing (like 16 2-way on a 4-way hardware) that is effectively handled by almost every virtualization software currently available.

I also don't really want to invest in a very specific solution if I don't have to since the idea is to limit the investements required.

And I cannot just increase the number of the threads as it has been cleverly suggested since :

• In one application, I would like to be able to test the self-regulation of process numbers.
• I want also to be able to see the impact of having a few number of big SMP (16+ way) hosts over a big number of small hosts (1 or 2 way).

Update(bis):

I finally found out that QEMU does exactly this.