cherouvim's questions

I have this fail2ban jail targeting an apache log (currently with some small values so I can experiment):

enabled = true
maxretry = 3
bantime = 10
findtime = 5
action = 429-ban

I stress test it using ab (with concurrency: 10, requests: 10000) and my custom filter catches the "high" traffic almost immediately and bans the IP.

The problem is that after 10 seconds, where the ban expires, if I continue running the ab, the fail2ban log is now filled with:

Ignore 192.168.XX.XX, expired bantime
Ignore 192.168.XX.XX, expired bantime
Ignore 192.168.XX.XX, expired bantime

From that point onward, and for a very long time, no ban occurs. At the same time fail2ban log reveals processing, even if I stop ab. If I wait long enough and confirm that the fail2ban log activity stops, then restarting the stress tests effectively bans the IP.

My questions are:

• What does this expired bantime exactly mean?
• It seems that fail2ban will keep on processing stuff even though stress testing with ab has stopped a long time ago. Is there a possibility that there is a buffer of some sort on fail2ban that I could reduce, which would probably also solve my first problem?

I have an Ubuntu 18 LTS host which I want to upgrade via ansible and force apt-get dist-upgrade to install any new configurations available overwriting any existing ones, as if I was there pressing the first option of that dialog:

I'm looking into the ansible apt_module documentation and cannot get this to work. In all cases my touched configurations are maintained, and the new vendor ones are stored in .ucf-dist files.

So far I've tried:

# 1:
ansible -i my-hosts foo -m apt -a 'update_cache=yes upgrade=safe dpkg_options=force-confnew'

# 2:
ansible -i my-hosts foo -m apt -a 'force_apt_get=yes update_cache=yes upgrade=safe dpkg_options=force-confnew'

# 3:
ansible -i my-hosts foo -m apt -a 'force_apt_get=yes update_cache=yes upgrade=dist dpkg_options=force-confnew'

FYI I reproduce this locally with VirtualBox snapshots where I revert after every test.

For SEO reasons I want to return "410 Gone" for some specific URLs which I will match using path_beg or path_sub.

I've tried the following:

frontend foo
    mode http
    bind :80

    # 1) this works but I cannot use 410
    http-request deny deny_status 408 if { path_sub bar }

    # 2) this doesn't work at all no matter the code
    http-response set-status 408 if { path_sub test }

I cannot use 410 in the first directive with http-request because https://cbonte.github.io/haproxy-dconv/1.8/configuration.html#errorfile:

<code> is the HTTP status code. Currently, HAProxy is capable of generating codes 200, 400, 403, 405, 408, 425, 429, 500, 502, 503, and 504.

The second directive with the http-response doesn't have any effect.

How can I return a 410?

I want to reduce, or mitigate the effects of, malicious layer 7 traffic (targeted attacks, generic evil automated crawling) which reaches my backend making it very slow and even unavailable. This regards load-based attacks as described in https://serverfault.com/a/531942/1816

Assume that:

1. I use a not very fast backend/CMS (e.g ~1500ms TTFB for every dynamically generated page). Optimizing this is not possible, or simply very expensive in terms of effort.
2. I've fully scaled up, i.e I'm on the fastest H/W possible.
3. I cannot scale out, i.e the CMS does not support master-to-master replication, so it's only served by a single node.
4. I use a CDN in front of the backend, powerful enough to handle any traffic, which caches responses for a long time (e.g 10 days). Cached responses (hits) are fast and do not touch my backend again. Misses will obviously reach my backend.
5. The IP of my backend is unknown to attackers/bots.
6. Some use cases, e.g POST requests or logged in users (small fraction of total site usage), are set to bypass the CDN's cache so they always end up hitting the backend.
7. Changing anything on the URL in a way that makes it new/unique to the CDN (e.g addition of a &_foo=1247895239) will always end up hitting the backend.
8. An attacker who has studied the system first will very easily find very slow use cases (e.g paginated pages to the 10.000th result) which they'll be able to abuse together with random parameters of #7 to bring the backend to its knees.
9. I cannot predict all known and valid URLs and legit parameters of my backend at a given time in order to somehow whitelist requests or sanitize the URL on the CDN in order to reduce unnecessary requests from reaching the backend. e.g /search?q=whatever and /search?foo=bar&q=whatever will 100% produce the same result because foo=bar is not something that my backend uses, but I cannot sanitize that on the CDN level.
10. Some attacks are from a single IP, others are from many IPs (e.g 2000 or more) which cannot be guessed or easily filtered out via IP ranges.
11. The CDN provider and the backend host provider both offer some sort of DDoS attack feature but the attacks which can bring my backend down are very small (e.g only 10 requests per second) and are never considered as DDoS attacks from these providers.
12. I do have monitoring in place and instantly get notified when the backend is stressed, but I don't want to be manually banning IPs because this is not viable (I may be sleeping, working on something else, on vacation or the attack may be from many different IPs).
13. I am hesitant to introduce a per-IP limit of connections per second on the backend since I will, at some point, end up denying access to legit users. e.g imagine a presentation/workshop about my service taking place in a university or large company where from tens or hundreds of browsers will almost simultaneously be using the service from a single IP address. If these are logged in, then they'll always reach my backend and not be served by the CDN. Another case is public sector users all accessing the service from very limited amount of IP addresses (provided by the government). So this would deny access to legit users and would not help at all to attacks from many IPs each of which only does a couple of requests.
14. I do not want to permanently blacklist certain large IP ranges of countries which sometimes are the origins of attacks (e.g China, eastern Europe) because this is unfair, wrong, will deny access to legit users from those areas and attacks from other places will not be affected.

So, what can I do to handle this situation? Is there a solution that I've not taken into consideration in my assumptions that could help?

I'm trying to figure out how to consistently turn off keepalive across various client machines which are issuing HTTP requests via curl.

This is my target server:

- Ubuntu 18.04.2 LTS
- 4.15.0-47-generic
- HA-Proxy version 1.8.19-1ppa1~bionic 2019/02/12

This is client 1 where from I'm issuing curl (vanilla installation):

- Ubuntu 16.04.3 LTS
- 4.4.0-62-generic
- curl 7.47.0 (x86_64-pc-linux-gnu) libcurl/7.47.0 GnuTLS/3.4.10 zlib/1.2.8 libidn/1.32 librtmp/2.3

This is client 2 where from I'm issuing curl (vanilla installation):

- Ubuntu 18.04 LTS
- 4.15.0-20-generic
- curl 7.58.0 (x86_64-pc-linux-gnu) libcurl/7.58.0 OpenSSL/1.1.0g zlib/1.2.11 libidn2/2.0.4 libpsl/0.19.1 (+libidn2/2.0.4) nghttp2/1.30.0 librtmp/2.3

To turn off keepalive I've tried using -H "Connection: close", --no-keepalive and --keepalive-time 1 and only the first option seems to work but only from client 1.

On client 1 (Ubuntu 16) the connection is not left open but from client 2 (Ubuntu 18) the connection is left open until it times out. I confirm that either by looking at the target server's watch -n 0.1 "netstat -na | fgrep CLIENT_IP_ADDRESS" or by using -vvv on both clients, which on client 1 is always * Closing connection 0 and on client 2 is always * Connection #0 to host www.example.com left intact.

The difference between client 1 and 2 are obviously the Ubuntu version and curl version. What is it that causes the the connection to be closed in client 1 but not in client 2?

I've also found out that if I change my target server to another machine running an old distro with an old apache httpd, whether I send out Connection: close or not doesn't make any difference and keepalive is always used from either of my 2 clients. So I suppose that the target server configuration also plays some role in that.

Edit: To complicate things even further, if from client 1 and 2 I issue requests to the target server via ab then the connection is killed instantly. That is expected because ab uses HTTP/1.0 and not HTTP/1.1. But then, if I target the old distro with apache then in both cases the connection remains open. Which means that, indeed, the receiving end plays a role as well.

Edit 2: I grabbed all /proc/sys/net/ipv4/ settings of both clients via:

for file in /proc/sys/net/ipv4/*
do
  echo "$file $(cat $file)"
done

and these can be found here:

• U16: https://pastebin.com/GcuuqhvA
• U18: https://pastebin.com/a2QWReFZ
• diff: https://pastebin.com/25Ccyz9h

The relevant part of my /etc/haproxy/haproxy.cfg is:

global
    maxconn 30000
    ...

defaults
    ...

frontend frontend_for_all_sites
    maxconn 22000
    mode http
    bind *:80

    acl acl_hostname_www hdr_dom(host)    www.example.com
    acl acl_hostname_static hdr_dom(host) static.example.com
    use_backend www_backend      if acl_hostname_www
    use_backend static_backend   if acl_hostname_static

backend www_backend
    server www 127.0.0.1:9090 maxconn 500

backend static_backend
    server s 127.0.0.1:8080 maxconn 5000

So, I have 2 backends with only 1 backend each, without load balancing, but just proxying requests to different backends based on the request's hostname.

On the www backend I've set maxconn 500 and on the static backend I've set it to 5000.

The stats page now looks like this:

My question is what does the "Backend Limit" of 2200 for both backends represent? I understand that this value is 10% of the 22000 maxxconn on frontend_for_all_sites. Does this 2200 mean that the maximum amount of connections that the backends will process is 2200 and above that haproxy will return a 503?

In /var/log/varnish/ I have some old varnish logs, produced by varnishlog which, due to logrotated, are now in gz format:

-rw-r--r--  1 varnishlog varnish 143068514 Aug 10 23:59 varnish.log.2017-08-10.gz
-rw-r--r--  1 varnishlog varnish 156373518 Aug 11 23:59 varnish.log.2017-08-11.gz
-rw-r--r--  1 varnishlog varnish 134255825 Aug 12 23:59 varnish.log.2017-08-12.gz
-rw-r--r--  1 varnishlog varnish 156992529 Aug 13 23:59 varnish.log.2017-08-13.gz
-rw-r--r--  1 varnishlog varnish 176751837 Aug 14 23:59 varnish.log.2017-08-14.gz
-rw-r--r--  1 varnishlog varnish 155948012 Aug 16 00:01 varnish.log.2017-08-15.gz
-rw-r--r--  1 varnishlog varnish 169977134 Aug 17 00:01 varnish.log.2017-08-16.gz

I've extracted those in another location and tried to view them via head/more but they look binary.

So I then tried opening them with varnishlog using either of the following parameters:

[-N filename]             VSM filename
[-r filename]             Binary file input

But that didn't work giving me:

Can't open log - retrying for 5 seconds

and:

Can't open log file (Not a VSL file:

Any idea on how can I inspect those historical logs from varnish?

p.s I use varnish-4.1.1

I usually work with Ubuntu LTS servers which from what I understand symlink /bin/sh to /bin/dash. A lot of other distros though symlink /bin/sh to /bin/bash.

From that I understand that if a script uses #!/bin/sh on top it may not run the same way on all servers?

Is there a suggested practice on which shell to use for scripts when one wants maximum portability of those scripts between servers?

I'm redirecting to some domain based on an acl condition but I want the user to end up on either http or https depending on the scheme/protocol of their request. I've managed to do that with the 2 lines shown bellow, but I'd like to convert them to just 1 line:

http-request redirect code 301 location http://www.example.com%[url] if acl_whatever !{ ssl_fc }
http-request redirect code 301 location https://www.example.com%[url] if acl_whatever { ssl_fc }

Ideally I'd like to redirect to something like %[scheme]://www.example.com%[url] but %[scheme] does not exist.

Is there a variable or function which can return http or https so I do not have to repeat the rule twice? Maybe by somehow setting a variable containing the scheme or maybe by using the value of %[ssl_fc] which is 0 or 1?

As shown in the image below, I have a HAProxy backend with 2 servers which have the following settings:

maxconn 64 check inter 5s fastinter 2s downinter 2s

My question regards the Queue column. How can I specify the limit of the backend queue? And why does the Backend at the bottom, show the value of 11 when api-1 and api-2 show 22 and 18 respectively? What does the number 11 represent in my case?

I have the following redirects in the <VirtualHost *:80>:

RewriteRule ^/old-url$  /new-url  [R=301,L]
RewriteRule ^/foo$      /bar      [R=301,L]
...

In front of apache I've got haproxy, listening on 80 and 443, the latter doing SSL termination and:

http-request set-header X-Forwarded-Port %[dst_port]
http-request set-header X-Forwarded-Proto https if { ssl_fc }

My problem with the redirects is that:

1. http://example.com/foo correctly redirects to http://example.com/bar
2. https://example.com/foo wrongly redirects to http://example.com/bar

How can I make sure http redirects to http and https to https? Note that I want to avoid having to write the redirect rules twice. How could I use the X-Forwarded-Proto or X-Forwarded-Port to redirect to the correct scheme?

I've configured the DNS settings for a host (admin.example-preprod.foobar.it with IP 100.100.100.100) but it does not resolve from every server I try (it does for some).

For example, I'm on a server which uses the following 3 DNS servers:

foobar@server:~$ cat /etc/resolv.conf
# Dynamic resolv.conf(5) file for glibc resolver(3) generated by resolvconf(8)
#     DO NOT EDIT THIS FILE BY HAND -- YOUR CHANGES WILL BE OVERWRITTEN
nameserver 200.200.200.1
nameserver 200.200.200.2
nameserver 200.200.200.3

dig works for all of them.

first:

foobar@server:~$ dig admin.example-preprod.foobar.it @200.200.200.1

; <<>> DiG 9.9.5-3ubuntu0.7-Ubuntu <<>> admin.example-preprod.foobar.it @200.200.200.1
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 6540
;; flags: qr aa rd; QUERY: 1, ANSWER: 1, AUTHORITY: 3, ADDITIONAL: 2
;; WARNING: recursion requested but not available

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 4096
;; QUESTION SECTION:
;admin.example-preprod.foobar.it.        IN      A

;; ANSWER SECTION:
admin.example-preprod.foobar.it. 86400 IN A      100.100.100.100

;; AUTHORITY SECTION:
foobar.it.               86400   IN      NS      qwert.foobar.it.
foobar.it.               86400   IN      NS      ns0.xname.org.
foobar.it.               86400   IN      NS      ns1.xname.org.

;; ADDITIONAL SECTION:
qwert.foobar.it.         86400   IN      A       200.200.200.1

;; Query time: 46 msec
;; SERVER: 200.200.200.1#53(200.200.200.1)
;; WHEN: Fri Nov 04 10:35:46 EET 2016
;; MSG SIZE  rcvd: 156

second:

foobar@server:~$ dig admin.example-preprod.foobar.it @200.200.200.2

; <<>> DiG 9.9.5-3ubuntu0.7-Ubuntu <<>> admin.example-preprod.foobar.it @200.200.200.2
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NXDOMAIN, id: 40127
;; flags: qr rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 1, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 4096
;; QUESTION SECTION:
;admin.example-preprod.foobar.it.        IN      A

;; AUTHORITY SECTION:
foobar.it.               85817   IN      SOA     qwert.foobar.it. webmaster.foobar.it. 2016092901 10800 900 1814400 10800

;; Query time: 3 msec
;; SERVER: 200.200.200.2#53(200.200.200.2)
;; WHEN: Fri Nov 04 10:36:05 EET 2016
;; MSG SIZE  rcvd: 111

third:

foobar@server:~$ dig admin.example-preprod.foobar.it @200.200.200.3

; <<>> DiG 9.9.5-3ubuntu0.7-Ubuntu <<>> admin.example-preprod.foobar.it @200.200.200.3
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 2392
;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 3, ADDITIONAL: 2

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 4096
;; QUESTION SECTION:
;admin.example-preprod.foobar.it.        IN      A

;; ANSWER SECTION:
admin.example-preprod.foobar.it. 86400 IN A      100.100.100.100

;; AUTHORITY SECTION:
foobar.it.               86400   IN      NS      qwert.foobar.it.
foobar.it.               86400   IN      NS      ns1.xname.org.
foobar.it.               86400   IN      NS      ns0.xname.org.

;; ADDITIONAL SECTION:
qwert.foobar.it.         81845   IN      A       200.200.200.1

;; Query time: 2309 msec
;; SERVER: 200.200.200.3#53(200.200.200.3)
;; WHEN: Fri Nov 04 10:35:56 EET 2016
;; MSG SIZE  rcvd: 156

but then, when I try to ping it gives me unknown host:

foobar@server:~$ ping admin.example-preprod.foobar.it
ping: unknown host admin.example-preprod.foobar.it

How is that possible and where should I look at to solve the problem?

I want to combine 2 different types of acl in haproxy (src and path_beg) to make a decision about which backend to use.

I tried the following which is not valid syntax:

acl my_ip src 192.168.0.50
acl api_path path_beg /api
use_backend private_backend if my_ip AND api_path
use_backend public_backend if api_path

The AND in the 3rd line does not work.

I've also tried the following which is not valid syntax:

acl my_ip_and_api_path src 192.168.0.50 path_beg /api

Using path_beg after the src and the IP doesn't work.

I have an openSUSE 13.1 VM (host runs Virtualbox 4.2.18, also on openSUSE 13.1) and restarting httpd (Apache/2.4.6) always takes 1.5 minute:

foobar:~ # time /etc/init.d/apache2 restart
redirecting to systemctl restart apache2.service

real    1m30.778s
user    0m0.004s
sys     0m0.000s

Immediately subsequent restart is normal (very fast):

foobar:~ # time /etc/init.d/apache2 restart
redirecting to systemctl restart apache2.service

real    0m1.023s
user    0m0.004s
sys     0m0.000s

5 minutes later the restart time goes again to exactly 90 seconds:

foobar:/tmp # time /etc/init.d/apache2 restart
redirecting to systemctl restart apache2.service

real    1m30.684s
user    0m0.000s
sys     0m0.000s

What I've looked for so far:

• top while apache is restarting doesn't show a lot (~0% usage).
• netstat also doesn't show any connections with the outside world.

Note that this is a VM which currently has 0 traffic and there are plenty of free GBs available in memory and disk.

I've also found that it's the "stop" part of the "restart" is what takes 90 seconds.

Any idea why this is happening or where should I look at next?

Edit: I found out that when stop takes 90 seconds I consistently get the following in /var/log/apache2/error_log:

[core:notice] [pid 3179] AH00052: child pid 3203 exit signal Segmentation fault (11)

I have an Apache/2.2.22 (Linux/SUSE) and it automatically restarts every day at the same time. I have no cron jobs configured to do that on any user.

I started collecting minutelly snapshots of the status page and from that I found out that:

Restart Time: Friday, 31-Jan-2014 12:15:03 EET
Restart Time: Saturday, 01-Feb-2014 12:15:04 EET
Restart Time: Sunday, 02-Feb-2014 12:15:05 EET
Restart Time: Monday, 03-Feb-2014 12:15:06 EET

From the miuntelly snapshots of the status page it turns out that no significance traffic occurst before the restart.

For each restart I get the following /var/log/messages without anything else of interest:

2014-02-03T12:15:02.576970+02:00 foobar systemd[1]: Reloading apache.
2014-02-03T12:15:03.225024+02:00 foobar start_apache2[15393]: Syntax OK
2014-02-03T12:15:03.298169+02:00 foobar systemd[1]: Reloaded apache.

Any ideas why this is happening or where should I look at next?

In vcl_recv I decide whether to pass or lookup based on the existence of a cookie:

sub vcl_recv {
    if (req.http.Cookie ~ "(JSESSIONID=)" ) {
            /* do not cache logged in users */
            return (pass);
    }
    return (lookup);
}

In vcl_fetch I fine tune caching for some pages:

sub vcl_fetch {

    /* custom rules block */
    if (req.url ~ "^/foo") { set beresp.ttl=30s; }
    if (req.url ~ "^/bar") { set beresp.ttl=1m; }
    if (req.url ~ "^/123") { set beresp.ttl=10m; }

    return (deliver);
}

How can I know within vcl_fetch whether I am in pass or lookup mode? I'd like to be able to avoid running the "custom rules block" if I am in lookup mode because the logged in users will anyway run in pass mode so these rules do not apply to them.

We have a LAN with ~40 workstations (mostly Windows) and a couple of servers. All of them use an internal DNS (196.168.0.4 running BIND 9.5.0-P2) and a gateway (192.168.0.1 running OpenBSD Packet Filter) which is a local PC acting as router.

For the last couple of months at some points during the workday the network is slowed down to an extent where doing anything internet related is not possible. On those bad times pinging 8.8.8.8 gives:

12:16:12.078: Timeout waiting for seq=11a1
12:16:13.484: From 8.8.8.8: bytes=60 SEQ=11a9 TTL=48 ID=0000 time=399.334ms
12:16:15.078: Timeout waiting for seq=11a4
12:16:15.437: From 8.8.8.8: bytes=60 SEQ=11ab TTL=48 ID=0000 time=355.409ms
12:16:18.078: Timeout waiting for seq=11a8
12:16:19.453: From 8.8.8.8: bytes=60 SEQ=11af TTL=48 ID=0000 time=376.317ms
12:16:21.078: Timeout waiting for seq=11aa
12:16:21.078: Timeout waiting for seq=11ac
12:16:21.390: From 8.8.8.8: bytes=60 SEQ=11b1 TTL=48 ID=0000 time=306.727ms
12:16:22.437: From 8.8.8.8: bytes=60 seq=11b2 TTL=48 ID=0000 time=364.351ms
12:16:23.453: From 8.8.8.8: bytes=60 seq=11b3 TTL=48 ID=0000 time=371.944ms
12:16:24.078: Timeout waiting for seq=11ad
12:16:24.078: Timeout waiting for seq=11ae
12:16:26.390: From 8.8.8.8: bytes=60 SEQ=11b6 TTL=48 ID=0000 time=307.729ms
12:16:27.078: Timeout waiting for seq=11b0
12:16:29.437: From 8.8.8.8: bytes=60 SEQ=11b9 TTL=48 ID=0000 time=361.575ms
12:16:30.078: Timeout waiting for seq=11b4
12:16:30.453: From 8.8.8.8: bytes=60 seq=11ba TTL=48 ID=0000 time=367.647ms
12:16:33.078: Timeout waiting for seq=11b5
12:16:33.078: Timeout waiting for seq=11b7

At that exact instance if I turn the DNS (at .0.4) off then after a couple of seconds the network's health goes very good again:

12:47:43.046: From 8.8.8.8: bytes=60 seq=190b TTL=48 ID=0000 time=70.555ms
12:47:44.046: From 8.8.8.8: bytes=60 seq=190c TTL=48 ID=0000 time=82.684ms
12:47:45.046: From 8.8.8.8: bytes=60 seq=190d TTL=48 ID=0000 time=72.368ms
12:47:46.062: From 8.8.8.8: bytes=60 seq=190e TTL=48 ID=0000 time=84.310ms
12:47:47.046: From 8.8.8.8: bytes=60 seq=190f TTL=48 ID=0000 time=75.137ms
12:47:48.046: From 8.8.8.8: bytes=60 seq=1910 TTL=48 ID=0000 time=75.791ms
12:47:49.062: From 8.8.8.8: bytes=60 seq=1911 TTL=48 ID=0000 time=94.252ms
12:47:50.046: From 8.8.8.8: bytes=60 seq=1912 TTL=48 ID=0000 time=76.547ms
12:47:51.046: From 8.8.8.8: bytes=60 seq=1913 TTL=48 ID=0000 time=70.251ms
12:47:52.046: From 8.8.8.8: bytes=60 seq=1914 TTL=48 ID=0000 time=83.033ms
12:47:53.046: From 8.8.8.8: bytes=60 seq=1915 TTL=48 ID=0000 time=76.589ms
12:47:54.046: From 8.8.8.8: bytes=60 seq=1916 TTL=48 ID=0000 time=82.060ms

This is very consistent and reproducible. The fact that I ping 8.8.8.8 (Google's public DNS) is completelly random and just a way I have to test internet connectivity. I could be pinging 206.190.36.45 (an IP of Yahoo's public website).

The DNS is not open to the outside world. So I think that maybe one (or more) of the workstations make very bad use of the DNS (probably indirectly via a virus) and flood it with requests or something. The problem is that I cannot trace that back. On the 0.4 machine top gives me no CPU suspicious activity and on 0.1 (the gateway) filtering using dst host 192.168.0.4 in pftop doesn't give me any internal IP using the DNS.

I've tried pluging out the ethernet cables the workstations one by one to find a possible offending workstation but this process is not very fast and accurate and by the time the network stabilizes I'm not really sure whether it was due to the last workstation I plugged out or whether the network simply went good again.

Any ideas on where to look at next?

In an openSUSE 11.1 I download, compile and install ImageMagick via:

wget ftp://.../pub/graphics/ImageMagick/ImageMagick-6.7.7-0.zip
unzip ImageMagick-6.7.7-0.zip
cd ImageMagick-6.7.7-0
./configure --prefix=/usr/local/ImageMagick
make
make install

Everything works nicelly until I discover that JPG is not supported:

identify -list format | grep -i jpg

[nothing related to JPG returned]

So I reconfigure and recompile using:

./configure --prefix=/usr/local/ImageMagick --with-jpeg=yes --with-jp2=yes
make
make install

But that changes nothing.

I end up uninstalling:

make uninstall

and installing via zypper:

zypper install ImageMagick

This installed version 6.4.3 and now it does support JPG:

identify -list format | grep -i jpg
JPG* JPEG      rw-   Joint Photographic Experts Group JFIF format

Any idea on what is going on here? What is a possible reason that this capability of ImageMagick was not there when compiled from source but was there when installed from rpm?

Note that I don't necessarily care a lot about ImageMagick (since it now works), but generally about his kind of behaviour, becase in one way or another I've seen this happen in other ocasions as well.

I've got a huge textfile which contains IP addresses:

123.33.22.33
221.23.128.2
123.33.22.33
92.222.192.12
92.222.192.12
123.33.22.33

I can sort it to:

123.33.22.33
123.33.22.33
123.33.22.33
221.23.128.2
92.222.192.12
92.222.192.12

and see (with bare eye) that first IP occurs three times, second once, and last one twice.

I'd like to be able to do this in huge logfiles, obviously in an automated way. Is it possible?

thanks

I need to store 100k files (around 40GB) in a USB drive. Each file has a unique int id (e.g 45000).

Option one is to put all files in a single folder:

root/
root/1.pdf
root/2.pdf
root/3.pdf
...
root/567.pdf
root/568.pdf
root/569.pdf
...
root/10001.pdf
root/10002.pdf
root/10003.pdf
...
root/99998.pdf
root/99999.pdf
root/100000.pdf

Option two is to create a [1-9][0-9]* folder hierarchy based on that id:

root/
root/1/file.pdf
root/2/file.pdf
root/3/file.pdf
...
root/5/6/7/file.pdf
root/5/6/8/file.pdf
root/5/6/9/file.pdf
...
root/1/0/0/0/1/file.pdf
root/1/0/0/0/2/file.pdf
root/1/0/0/0/3/file.pdf
...
root/9/9/9/9/8/file.pdf
root/9/9/9/9/9/file.pdf
root/1/0/0/0/0/0/file.pdf

Which option will scale better? I can understand that the second option will require tons of folders but each folder will at most contain 10 folders and 1 file. Maintenance will not be an issue since everything will be controlled by an application.

Note that this is a USB drive on linux and based on the above I'd also like to know whether I should go with FAT32 or NTFS.