Steve Bennett's questions

I'm developing a server deployment script that installs a bunch of stuff and generates a log in the process. I'd like users of the script to be able to view what's happening, on a web page, in real time (eg, without browser refresh). What's a simple way to make this happen?

I tried log.io but missed the part about "no persistence layer". Tools like nagios are far too complex for this.

Basically I'm looking for something that goes:

1. sudo apt-get install -y logviewer
2. nohup logviewer --port 1234 < /var/log/mylog.log &

The target server is Ubuntu on OpenStack. The log is mostly generated by Salt Stack.

When I need to start a background process on a low-importance server, I tend to use nohup:

nohup ./server.sh &

Most colleagues seem to prefer screen:

screen -D -R mydaemon
./mydaemon.sh
^A ^D

Are there any major differences in the effect of these two approaches? What are the benefits one way or the other?

I'm running a database loading process (osm2pgsql) which is failing:

Processing: Node(17404k 148.8k/s) Way(1351k 6.38k/s) Relation(9520 29.94/s)way_done failed: ERROR:  could not extend file "base/140667/152463": No space left on device
HINT:  Check free disk space.
(7)
Arguments were: 187226311, 

At the start of the import, mem reports:

             total       used       free     shared    buffers     cached
Mem:           31G        29G       2.4G         0B       178M        24G
-/+ buffers/cache:       4.5G        26G
Swap:           0B         0B         0B

Shortly before the end:

             total       used       free     shared    buffers     cached
Mem:           31G        31G       227M         0B       178M        26G
-/+ buffers/cache:       4.8G        26G
Swap:           0B         0B         0B

Meanwhile, df at the start:

Filesystem     1K-blocks      Used Available Use% Mounted on
/dev/vda1       10309828   7879412   1997036  80% /
udev            16470572        12  16470560   1% /dev
tmpfs            6590080       260   6589820   1% /run
none                5120         0      5120   0% /run/lock
none            16475196         0  16475196   0% /run/shm
none              102400         0    102400   0% /run/user
/dev/vdb       247709760 105978300 129148548  46% /mnt

And from about 3/4 of the way through the process. Use sits at 100%.

Filesystem     1K-blocks      Used Available Use% Mounted on
/dev/vda1       10309828   9854348     22100 100% /
udev            16470572        12  16470560   1% /dev
tmpfs            6590080       260   6589820   1% /run
none                5120         0      5120   0% /run/lock
none            16475196         0  16475196   0% /run/shm
none              102400         0    102400   0% /run/user
/dev/vdb       247709760 105978300 129148548  46% /mnt

I'm unable to identify any actual files on disk that are filling up /dev/vda1:

du -h -d 3 / 2>/dev/null | grep -v ^0 > /tmp/o2p1.txt
[start import]
du -h -d 3 / 2>/dev/null | grep -v ^0 > /tmp/o2p2.txt

diff /tmp/o2p1.txt /tmp/o2p2.txt

That reveals nothing.

What's going on?

I'm trying to set up an nginx config as follows: When receiving a request like /tile/SteveCountryVic/1/2/3.png:

1. Attempt to pass it through to http://localhost:5005/1/2/3.png
2. If that 404s, pass it to another server as /tile/SteveCountryVic/1/2/3.png

Here's my config which isn't quite working:

server {
   listen 80;
   server_name localhost;   error_log  /tmp/nginx.error.log notice;
   access_log   /tmp/nginx.access.log;
   location /tile/SteveCountryVic/ {
        rewrite_log on;        
        #rewrite ^.*/(\d+)/(\d+)/(\d+).*$ /$1/$2/$3.png break;

        proxy_intercept_errors on;
        error_page 404 = @dynamiccycletour;        
        #proxy_set_header Host $http_host;
        #proxy_pass http://127.0.0.1:5005;
        proxy_redirect /tile/SteveCountryVic/ http://localhost:5005/;

   location @dynamiccycletour {
        rewrite_log on;
        #rewrite ^(\d+)/(\d+)/(\d+).*$ /tile/SteveCountryVic/$1/$2/$3.png break;
        proxy_pass http://115.x.x.x:20008;


   }

   location /tile/ {
        proxy_set_header Host $http_host;
        proxy_pass http://127.0.0.1:20008;


        proxy_cache my-cache;
        proxy_cache_valid  200 302  60m;
        proxy_cache_valid  404      1m;
    }
    ...

In this configuration, all requests seem to get redirected to the proxied server, but images are ultimately served. In addition, the error log contains these lines:

2013/09/10 09:44:11 [error] 564#0: *138 open() "/etc/nginx/html/tile/SteveCountryVic/13/7399/5027.png" failed (2: No such file or directory), client: 118.x.x.x, server: localhost, request: "GET /tile/SteveCountryVic/13/7399/5027.png?updated=15 HTTP/1.1", host: "mydomain.org"

If instead of using proxy_redirect, I use rewrite and proxy_pass:

        rewrite ^.*/(\d+)/(\d+)/(\d+).*$ /$1/$2/$3.png break;
        proxy_pass http://127.0.0.1:5005;

Then now I actually see the 404 messages in the browser (ie, they don't get intercepted).

My questions:

1. What am I doing wrong?
2. Why on earth is nginx looking for files in /etc/nginx/html/...?
3. Is there a way to get even more logging information (specifically, to better understand proxy_redirect)?

I have a TileMill/PostGIS stack running on an 8 core Ubuntu 12.04 VM on an OpenStack cloud. It's a rebuild of a very similar system which was running nicely on very similar hardware (same cloud, but different physical hardware, I believe) last week. I've attempted to rebuild the stack exactly the same as it was (using some scripts I had built).

Everything runs, but the database is carrying out queries excruciatingly slowly, which manifests itself ultimately with very slow tile generation. An example query (count the number of pubs within a radius of every town in Australia), which previously took something like 10-20 seconds is now taking over 10 minutes:

explain (analyze, buffers) update places set pubs = 
(select count(*) from planet_osm_point p where p.amenity = 'pub' and st_dwithin(p.way,places.way,scope)) +
(select count(*) from planet_osm_polygon p where p.amenity = 'pub' and st_dwithin(p.way,places.way,scope)) ;
 Update on places  (cost=0.00..948254806.93 rows=9037 width=160) (actual time=623321.558..623321.558 rows=0 loops=1)
   Buffers: shared hit=132126300
   ->  Seq Scan on places  (cost=0.00..948254806.93 rows=9037 width=160) (actual time=68.130..622931.130 rows=9037 loops=1)
         Buffers: shared hit=132107781
         SubPlan 1
           ->  Aggregate  (cost=12.95..12.96 rows=1 width=0) (actual time=0.187..0.188 rows=1 loops=9037)
                 Buffers: shared hit=158171
                 ->  Index Scan using planet_osm_point_index on planet_osm_point p  (cost=0.00..12.94 rows=1 width=0) (actual time=0.163..0.179 rows=0 loops=9037)
                       Index Cond: (way && st_expand(places.way, (places.scope)::double precision))
                       Filter: ((amenity = 'pub'::text) AND (places.way && st_expand(way, (places.scope)::double precision)) AND _st_dwithin(way, places.way, (places.scope)::double precision))
                       Buffers: shared hit=158171
         SubPlan 2
           ->  Aggregate  (cost=104917.24..104917.25 rows=1 width=0) (actual time=68.727..68.728 rows=1 loops=9037)
                 Buffers: shared hit=131949237
                 ->  Seq Scan on planet_osm_polygon p  (cost=0.00..104917.24 rows=1 width=0) (actual time=68.138..68.716 rows=0 loops=9037)
                       Filter: ((amenity = 'pub'::text) AND (way && st_expand(places.way, (places.scope)::double precision)) AND (places.way && st_expand(way, (places.scope)::double precision)) AND _st_dwithin(way, places.way, (places.scope)::double precision))
                       Buffers: shared hit=131949237
 Total runtime: 623321.801 ms

(I'm including this query as a symptom, not directly the problem to be solved. This particular query only gets run once a week or so.)

The server has 32 GB of RAM, and I've configured Postgres as follows (following advice found on the web):

shared_buffers = 8GB
autovacuum = on
effective_cache_size = 8GB
work_mem = 128MB
maintenance_work_mem = 64MB
wal_buffers = 1MB
checkpoint_segments = 10

iostat shows nothing being read, a bit of data being written (no idea where or why), and 95% idle CPU:

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           5.40    0.00    0.00    0.11    0.00   94.49

Device:            tps    kB_read/s    kB_wrtn/s    kB_read    kB_wrtn
vda               0.20         0.00         0.80          0          8
vdb               2.30         0.00        17.58          0        176

Sample output from vmstat:

  procs -----------memory---------- ---swap-- -----io---- -system-- ----cpu----
 r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa
...
 1  0      0 18329748 126108 12600436    0    0     0    18  148  140  5  0 95  0
 2  0      0 18329400 126124 12600436    0    0     0     9  173  228  5  0 95  0

Clutching at straws, I moved the Postgres data directory from vda to vdb but of course that made no difference.

So I'm at a loss. Why is Postgres only using 5% of available CPU when it's not waiting for any I/O? I'd welcome any suggestions for further investigation, other tools, random things to try.

Update

I snapshotted the server and launched it on a different part of the same cloud (a different availability zone). The results were a bit weird. vmstat on this server reports 12% CPU usage (which I now understand as the expected value for a single Postgres query on an 8 core VM) - although the actual query execution time is virtually identical (630 seconds vs 623).

I now realise that this particular query is probably not a good sample for this reason: it can only use one core, and it's an update (whereas tile rendering is just selects).

I also didn't notice in the explain that apparently planet_osm_polygon isn't using an index. That could well be the cause, so I'll chase that next.

Update2

The problem definitely seems to be that the planet_osm_polygon index(es) is/aren't being used. There are two (one created by osm2pgsql, one created by me following some random guide):

CREATE INDEX idx_planet_osm_polygon_tags
  ON planet_osm_polygon
  USING gist
  (tags);


CREATE INDEX planet_osm_polygon_pkey
  ON planet_osm_polygon
  USING btree
  (osm_id);

The stats on planet_osm_polygon and planet_osm_point are pretty revealing, I think:

planet_osm_polygon:

Sequential Scans    194204  
Sequential Tuples Read  60981018608 
Index Scans 1574    
Index Tuples Fetched    0

planet_osm_point:

Sequential Scans    1142    
Sequential Tuples Read  12960604    
Index Scans 183454  
Index Tuples Fetched    43427685

If I read that right, Postgres has searched the planet_osm_polygon 1574 times, but never actually found anything, so has done a ridiculously large number of brute force searches.

The new question: why?

Mystery solved

Thanks to Frederik Ramm's answer, the answer turns out to be fairly simple: there was no spatial index, for some reason. It was trivial to regenerate them:

create index planet_osm_polygon_polygon on planet_osm_polygon using gist(way);
create index planet_osm_polygon_point on planet_osm_point using gist(way);

Running that query now takes 4.6 seconds. Spatial indexes matter! :)

I understand that (almost?) all SSH command line arguments can be replaced with a property in ~/.ssh/config. But it's not at all obvious how to do this systematically: the man page for ssh only refers loosely to the config page (eg, "Port forwardings can also be specified in the configuration file."). The man page for ssh_config never refers to command line options.

So, is there a definitive list of the corresponding config parameters and command line options? For example, what does -R correspond to?

In MySQL, how do you allow another user to create other users? In turn, those users should only be able to view some tables.

You add a rule like this:

ufw allow 22/tcp

The rule is saved, and is applied even after reboot. But it's not written anywhere in /etc/ufw. Where is it saved to? (Ubuntu, using ufw as pre-installed.)

I'm looking for a SMTP server that will just be used to forward a few email addresses. Two requirements:

1. Easy to install and configure on Ubuntu.
2. Has an aliases file that is easy to automatically add to. Ideally, it would be a text file consisting of rows like "[email protected] [email protected]".
3. (Ideally) Can be easily configured to only accept mail from specific hosts.

I've been trying Postfix, but I'm getting bogged down in error messages like Recipient address rejected: User unknown in virtual alias table and Recipient address rejected: User unknown in local recipient table. So I wonder if there is an easier solution.

I have a directory structure (sample data) that I want to copy from within a Chef recipe. It seems the only way to do this is to explicitly create each individual directory and file:

directory "/mnt/data/experiment1/dataset1" do
   recursive true
   only_if { node.chef_environment == "dev" }
end


directory "/mnt/data/experiment1/dataset2" do
   recursive true
   only_if { node.chef_environment == "dev" }
end

directory "/mnt/data/experiment2/dataset1" do
   recursive true
   only_if { node.chef_environment == "dev" }
end


directory "/mnt/data/experiment1/dataset2" do
   recursive true
   only_if { node.chef_environment == "dev" }
end


cookbook_file "/mnt/data/experiment1/dataset1/testfile1.txt" do
   owner "atom"
   group "atom"
   mode "0644"
   source "sampledata/experiment1/dataset1/testfile1.txt"
   only_if { node.chef_environment == "dev"}
end

...

Is there a way to simply recursively copy an entire directory structure from the cookbook? Specifying the name of each file inside the recipe seems redundant and error-prone (ie, if we add a file to the tree but forget to reference it in the recipe, it won't be copied.)

I guess a hack workaround would be to work out where all the chef files get copied to on the target machine and do a cp -r but is there something cleaner?

Or am I going about this the wrong way?

Let's say you have a standard Chef repository with directories as follows:

cookbooks
data_bags
environments
roles

Is there way to upload it all in one go? Otherwise you have to do this:

knife cookbook upload -a
knife data bag from file data_bags/*.json
knife environment from file environments/*.rb
knife role from file roles/*.json

Perhaps there are third-party tools to do this kind of thing?

I want to run a cookbook_file resource only if the current environment is "dev". How can this be expressed?

The documentation suggests this:

In a recipe, a code block like this would be useful:

qa_nodes = search(:node,"chef_environment:QA")      
qa_nodes.each do |qa_node|                          
    # Do useful specific to qa nodes only
end

But I'm not sure that's what I want - the fact it's a loop seems wrong.

This is the way recommended in the Chef Fast Start tutorial:

knife ssh name:mynode -a ipaddress  -x ubuntu -i mycredentials.pem "sudo chef-client"

This is really clumsy. Is there really not a better way, or is the idea that in a real production environment, you'll have nodes auto-updating anyway?

Say you're seeing this message:

FATAL:  Ident authentication failed for user "..."

What are the causes of this error message?

Say I have a command foo which takes a filename argument: foo myfile.txt. Annoyingly, foo doesn't read from standard input. Instead of an actual file, I'd like to pass it the result of another command (in reality, pv, which will cat the file and output a progress meter as a side effect).

Is there a way to make this happen? Nothing in my bag of tricks seems to do it.

(foo in this case is a PHP script which I believe processes the file sequentially).

I'm using Ubuntu and Bash

EDIT Sorry for the slightly unclear problem description, but here's the answer that does what I want:

pv longfile.txt | foo /dev/stdin

Very obvious now that I see it.

This seems straightforward, but I can't make it work.

I have a pretty trivial webserver that only needs to do two things:

1. example.com/status.html returns a local file (this works)
2. example.com/atom redirects the contents of the firewalled server running on the same machine on port 4000

This didn't work:

RewriteRule ^$ http://localhost:4000

That redirected traffic back to localhost:4000 from the requestor's point of view (ie, on the client's machine).

My limited understanding of VirtualHost indicates that that's for something like atom.example.com, not example.com/atom

I think I have to use ProxyReverse, but I can't find the right combination.

UPDATE: Trying the ProxyPass/ProxyReverse suggestion given by Shane Madden produces this in the error log file:

[Thu Mar 15 11:59:15 2012] [error] (13)Permission denied: proxy: HTTP: attempt to connect to 127.0.0.1:4000 (localhost) failed
[Thu Mar 15 11:59:15 2012] [error] ap_proxy_connect_backend disabling worker for (localhost)
[Thu Mar 15 11:59:17 2012] [error] proxy: HTTP: disabled connection for (localhost)

(The local server is definitely running on 127.0.0.1:4000)

$ ps | grep django
28006 ttys004    0:01.12 /usr/bin/python bin/django celeryd --beat
51393 ttys005    0:01.45 /usr/bin/python bin/django celeryd -l INFO
51472 ttys005    0:01.29 /usr/bin/python bin/django celeryd -l INFO
51510 ttys005    0:01.89 /usr/bin/python bin/django celeryd -l INFO
51801 ttys005    0:01.83 /usr/bin/python bin/django celeryd -l INFO
53470 ttys005    0:03.97 /usr/bin/python bin/django celeryd -l INFO
53780 ttys005    0:00.00 grep django

Is there a way to prevent the last process (that is, the grep that was started at the same time as my ps command) being reported?

(I started trying to come up with a regex that would match the literal but not match itself, but that seemed, um, not the right approach...)

Is logrotate hiding somewhere on OSX, or is there an equivalent? It's not in /usr/sbin.

Say you have directory ~/mytool that softlinks to /usr/local/mytool. Then cd mytool leaves your current directory as ~/mytool, which may cause scripts to behave incorrectly. Is there a way to avoid this behaviour?

With some googling, I see that you can achieve this as follows:

cd $1 ; cd `pwd -P`

Is there no switch to 'cd'? Environment variables?

I've recently had an experience of writing a shell script which crashed a server (and damaged a partition) by consuming all resources. It was hooked up to a cron job, and it seems it took longer to run than the interval between executions, over time snowballing out of control.

Now, I've since modified it to record its running state, and to not run more than once simultaneously. My question is: are there other, simple ways to safeguard a script against causing harm? Is there a standard list of things a script should do to behave properly, not consume too many resources, to fail gracefully, alert the right people, etc?

Basically: what other pitfalls should I avoid?