I have a metric that should only be checked and logged daily.
Calculation is CPU-intensive, and querying multiple times per day returns the same value (i.e. it just wastes CPU time).
Is it possible to make a plugin that is queried only once per day?
I've tried to specify "update_rate 3600" in munin-node config for this plugin, but it was still queried every 5 minutes. Graphs are also displayed with 5 minutes granularity.
Surely, local cache is an option, but makes things more complicated.
On my local file server I have raid-6 on 7x HDD drives.
dd if=/dev/zero of=tempfile bs=1M count=2048 conv=fdatasync
Local speed test gives me 349 MB/s write speed.
Remote writes to Samba from SSD (>2Gb/s read speed) gives me 259 MB/s writes. But remote writes to iSCSI drive (on Win10 iSCSI initiator) gives me mere 151 Mb/s writes.
raid6 config - 128K chunk size, stripe_cache_size = 8191. Write intent bitmap is on SSD (Samsung 860 PRO, 4096K bitmap chunk).
Array mounted with options: rw,noatime,nobarrier,commit=999,stripe=128,data=writeback
open-iscsi setup: target is based on a 4Tb file.
Any hints why iSCSI is slower than Samba on writes? Any hints on how to improve iSCSI writes speed?
I assume it has something to do with the desire of open-iscsi to flush writes to disk after each operation, which increases write amplification on raid6 due to excessive parity rewrites. But I am not sure how to fix it. Speed it more important than safety of currently written data in case of power outage.
As a side note older ietd iSCSI target had the ability to enable write-back mode (using IOMode=wb) and the sustained write speed was much faster. Unfortunately it seems to be currently unmaintained.
I am serving large static files from a backend storage server with slow spinning disks. This server is hidden behind fast nginx reverse proxy with local cache on SSD. It works great and fast.
Now I want to change storage backend, and as a result I cannot maintain same location of stored files on a backend. Instead of root of the server, they will have to be served from a subdirectory. How can I modify nginx reverse proxy config so that it proxies all non-cached requests to backend to a subdirectory, and clients are not aware that anything has changed?
I cannot do anything on storage server to maintain old URL scheme, so I have to do it on a frontend. No 301/302 headers are supposed to be passed anywhere.
So currently I have:
What I want to achieve:
I have tried many dozens of configurations, without luck. When I am trying this configuration - instead of silently fetching data from different URL, it ends up in an infinite loop of adding test via regexp.
location / {
rewrite /(.*) /test/$1 break;
proxy_pass http://f002.backblazeb2.com;
proxy_redirect off;
proxy_set_header Host $host;
proxy_cache_use_stale error timeout http_500 http_502 http_503 http_504;
proxy_cache_revalidate on;
proxy_read_timeout 2;
proxy_connect_timeout 3;
proxy_http_version 1.1;
proxy_set_header Connection "";
proxy_cache_valid 200 302 60s;
proxy_cache_valid 404 1m;
limit_conn perip 23;
limit_req zone=dynamic burst=60;
expires 24h;
}
I've written my own simple munin-node implementation, but cannot get it to show on an master (2.0.33-1) in munin-cgi-html.
munin.conf configuration:
[uberlamp3]
address 192.167.0.21
use_node_name yes
port 19270
I monitor conversation between the master and the node and see see that node is asked for:
1: cap (I return empty caps)
2: list (I return the only plugin - temp)
3: config temp. I return:
graph_args --title "Lamp Temperatures"
graph_category Temperatures
temp.label temp
.
4: fetch temp. I return:
temp.value 47.3
5: quit - I drop connection. No errors in munin-update.log
All that is processed normally on munin-update. No errors in munin-update.log. Corresponding rrd file is created and updated with correct values (checked with rrdtool). But munin-cgi-html never show this new node in generated HTML.
munin-update.log:
2017/06/23 07:40:01 [INFO] starting work in 9170 for uberlamp3/192.167.0.21:19270.
2017/06/23 07:40:01 [INFO] node uberlamp3 advertised itself as uberlamp.14.by instead.
2017/06/23 07:40:01 [INFO]: Munin-update finished for node uberlamp3;uberlamp3 (0.33 sec)
2017/06/23 07:40:06 [INFO] Reaping Munin::Master::UpdateWorker<uberlamp3;uberlamp3>. Exit value/signal: 0/0
Few more observations:
1) htmlconf.storable - contains references to uberlamp3 node
2) I've tried to add one more native munin-node, which added without issues - and after update shows in html/graphs. uberlamp3 though still not showing.
Any suggestions?
Munin version: 2.0.33-1 Ubuntu Version: tried 16.10 and 17.04
Is there a way to mount network drive on Windows with client-side read/write cache enabled? In my case drive is guaranteed to be mounted by 1 user, and performance is way more important than reliability in case of network/power fails.
I have full control over the Linux file server, so any services are possible (some special iSCSI or something else).
Just raw non-cached performance limited by Gigabit network is not enough.
I am optimizing my VPS where I unfortunately have to run ROR under passenger. I've set it up so that it shutdown all application instances after 1 minute idle time, but there is still substancial amount of memory consumed by the following parts, which are running forever:
23716 www-data 20 0 75300 60m 2384 S 0 15.2 0:03.15 ruby1.8
23712 root 20 0 30584 19m 2424 S 0 4.8 0:01.20 ruby1.8
23675 root 20 0 15872 6340 1572 S 0 1.6 0:00.47 ruby1.8
23678 nobody 20 0 10476 3036 2484 S 0 0.7 0:00.00 PassengerLoggin
23673 root 20 0 14172 1964 1728 S 0 0.5 0:00.02 PassengerHelper
23670 root 20 0 5328 1604 1420 S 0 0.4 0:00.00 PassengerWatchd
Any suggestions on what can I do with them? ROR is rarely used, but due to these idle processes it eats like 25% of VPS RAM :-(
PS. I've already switched to Enterprise edition of ruby to try to reduce memory consumption....
I am building home cluster where I going to have about 16 nodes which can live with 1G ports, but I want to have 10GE on file server & central node. It's all local, so no need for cables longer than 3-5m. And of course I want to spend as little money as possible (not going to spend more than whole cluster costs) :-)
What are my options?
I am migrating from AIX to Solaris 10, but the version of /bin/sh there is so old, that our scripts does not work. Can I somehow force perl to execute commands in `` via different shell (in my case, /bin/ksh)?
I have A LOT of places with system calls, so changing them in every place is not very feasible. It would be great to find some global setting or something.
I am on dual core C2D processor, Windows 7 (although this behavior is the same in all MS OSes I've seen). When I run any single threaded CPU-intensive task, it does not stick to 1 core, but rather randomly occupy some 50% of first core, and some 50% on second.
Why they do that? Doesn't that lead to lowering efficiency of L1 cache?
I know that I can set process affinity, I am just wondering the reasouns of such behavior.
Why in many examples people try to create separate repository per project?
Why not have single repo and manage access via authz file?
Now it looks for me like extra administrative burden with little benefit...
What are the ways to improve performance of the following procedure:
Every day I take a datafile, bcp it into an intermediate table without indexes (so that data import is as fast as possible), and then move it to a second table which has a lot of complex indexes which guarantee data validity. Number of rows inserted each day is like 1 million.
Data retention on the intermediate table does not matter, so I was thinking about using in-memory table (which are unfortunately available only in 15.5) or disabling transaction log(is it possible to just 1 table?), or something else...
Any clues?
I just faced a problem with encrypted home directory in Ubuntu Server 10.04.
While I know a workaround, I want to just to completely remove encryption from everything.
Any hints how to do that without complete backup & reinstall?
I'am using Putty on windows with key-based authentication to access some of mine servers.
It works totally fine with ~3700-bit key, but with ~17000-bit key it thinks for like 20 seconds on client-side and then just says "Access denied" and asks for a password.
Is there any key length limit or timeout in OpenSSH for key-based authentication?
I understand that using such large keys have not much practical sence, especially when looking at these 20 seconds of calculation, just trying to solve any problems I face :-)...
I am trying to optimize my PHP<>MySQL on this test script:
<?
for($i=0;$i<100;$i++)//Iterations count
$res.= var_dump(loadRow("select body_ru from articles where id>$i*50 limit 100"));
print_r($res);
?>
I have APC, and article table have an index on id. Also, all these queries are hitting query cache, so sole MySQL performance if great.
But when I am using ab -c 10 -t 10 to bench this script, I am getting:
100 iterations: ~100req/sec (~10'000 MySQL queries per second)
5 iteration: ~200req/sec
1 iteration: ~380req/sec
0 itteration: ~580req/sec
I've tried to disable persistent connections in PHP - it made it slower a bit. So, how can I make it work faster, provided that MySQL is not limiting performance here?
So this looks like establishing connection or something else is taking too much time...
When some domain have primary NS, and several secondary NSes, will clients ask them randomly to reduce the load, or they will hit primary NS only, and proceed to secondary only when primary fails?
I am tweaking my homepage for performance, currently it handles about 200 requests/second on 3.14.by which eats 6 SQL queries, and 20 req/second on 3.14.by/forum which is phpBB forum.
Strangely enough, numbers are about the same on some VPS and dedicated Atom 330 server.
Server software is the following: Apache2+mod_php prefork 4 childs (tried different numbers here), php5, APC, nginx, memcached for PHP sessions storage.
MySQL is configured to eat about 30% of available RAM (~150Mb on VPS, 700Mb on dedicated server)
This looks like there is a bottleneck somewhere not allowing me to go higher, any suggestions? (i.e. I know that doing less than 6 SQL's would make it faster, but this does not look like a limiting factor, as sqld eats no more than a few % in top due to cached queries)
Has anyone tested that kicking preforked apache2 and leaving just nginx+php is much faster?
Some more benchmarks
Small 40-byte static file: 1484 r/s via nginx+apache2, 2452 if we talk to apache2 directly.
Small "Hello world" php script: 458 r/s via ngin+apache2.
Update: It appears bottleneck is MySQL performance on cached data. Page with single SQL shows 354req/sec, with 6 SQL's - 180 req/sec. What do you think I can tweak here? (I can fork out 100-200Mb for MySQL)
[client]
port = 3306
socket = /var/run/mysqld/mysqld.sock
[mysqld_safe]
socket = /var/run/mysqld/mysqld.sock
nice = 0
[mysqld]
default-character-set=cp1251
collation-server=cp1251_general_cs
skip-character-set-client-handshake
user = mysql
pid-file = /var/run/mysqld/mysqld.pid
socket = /var/run/mysqld/mysqld.sock
port = 3306
basedir = /usr
datadir = /var/lib/mysql
tmpdir = /tmp
skip-external-locking
bind-address = 127.0.0.1
key_buffer = 16M
max_allowed_packet = 8M
thread_stack = 64K
thread_cache_size = 16
sort_buffer_size = 8M
read_buffer_size = 1M
myisam-recover = BACKUP
max_connections = 650
table_cache = 256
thread_concurrency = 10
query_cache_limit = 1M
query_cache_size = 16M
expire_logs_days = 10
max_binlog_size = 100M
[mysqldump]
quick
quote-names
max_allowed_packet = 8M
[mysql]
[isamchk]
key_buffer = 8M
!includedir /etc/mysql/conf.d/
I am running my own Authoritative DNS on bind9 for like 5 of my domains with little traffic. I am running them on 2 separate VPS servers (so every Mb of memory costs me some $x2).
Is there any way to reduce it's memory consumption down from ~45-50Mb? (I would be happy with 10-30Mb)
I know there are lighter DNS server alternatives, but this would require to replace server managing software, so I would prefer to stick with bind.
Suggestions? I've heard bind8 is better on memory consumptoin. Will I fall to hell for using it?