dpc.pw's questions

I have an application that is constantly pumping data to MongoDB. MongoDB instance is running with 2-replicas, each with 3TB gp2 EBS volume.

As you can see on the following graph of

irate(node_disk_reads_completed_total{}[1m])
irate(node_disk_writes_completed_total{}[1m])

The reading performance is slow and steady, which is normal, but the writes seem to be under-performing.

Even in the peaks we never reach theoretical 3IOPS/GB * 3000 GB = 9k IOPS. The application itself is spending most of the clock time pumping more and more data to the DB, so from the perspective of the app the DB is the clear bottleneck.

So why can't it go faster? And why is it such a wave? I would expect that writing to WAL will provide constant source of write activity and things like periodic fsync to disk would not really cause such extreme up and down patterns.

Could it be that syncing between replicas are causing pauses in the throughput? But I am using the default write concern, which should be w: 1, j: true and not require waiting for replicas.

Anything else that I might be missing?

I'm researching a topic of high availability database for a Rails application. I'm looking for Open Source based solutions.

At the moment our system is running on one application host + one db host, which means any of this two goes down I'm in trouble. I'd much prefer to have this configuration doubled and if possible running each on different site (but this is not a requirement).

The first thought was active-active replication, but it seems Postgresql is weak at this and MySQL is not rock solid neither (http://www.mysqlperformanceblog.com/2007/03/15/mysql-myisam-active-active-clustering-looking-for-trouble/). I'm thinking about a VoltDB, which seems like a sexy topic and can be used with Rails it seems.

Log sharing seems like a nice idea, but I'm quite new with this concept and I'm wondering about cost of such solution and how ie. MySQL is going to work with it. And it excludes any way of running the application at two different sites I guess.

I could really use some tips here, so feel free to ask questions and propose solutions. I'll edit this post to add any necessary information.

I'm looking for something like Cacti or Zabbix, but driven from a command line.

I hate clicking, I like scripting and shell automagic. I'm looking for some monitoring suite that would allow me to edit host list, alerts, etc. using Emacs/Vim.

Any ideas/links are appreciated.

I'm from Linux world. In Linux I've been doing firewalls, QoS, dynamic routing, tunnels, ipsec, ipv6 and even written custom networking extensions to kernel.

And now I got some Cisco 681 under my control. I understand the basics but I'm constantly hitting some problems, like:

• it's hard to debug (where is tcpdump?!)
• it's easy to hang/crash the thing (especially when debugging)
• I'd need some pracitcal advices on how to manage the thing (copying running-config via scp and back fails ...)
• most of the documentation I'm finding is IMHO corporate crap (to many words, to less information)

I've been googlin' around and reading things, but still I fell very unskilled when dealing with the thing.

For example I'm trying to add a simple class-based QoS for a working configuration of one of the routers.

I've prepared a simple config:

no ip access-list extended RT_PROTOCOLS                                         
ip access-list extended RT_PROTOCOLS                                            
 permit icmp any any                                                            
 permit udp any any eq 53                                                       
 permit tcp any any eq 22                                                       
!                                                                               
no ip access-list extended HIGH_PROTOCOLS                                       
ip access-list extended HIGH_PROTOCOLS                                          
 permit tcp any any eq 80                                                       
 permit tcp any any eq 443                                                      
!                                                                               
class-map match-any RT_CLASS                                                    
 match access-group name RT_PROTOCOLS                                           
class-map match-any HIGH_CLASS                                                  
 match access-group name HIGH_PROTOCOLS                                         
!                            
policy-map INTERNET_OUT_POLICY                                                  
 class RT_CLASS                                                                 
  bandwidth percent 10                                                          
  random-detect                                                                 
 class HIGH_CLASS                                                               
  bandwidth percent 40                                                          
  random-detect                                                                 
 class class-default                                                            
  fair-queue                                                                    
  bandwidth percent 40                                                          
  random-detect                                                                 
!                                                                               
policy-map INTERNET_OUT_QOS                                                     
 class class-default                                                            
  shape average 8000                                                            
  service-policy INTERNET_OUT_POLICY                                            
!                                                                               
interface FastEthernet4                                                         
 no fair-queue                                                                  
 service-policy output INTERNET_OUT_QOS                                         
!     

and I'm copying it via copy scp://source running-config. The router immediately causes high ping and after around 20 seconds hangs completely.

How do I debug what is wrong?

Any links, docs, advices and tips will be appreciated.