Here is my Redis graph from graphite for the past 72 hours displaying current_keys.
as you may notice one instance ( green ) is dropping keys regularly, while other one is ok.. Both of them were configured with Chef and running same Redis version ( redis_version:2.6.5 ) and same config.
Any ideas what could cause keys to drop? Both instances are behind nutcracker/twemproxy ( redis proxy from Twitter https://github.com/twitter/twemproxy ) so should get equal amount of keys..
Installed opscenterd on the separate node outside of the cluster, but within firewall ( aws security group ). Tested all possible ports between agents and opcenter server. No errors in the log..
2013-10-30 01:07:23+0000 [FC_Cluster] INFO: Initializing event storage.
2013-10-30 01:07:23+0000 [FC_Cluster] INFO: Attempting to load all persisted alert rules
2013-10-30 01:07:23+0000 [FC_Cluster] INFO: Done loading persisted alert rules
2013-10-30 01:07:23+0000 [FC_Cluster] INFO: Done initializing event storage.
2013-10-30 01:07:23+0000 [FC_Cluster] INFO: Done loading persisted scheduled job descriptions
2013-10-30 01:07:23+0000 [FC_Cluster] INFO: OpsCenter starting up.
2013-10-30 01:07:23+0000 [] INFO: Finished starting new cluster services for FC_Cluster
2013-10-30 01:08:04+0000 [FC_Cluster] INFO: Agent for ip 10.34.10.185 is version u'3.2.2'
2013-10-30 01:08:04+0000 [FC_Cluster] INFO: Agent for ip 10.32.37.251 is version u'3.2.2'
2013-10-30 01:08:04+0000 [FC_Cluster] INFO: Agent for ip 10.82.226.252 is version u'3.2.2'
The most interesting part that I can see some data in the opscenter UI, when I stop agents, there is no data displayed, when I start - it show up again, but at the same time it shows 0 connected nodes. Storage capacity is even funnier - 3 of 0 nodes..
Any ideas why that could be happening?
I would like to run datastax cassandra opscenter behind Nginx. But it keeps adding opscenter port (8888 by default) to all urls. Is there a config setting to disable that or something?
Here is my super simple nginx config:
server {
server_name opscenter.hostname.com;
location / {
proxy_pass http://127.0.0.1:8888;
}
}
Is there something I'm doing wrong maybe?
I'm not familiar with python, maybe someone could explain what's going on here?
ec2-user@prod-opscenter-01:~ % java -version
java version "1.7.0_45"
Java(TM) SE Runtime Environment (build 1.7.0_45-b18)
Java HotSpot(TM) 64-Bit Server VM (build 24.45-b08, mixed mode)
ec2-user@prod-opscenter-01:~ % python -V
Python 2.6.8
ec2-user@prod-opscenter-01:~ % openssl version
OpenSSL 1.0.1e-fips 11 Feb 2013
And now the error
ec2-user@prod-opscenter-01:~ % sudo /etc/init.d/opscenterd start
Starting Cassandra cluster manager opscenterd
Starting opscenterdUnhandled Error
Traceback (most recent call last):
File "/usr/lib64/python2.6/site-packages/twisted/application/app.py", line 652, in run
runApp(config)
File "/usr/lib64/python2.6/site-packages/twisted/scripts/twistd.py", line 23, in runApp
_SomeApplicationRunner(config).run()
File "/usr/lib64/python2.6/site-packages/twisted/application/app.py", line 386, in run
self.application = self.createOrGetApplication()
File "/usr/lib64/python2.6/site-packages/twisted/application/app.py", line 451, in createOrGetApplication
application = getApplication(self.config, passphrase)
--- <exception caught here> ---
File "/usr/lib64/python2.6/site-packages/twisted/application/app.py", line 462, in getApplication
application = service.loadApplication(filename, style, passphrase)
File "/usr/lib64/python2.6/site-packages/twisted/application/service.py", line 405, in loadApplication
application = sob.loadValueFromFile(filename, 'application', passphrase)
File "/usr/lib64/python2.6/site-packages/twisted/persisted/sob.py", line 210, in loadValueFromFile
exec fileObj in d, d
File "bin/start_opscenter.py", line 1, in <module>
from opscenterd import opscenterd_tap
File "/usr/lib/python2.6/site-packages/opscenterd/opscenterd_tap.py", line 37, in <module>
File "/usr/lib/python2.6/site-packages/opscenterd/OpsCenterdService.py", line 13, in <module>
File "/usr/lib/python2.6/site-packages/opscenterd/ClusterServices.py", line 22, in <module>
File "/usr/lib/python2.6/site-packages/opscenterd/WebServer.py", line 40, in <module>
File "/usr/lib/python2.6/site-packages/opscenterd/Agents.py", line 18, in <module>
exceptions.ImportError: cannot import name _parse
Failed to load application: cannot import name _parse
Maybe there are open source alternatives to monitoring cassandra I should look at?
Thanks a lot
I'm trying to figure out how can I construct the path to templates (or files, since it works the same way) folder, then I could read it and use template resource in the loop, so each template file could trigger notification on change.
I can't use remote_directory because I want to notify services only if template changes.
I also would like to avoid manual template specification, since there could be many files inside those dirs. Plus it would allow us to change just configs in the templates folder without touching recipe.
The main problem is those subdirectories like default, host, host-version and the logic chef goes through to determine correct templates folder. I was thinking maybe there is a method in one of the Chef classes I could call from my custom cookbook to get to the starting point for my logic (loop).
It should be something like this I think:
entry_point = CHEF::...getEntryPointDir
entry_point.glob..
.each do
template fname do
...
end
end
I would appreciate any help!
I'm looking for ideas about chef cookbooks versions management. I know that you pin specific versions in the environment, but I'm not sure how to go about it.
We use librarian-chef which installs 3rd party community books into the cookbooks folder. We never touch those books and just update to more recent versions from time to time.
We also have our custom site-specific cookbooks from where we include community ones (include_recipe).
Theoretically we could specify specific versions of community books our custom books depend on and then set our cookbook versions in the environment config, but the problem is that those community books could rely on some other books without specified versions. And that deep nested dependency could go on an on.
So there is no gurantee that when you upload cookbook to the chef server, it won't break prod, since dependent cookbooks might change as well.
The only solution I can see at the moment is to specify every single cookbook verion we use in the environment config, including community and custom ones. But then I have to go through each cookbook and figure out those versions.
We also do librarian-chef update for time to time and I imagine it might become difficult to track down versions that changed and not to forget to update version in the environmet when time comes.
Please, share your experience and best practices. I'm sure it will be super useful for other people.
I'm curious is it a good strategy to prebuild binaries (install packages) and use chef for config management only. Theoretically it should speed up init process when we need new machines (traffic spike), especially if we use clouds like AWS where we can create our own AMI (images), at the same time it could be not flexible in terms of how other cookbooks work..
I'm looking forward to your thoughts on the question along with best practices, experiences and ideas!
Here my upstart config for solr which is located under /etc/init/solr.conf
# /etc/init/solr.conf
description "Starts Solr server"
respawn
start on startup solr_push
script
echo "Starting solr server..."
cd /opt/solr/solr-4.2.1/example
exec java -Xms50M -Xmx80M -Djava.util.logging.config.file=/etc/solr/logging.properties -Dsolr.solr.home=/etc/solr -jar start.jar >> /var/log/solr/solr-console.log 2>&1
end script
When I update this upstart config, for example changing java options from -Xmx80M to -Xmx70M and then do
sudo restart solr
ps aux | grep solr
I still see -Xm80M over there, even though solr process was started with new process id.
When I do sudo stop solr and then sudo start solr manually - everything looks as expected and I can see new values in the ps aux.
I was under impression that upstart restart does stop and start behind the scenes, but then why results of maunal stop and start differ from restart?
Please advice, maybe I'm doing something incorrectly?
I was wondering if there is a way to control lifetime of the redirects in Nginx?
We would liek to cache 301 redirects in CDN for specific amount of time, let say 20 minutes and the CDN is controlled by the standard caching headers. By default there is no Cache-control or Expires directives with the Nginx redirect. That could cause the redirect to be cached for a really long time. By having specific redirect lifetime the system could have a chance to correct itself, knowing that even "permanent" redirect change from time to time..
The other thing is that those redirects are included from the Server block, which according the nginx specification should be evaluated before locations.
I tried to add add_header Cache-Control "max-age=1200, public"; to the bottom of the redirects file, but the problem is that Cache-control gets added twice - first comes let say from the backend script and the other one added by the add_header directive..
In Apache there is the environment variable trick to control headers for rewrites:
RewriteRule /taxonomy/term/(\d+)/feed /taxonomy/term/$1 [R=301,E=expire:1] Header always set Cache-Control "store, max-age=1200" env=expire
But I'm not sure how to accomplish this in Nginx.
I got 2 domains dev.domain.com and beta.domain.com. At the moment I have one server section (default) with a bunch of specific locations and rules. Now I need to password protect beta.domain.com. Is there a way to do this without creating additional server section and essentially duplicating all my location and other rules?
Generally I would like to know how other people manage complex nginx configurations. Do they just copy sections (duplicate) or include common rules somehow?
I'm running Ubuntu 10.04
I installed kvm and virtual manager and created a guest OS. Everything worked fine until I rebooted my pc. After reboot libvirt was not running despite of init script inside /etc/init.d If I start it manually it works. It's just annoying to do it manually every time. Is there a way I can debug why my libvirt fails to start automatically?
Second question:
As I said I'm using virtual manager to create and run my guest OS. Initially I installed my first os (Windows 7) from .iso file on USB drive, and when I disconnected that drive after guest OS installation, virtual manager complained about missing .iso file. Basically it fails to start without .iso.. If I plug back that USB drive - it works. Is it how it's suppose to be? I was thinking that once you installed your guest OS on a virtual hardrive space, you don't need .iso file any more..
Would appreciate any addition information on that topic.
Thanks a lot!