This Consul Server node, in another DC, keeps joining some time after I remove it.

The goal:

A cluster of 5 Consul Servers in DC alpha0, whose KV Store a alpha0 Vault cluster uses as a Storage backend:

• alpha0consulserver1.alpha0
• alpha0consulserver2.alpha0
• alpha0consulserver3.alpha0
• alpha0consulserver4.alpha0
• alpha0consulserver5.alpha0

A cluster of 5 Consul Servers in DC prd0, whose KV Store a prd0 Vault cluster uses as a Storage backend:

• prd0consulserver1.prd0
• prd0consulserver2.prd0
• prd0consulserver3.prd0
• prd0consulserver4.prd0
• prd0consulserver5.prd0

WAN connection is OK. But I am concerned that if they sync their KV stores, this may affect the two separate HashiCorp Vault clusters that each use them as a Back-end.

The problem:

A poorly tested Puppet script I wrote has resulted in one Consul node prd0consulserver5, connecting to another in a different DC, alpha0consulserver1.

I have completely purged and re-installed Consul for prd0consulserver5, but alpha0consulserver1 keeps connecting to it.

Here is an example of one of the configuration files, specifically, the one for alpha0consulserver1.alpha0:

nathan-basanese-zsh8 % sudo cat /etc/consul/config.json
{
    "bind_addr": "192.176.100.1",
    "client_addr": "0.0.0.0",
    "data_dir": "/opt/consul",
    "domain": "consul.basanese.com",
    "bootstrap_expect": 5,
    "enable_syslog": true,
    "log_level": "DEBUG",
    "datacenter": "bts0",
    "node_name": "alpha0consulserver1",
    "ports": {
        "http": 8500,
        "https": 8501
    },
    "recursors": ["192.176.176.240", "192.176.176.241"],
    "server": true,
    "retry_join": ["192.176.100.3", "192.176.100.2", "192.176.100.1"]
}

Here are some relevant logs from prd0consulserver5, but I can post more upon request:

2017/05/26 23:38:00 [DEBUG] memberlist: Stream connection from=192.176.100.1:47239
2017/05/26 23:38:00 [INFO] serf: EventMemberJoin: alpha0consulserver2.alpha0 192.176.100.2
2017/05/26 23:38:00 [INFO] serf: EventMemberJoin: alpha0consulserver1.alpha0 10.240.112.3
2017/05/26 23:38:00 [INFO] consul: Handled member-join event for server "alpha0consulserver2.bts0" in area "wan"
2017/05/26 23:38:00 [INFO] serf: EventMemberJoin: alpha0consulserver3.alpha0 192.176.100.3
2017/05/26 23:38:00 [INFO] consul: Handled member-join event for server "alpha0consulserver1.bts0" in area "wan"
2017/05/26 23:38:00 [INFO] consul: Handled member-join event for server "alpha0consulserver3.bts0" in area "wan"

Eventually, I get to this:

2017/05/26 23:39:02 [DEBUG] memberlist: Initiating push/pull sync with: 192.176.100.2

I shut down the node, as I don't want the keys I write to the KV store on alpha0 nodes to appear on prd0 nodes.

What I have tried so far:

I've tried the following:

https://www.consul.io/api/agent.html#graceful-leave-and-shutdown

I didn't try force-leave since it doesn't work on nodes outside of the configured DC.

I've also tried deregistering ALL prod0 hosts from the alpha0 hosts.

https://www.consul.io/api/catalog.html#deregister-entity

I'm at my wit's end, here, and can't seem to find a way

I've searched it on search engines, using this query and many similar queries: https://duckduckgo.com/?q=totally+deregister+consul+node&t=hc&ia=software

The following two results seemed to have a slightly similar problem, but nothing as simple as keeping a cluster of 5 Consul servers separate from another cluster of 5 Consul Servers.

https://github.com/hashicorp/consul/issues/1188 https://groups.google.com/forum/#!msg/consul-tool/bvJeP1c3Ujs/EvSZoYiZFgAJ

I think this may be handled by the "join_wan": configuration setting, but it doesn't seem to have a way to explicitly turn it off. Plus, that seems like a hack-ey way to fix this problem.

I've also considered IPTables.

Anyway, I feel like there's something missing. I've started digging in to the Raft protocol, but I feel like maybe I've started going off on a tangent in my search. Any guidance appreciated, be it a comment or an answer.

More precisely, how do I keep the prd0 Consul Server Nodes having their own separate KV store and Consul Leader from the alpha0 Consul Server Nodes?

I recently ran puppet module generate, and ended up with a whole lot of crap that looked unnecessary for a simple module. I mean, a lot of minimal modules amount to templating a configuration file or two and letting an open source Puppet module do the heavy lifting.

Puppet's documentation explained most of puppet module generate here:

https://docs.puppet.com/puppet/4.9/modules_fundamentals.html

However, I didn't see anything in there about the Gemfile.

Of what use is the Gemfile generated by puppet module generate?

$ puppet module generate nathan-myapp_consul
$ cd myapp_consul/
$ ls
Gemfile     README.md   Rakefile    examples    manifests   metadata.json   spec

Here are the contents of this file:

source ENV['GEM_SOURCE'] || 'https://rubygems.org'

puppetversion = ENV.key?('PUPPET_VERSION') ? ENV['PUPPET_VERSION'] : ['>= 3.3']
gem 'metadata-json-lint'
gem 'puppet', puppetversion
gem 'puppetlabs_spec_helper', '>= 1.0.0'
gem 'puppet-lint', '>= 1.0.0'
gem 'facter', '>= 1.7.0'
gem 'rspec-puppet'

# rspec must be v2 for ruby 1.8.7
if RUBY_VERSION >= '1.8.7' && RUBY_VERSION < '1.9'
  gem 'rspec', '~> 2.0'
  gem 'rake', '~> 10.0'
else
  # rubocop requires ruby >= 1.9
  gem 'rubocop'
end

Gemfile produced by puppet module generate, what is the point of you?

I’m looking for some information about whether PostGreSQL EDB can provide balance between nodes similarly to the way Oracle RAC does this.

AFAIK, it does not.

The closest question / answer combination I could find to this subject follows:

https://stackoverflow.com/questions/1498793/does-oracle-rac-allow-completely-transparent-failovers-between-nodes

However, that question and its answers at the above link discuss failovers, not load balancing.

I've also reviewed the e-booklet here:

http://www.enterprisedb.com/postgres-plus-edb-blog/gary-carter/comparing-edb-postgres-and-oracle

The ol' internet and a search on serverfault.com have not given me much joy, so far. I keep coming up with generic guidelines or corporate leaflets (without detailed examples of load balancing behavior) on either product.

So, for any of you who might have used PostGreSQL EDB, does it provide load balancing among nodes in a similar way to Oracle RAC?

Do you guys know how one might run r10k properly on a CEntOS 6.7 server with Puppet 3.6? Is that a contradiction in terms?

I checked the r10k source code's list of requirements: https://github.com/puppetlabs/r10k#requirements

It looks like it requires Ruby 1.9. Fair enough.

But...

$ facter rubysitedir rubyversion
rubysitedir => /usr/lib/ruby/site_ruby/1.8
rubyversion => 1.8

Thus begins the ignoble quest for the fabled Ruby.

Third Party Repositories

But most 3rd party repos don't offer Ruby 1.9 for CEntOS 6.7.

• https://dl.iuscommunity.org/pub/ius/stable/CentOS/6/x86_64/repoview/letter_r.group.html :(

• https://dl.fedoraproject.org/pub/epel/6/x86_64/

I failed to properly install The one package I did find, in good ol' Remi's repo, failed due to a lack of the mockbuild user. I would think that'd only be necessary if built from source.

Ruby 'managers'

I installed RVM.

That, however, required glibc = 2.12-1.166.el6_7.7, and my system does not have this particular version available (for good reasons).

Regardless, that would only have worked for specific users. Things could get weird with system users that need to use Ruby if RVM was installed. rbenv, which might have mitigated that issue, gave me the same guff about glibc:

$ mkdir ~/.tmp && export TMPDIR=~/.tmp && rbenv install 1.9.3-p551  ##  How did it come to this...
    Downloading yaml-0.1.6.tar.gz...
-> https://dqw8nmjcqpjn7.cloudfront.net/7da6971b4bd08a986dd2a61353bc422362bd0edcc67d7ebaac68c95f74182749
Installing yaml-0.1.6...

BUILD FAILED (CentOS release 6.7 using ruby-build 20160913)

Inspect or clean up the working tree at /export/home/nger/.tmp/ruby-build.20161117064013.21057
Results logged to /home/nger/.tmp/ruby-build.20161117064013.21057.log

Last 10 log lines:
checking whether build environment is sane... yes
checking for a thread-safe mkdir -p... /bin/mkdir -p
checking for gawk... gawk
checking whether make sets $(MAKE)... yes
checking for gcc... no
checking for cc... no
checking for cl.exe... no
configure: error: in `/home/nger/.tmp/ruby-build.20161117064013.21057/yaml-0.1.6':
configure: error: no acceptable C compiler found in $PATH
See `config.log' for more details

System Ruby

ANYWAY, I then thought that I might be able to get away with using Ruby 1.8, CEntOS 6's system Ruby:

Monkey Patch for 1.8 :)

However, r10k devs later removed support for Ruby 1.8 :(

Puppet Collections

Next, at the behest of commenters, I checked the Puppet Collections documentation.

A. http://yum.puppetlabs.com/el/6/ has nothing for Ruby.

B. The puppet-agent package does not update the system Ruby, as (I thought?) advertised:

HOWEVER, after installing, it looks like this does, indeed, produce a bespoke Ruby just for Puppet. Refer to the answer below for more details.

Is CEntOS 6 even supported by r10k?

Does CEntOS 6 support Ruby 1.9?

I'm close to giving up here, serverfault.com is my 'Hail Mary' play.

I have more options other than actual work, of course. I might just beg the RedHat package maintainers to update their Ruby, or pester certain people (you know who you are) to step up the pace on their CEntOS 7 efforts...

Notes:

Note 1: I did not find much on serverfault about this question. https://serverfault.com/search?q=r10k+on+CEntOS+6

Note 2: I have asked this as part of my research on the following prior question:

How do I add third party puppet modules without committing them to source control?

tl;dr: How do I manage modules for a Puppet Master, like I do using requirements.txt for Python or using Bower for JavaScript?

Here's an example.

Say I want to install the puppetlabs/firewall module, but I don't want to commit its source to source control.

https://forge.puppet.com/puppetlabs/firewall advises the following:

puppet module install puppetlabs-firewall --version 1.8.1

However, this seems... off to me, somehow. I'm not saying code-smell, but maybe my code-nose twitched a little, you know what I'm saying?

Especially since I was really hoping to use the puppetmaster module to get rid of our ticky-tacky little fabric+jenkins+bash+cron mess that currently manages our Puppet Masters.

I'd hoped to find something like a puppetmodule provider for package resources that would allow me to automate Puppet Module installation on our Puppet Master.

Is it as simple as adding all the modules I need as dependencies in the puppetmaster module's modulefile? https://github.com/fsalum/puppet-puppetmaster/blob/master/Modulefile

That feels hacky.

Anyway, I'm used to stuff like requirements.txt in Python and Bower in JavaScript.

BTW, I tried searching the forums for this, but I ended up with answers like the following:

Best way to manage third-party / custom-built software with Puppet?

How to install packages from source code with Puppet?

What does Puppet Master have like requirements.txt or Bower that I can use for Puppet modules?

There seems to be a bit of a "chicken and egg" problem with the passwords to the password managers like Hashicorp Vault for Linux.

While researching this for some Linux servers, someone clever asked, "If we're storing all of our secrets in a secrets storage service, where do we store the access secret to that secrets storage service? In our secrets storage service?"‡

I was taken aback, since there's no point to using a separate secrets storage service if all the Linux servers I'd store the secrets on anyway have its access token.

For example, if I move my secrets to Vault, don't I still need to store the secrets to access Hashicorp Vault somewhere on the Linux server?

There is talk about solving this in some creative ways, and at least making things better than they are now. We can do clever things like auth based on CIDR or password mashups. But there is still that trade-off of security For example, if a hacker gains access to my machine, they can get to vault if the access is based on CIDR.

This question may not have an answer, in which case, the answer is "No, this has no commonly accepted silver bullet solution, go get creative, find your tradeoffs bla bla bla"

I want an answer to the following specific question:

Is there a commonly accepted way that one secures the password to a remote, automated secrets store like Hashicorp Vault on modern Linux servers?

Obviously, plaintext is out of the question.

Is there a canonical answer to this? Am I even asking this in the right place? I considered security.stackexchange.com, too, but this seemed specific to a way of storing secrets for Linux servers. I'm aware that this may seem too general, or opinion based, so I welcome any edit suggestions you might have to avoid that.

‡We laugh, but the answer I get on here may very well be "in vault". :/ For instance, a Jenkins server or something else has a 6-month revokable password that it uses to generate one-time-use tokens, which they then get to use to get their own little ephemeral (session limited) password generated from Vault, which gets them a segment of info.

Something like this seems to be along the same vein, although it'd only be part of the solution: Managing service passwords with Puppet

I'm getting the following error when I run puppet-lint:

$ puppet-lint manifests/*
manifests/init.pp - WARNING: class inheriting from params class on line 72

I had myself a quick search on duckduckgo.com, and got this:

http://puppet-lint.com/checks/class_inherits_from_params_class/

However, our Puppet Agent versions are all 2.7 or later, and our Puppet Masters are all 3.0 or later.

For reference, the init.pp code in question follows:

class myclass (
    $zone = 'top',
    $::myclass::params::base_url,
    $::myclass::params::username,
) inherits myclass::params {
...

The code in params.pp is as follows:

class myclass::params {
    $base_url  = hiera('myclass::base_url','https://beta.tpsreports.com/coversheets/')
    $username = hiera('clap::base_url','prod')
}

Even if the Hiera lookup fails, I still shouldn't be getting errors like this:

err: Could not retrieve catalog from remote server: Error 400 on SERVER: Must pass ::myclass::params::base_url to Class[Myclass] at /etc/puppet/manifests/nodes/beta_servers_0.pp:126 on node beta-web-server-0.tpsreports.com

Now that I've hashed through some of that background, to which I'm more than willing to add, if anyone asks, my questions follow:

1. If my params class will provide parameters even if the hiera lookup somehow fails, why am I getting this error?
2. Do I have to use the horrible workaround (that is, "What you should have done" from the puppet-lint.com link, even though I have a Puppet version higher than 2.6.2 in all cases?

Say I have a quick network partition test that I'd like to run, like disconnecting two halves of a ReDiS cluster from each other, and I want to use IPTables to temporarily disconnect one group of servers from another group.

This is very similar to the question asked on the fedora mailing list:

https://www.redhat.com/archives/rhl-list/2006-January/msg03380.html

If I don't see an EXISTING,RELATED in the output from iptables --list, do I have to worry about this?

The following answer on the fedora mailing list seems to say that yes, existing connections will close if I don't see an EXISTING,RELATED in the output from iptables --list.

https://www.redhat.com/archives/rhl-list/2006-January/msg03396.html

A note for the reflexive flaggers, out there: This question, and, more importantly, its answers, would discuss whether IPTables drops existing connections upon updates to its rules.

So far as I can tell, other questions on this site, on this subject, do not address the differences between existing and attempted connections:

How to close certain TCP/UDP ports (incoming) for ALL networks except listed through IPTABLES

I found most of my research results from the Google search at the page at the following URL link:

https://duckduckgo.com/?q=iptables+close+existing+connections

According to http://redis.io/topics/security, I can set a requirepass value, for example, requirepass foobared, in the Redis configuration files.

However, I don't want this in my configuration files.

Basically, I want to keep this password out of the .conf files, and set it using redis-cli.

################################## SECURITY ###################################

# Require clients to issue AUTH <PASSWORD> before processing any other
# commands.  This might be useful in environments in which you do not trust
# others with access to the host running redis-server.
#
# This should stay commented out for backward compatibility and because most
# people do not need auth (e.g. they run their own servers).
#
# Warning: since Redis is pretty fast an outside user can try up to
# 150k passwords per second against a good box. This means that you should
# use a very strong password otherwise it will be very easy to break.
#
# 2015.12.14 nbasanes: Although I prefer enabling this, I'm not 
# sure it buys us much, since a clear password in a config 
# file is bad for security:
# requirepass foobared

# 2015.12.14  nbasanes: This could be interesting, although 
#   I don't put much weight in security-by-obscurity:
# Command renaming.
#
# It is possible to change the name of dangerous commands in a shared
# environment. For instance the CONFIG command may be renamed into something
# hard to guess so that it will still be available for internal-use tools
# but not available for general clients.
#
# Example:
#
# rename-command CONFIG b840fc02d524045429941cc15f59e41cb7be6c52
#
# It is also possible to completely kill a command by renaming it into
# an empty string:
#
# rename-command CONFIG ""
#
# Please note that changing the name of commands that are logged into the
# AOF file or transmitted to slaves may cause problems.

Some people I work with actually give each Puppet environment its own $modulepath, but also have the common $modulepath.

Please find an example of this below:

[redis]
        modulepath = /usr/share/puppet/modules:/etc/puppet/modules:/etc/puppet/environments/redis/modules/
        manifest=/etc/puppet/environments/redis/manifests/site.pp

I'm not sure that this is wise, but I wanted to ask, "What is the right way to do this?"

I'm open to answers from the perspective of Puppet, and also from the perspective of configuration management in general.

It's actually pretty specific. If I need the Redis module for one environment, the statsd module for another, and an Apache module for all of them, can I handle this by setting module paths? Should I?

I understand that the later version addresses security vulnerabilities:

http://httpd.apache.org/security/vulnerabilities_22.html

Supposedly, http://httpd.apache.org/download.html has a full list of changes at http://httpd.apache.org/[preferred]/httpd/CHANGES_2.2, but apache.org no longer seems to have a CHANGES_2.2 file available.

Anyway, without poring over the release notes, has anyone dealt with this sort of upgrade before to know what configuration differences affect the change between CentOS httpd version 2.2.3-22.el5.centos and CentOS httpd version 2.2.15-39.el6?