Questions[puppet-foreman](server)

This is with Puppet 5.5.22 and Foreman 1.22.0-develop. I know this is an outdated Foreman and Puppet installation but until I replace them with new versions (a soon upcoming project) they are what I must deal with.

# puppet agent -tv
Notice: Local environment: 'production' doesn't match server specified node environment 'development', switching agent to 'development'.
Info: Retrieving pluginfacts
Info: Retrieving plugin
Info: Retrieving locales
Info: Loading facts
Notice: Local environment: 'development' doesn't match server specified environment 'test', restarting agent run with environment 'test'
Info: Retrieving pluginfacts
Info: Retrieving plugin
Info: Retrieving locales
Info: Loading facts
Info: Caching catalog for servername.example.com
Info: Applying configuration version '1694623566'
Notice: Applied catalog in 12.93 seconds

I understand that the original local environment 'production' can come either from /etc/puppet/puppet.conf or is the default if no environment specified in that file. In this case it's the latter (no environment specified locally).

The node environment 'development' was set for this host in Foreman (the ENC or external node classifier for this Puppet installation) and is the desired correct environment assignment.

The environment 'test' is the environment that Foreman is setting, broadly and generally, for all of our hosts at this time and is the environment that this particular host was on before I changed it through the Foreman UI to be 'development.' (In other words, 'test' is actually our production environment; the environment called 'production' is not in use at this time.)

I tried consulting the Foreman documentation and didn't find anything about the underpinnings of how environment assignments work; the documentation just amounted to a tour of the UI which is not helpful for troubleshooting.

Likely related, there is another oddity here in that servername.example.com appears twice in Foreman's UI.

If I try to edit the environment for either entry by clicking the "Edit" button, when I then click "Submit" I get "has already been taken" showing in red next to the "Name" field as though I were trying to create a new host entry.

However if I instead select the host with the checkbox to the left from the list view and then go to the top and choose "Select Action" -> "Change Environment", then it appears to work. Both the entries then show "development" in the UI. But the next Puppet run on servername will produce the output shown at the top of this post, and then after that refreshing the Foreman UI will again show one entry with "development" and one entry with "test."

What could be causing this? How can I assign hosts to environments and have it stick?

I'm having a little trouble working with the Global Variables in Foreman. I have a global variable called ALLOW_AD_USERS which I can see in the facts section against the host. I'm trying to use this variable with in the YAML config associated with a host. It just seemed to copy this in like-for-like.

After reading the manual, I would have thought I could add "${$foreman::ALLOW_AD_USERS}" and it auto file the names I have defined within the global variable.

Any help would be great here.

sssd:
  domains: example.com
  config_file_version: 2
  services:
  - nss
  - pam
domain/example.com
  ad_domain: example.com
  krb5_realm: EXAMPLE.COM
  realmd_tags: manages-system joined-with-adcli
  cache_credentials: true
  id_provider: ad
  access_provider: simple
  simple_allow_users:
  - "${$foreman::ALLOW_AD_USERS}"
  ignore_group_members: true
  dyndns_update: true
  ad_gpo_map_permit: "+polkit-1"

We make some use of the Global Parameters in Foreman, and I need to list all hosts, which have the Global Parameter foo set to "bar".

Can I do that with the REST API or via some other method? Going through the list of API-calls, I do not find anything applicable -- is there hope?

Alternatively, I'm thinking of creating a special Puppet-class, which will do nothing but restate the global parameters as its own variables. By making it an exported resource, I may be able to collect the data from all hosts... Seems nasty, however -- is there a better way?

I got a little problem: I installed puppet agent on some nodes. I'm managing the nodes with foreman.

Now some nodes are "out of sync" until I run puppet agent -t on the node manually.

The service is running on the nodes:

root@node1:/var/log# ps aux | grep puppet
root     10079  0.0  0.0   9240   880 pts/0    R+   17:43   0:00 grep puppet
root     31645  0.0  0.2 196188 36780 ?        Ssl  14:17   0:02 /usr/bin/ruby /usr/bin/puppet agent

root@node1:/var/log# service puppet status
[ ok ] agent is running.

What can I do here?

When attempting a "run puppet" from the UI (1.11.0 - thanks for the improved UI speed, btw) on a hostgroup (same config/installs/OS, etc), the result will be a "Failed to apply catalog: Broken pipe - " error. All hosts are Ubuntu Trusty. Here's a syslog output from a UI Puppet Run on 20 nodes:

Apr 14 11:34:27 pn02 puppet-agent[45865]: Retrieving pluginfacts
Apr 14 11:34:27 pn02 puppet-agent[45865]: Retrieving plugin
Apr 14 11:34:28 pn02 puppet-agent[45865]: Loading facts
Apr 14 11:35:15 pn02 puppet-agent[45865]: Caching catalog for pn02.blahblah.org
Apr 14 11:35:22 pn02 puppet-agent[45865]: Failed to apply catalog: Broken pipe - <STDOUT>

However, when running puppet manually from the node, or when the regularly scheduled puppet run executes, the run will complete successfully. Additionally, individual Puppet Runs, as well as runs on, e.g. 5 nodes from the UI also succeed - it's just the attempt to execute on 20 nodes that produces errors.

Any thoughts on next steps to diagnose? Is this network-congestion related? Should my master's hardware be boosted?