Zama Ques's questions

I am trying to install velero on a GKE Cluster from a GCP Compute Engine Host using below steps

https://github.com/vmware-tanzu/velero-plugin-for-gcp

I am installing velero from the VM host using below command

 velero install --provider gcp --plugins velero/velero-plugin-for-gcp:v1.1.0 --bucket ${VELEROBUCKET} --secret-file ./credentials-velero

but it is failing with below error

Error installing Velero. Use `kubectl logs deploy/velero -n velero` to check the deploy logs:
Error creating resource ClusterRoleBinding/velero: clusterrolebindings.rbac.authorization.k8s.io is forbidden: 
User "116865650821658545991" cannot create resource "clusterrolebindings" in API group "rbac.authorization.k8s.io" at the cluster scope: requires one of ["container.clusterRoleBindings.create"] permission(s)  

To resolve this error , I am trying with this

  kubectl create clusterrolebinding cluster-admin-binding --clusterrole cluster-admin --user velero

but this too failing with below error

error: failed to create clusterrolebinding: clusterrolebindings.rbac.authorization.k8s.io is forbidden: User "116865650821658545991" cannot create resource "clusterrolebindings" in API group "rbac.authorization.k8s.io" at the cluster scope: requires one of ["container.clusterRoleBindings.create"] permission(s).

Two queries for the above scenario

1. Is it correct to create clusterrolebinding resource to resolve the error I am seeing while installing velero ?

2. How to identify the corresponding user for "116865650821658545991" as seen in second error and what role needs to be assigned to create clusterrolebinding resource ?

How to change the password for MySQL root user in GCP CloudSQL DB instance with private IP address . I am able to change the password using the below command from Cloud Shell with Public IP

    gcloud sql users set-password root  --instance=testdb --prompt-for-password

but it does not work for instances with Private IP address .

I can use the GCP Console to change the password but the root user is not listed under Users TAB

Console--> CloudSQL--> DB Instance --> Users

Other option seems installing gcloud on a VM in the same network as CloudSQL and change it using below command

gcloud sql users set-password root  --instance=testdb --prompt-for-password

Any other suggestion to change password for root user in a MySQL private DB instance . I need to change the password as initially the DB was created skipping the password.

Also is it more secure to use --host=% option while changing the password with gcloud command.

gcloud sql users set-password root --host=% --instance=testdb  --prompt-for-password

I am looking to understand things regarding Google Kubernetes Cluster backup . I came across this document but this seems to be more on GCP Anthos On premises GKE Cluster .

https://cloud.google.com/anthos/gke/docs/on-prem/archive/1.1/how-to/administration/backing-up

I saw few blogs talking of a GKE feature to create a clone of existing GKE Cluster but I cannot find any option in GCP Console to create new cluster by cloning an existing GKE Cluster.

https://blog.doit-intl.com/google-kubernetes-engine-cluster-migration-with-velero-4a140b018f32

Can somebody please confirm if this cloning feature is still available in GKE or it is deprecated ?

Apart from Cloning a GKE Cluster , we need to take backup of Cluster resources and PersistentVolumes. It seems Veloro is a useful tool for this and it is GKE aware.

https://velero.io/

I am looking for further suggestions regarding GKE Cluster backup which takes care of both Cluster resources and persistent volumes. Any recommendation/best practises from Google on GKE backup

What is the impact for CLoud functions during deployment ? Does the function become unavailable during deployment or any trigger during this deployment time will fail ?

I can deploy using GCP Console or by using below gcloud command

gcloud functions deploy HelloWorld --source golang-samples-master/functions/codelabs/gopher --update-labels env=dev,name=automate --trigger-http --runtime go111

When we change labels for existing CLoud Run serivces using gcloud command , it does a deployment as well as revision change

      $gcloud run services update test --update-labels env=prod,test=test1
      ✓ Deploying... Done.                                         
      ✓ Creating Revision...
      ✓ Routing traffic...
      Done.
      Service [test1] revision [test1-00003-wik] has been deployed and is serving 100 percent of traffic at https://test1-ukliefksia-uc.a.run.app

Does it cause downtime during label only change without any code change ? It seems traffic is routed and so no downtime for application . Please confirm

Why it is not allowing my IP to access Apache Load Balancer

  <Location /balancer-manager>
   SetHandler balancer-manager
   Require host demo-002
   Order deny,allow
   Deny from all
   Allow from 127.0.0.1 10.201.165.186
  </Location>

Seeing the following error

 [access_compat:error] [pid 22702] [client 10.201.165.86:59524] AH01797: client denied by server configuration: /var/www/html/balancer-manager

Apache version: Apache/2.4.6

Please suggest .

We upgraded from httpd 2.4.23 to 2.4.32 . After upgrade, we noticed mpm module got changed from preforx to event in 2.4.23

    $ /apps/httpd.2.4.23/bin/httpd -M | grep mpm
    mpm_prefork_module (static
    $ /apps/httpd-2.4.34/bin/httpd -M  | grep mpm
    mpm_event_module (static)    

But the memory utilization is too high when used with wordpress sites for event module . OOM are getting invoked because of high memory utilization

     PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
     25624 apachewp  20   0 3285m 1.1g  72m S  0.0  6.9   1:30.64 httpd
     25487 apachewp  20   0 3278m 1.1g  65m S  0.0  6.8   2:36.70 httpd
     25549 apachewp  20   0 3230m 1.0g  86m S  0.0  6.7   5:22.49 httpd
     25568 apachewp  20   0 3284m 1.0g  61m S  0.0  6.6   0:40.90 httpd
     25489 apachewp  20   0 3182m 1.0g 101m S 21.2  6.6  13:12.10 httpd
     25586 apachewp  20   0 3236m 1.0g  80m S  0.0  6.6   1:59.98 httpd
     25603 apachewp  20   0 3126m 989m 101m S  0.0  6.2   8:51.16 httpd
     25639 apachewp  20   0 3198m 975m  63m S  0.0  6.1   0:50.38 httpd
     25614 apachewp  20   0 3158m 956m  69m S  0.0  6.0   0:41.13 httpd
     25650 apachewp  20   0 3098m 889m  64m S  0.0  5.6   1:13.60 httpd
     25515 apachewp  20   0 3046m 868m  75m S  0.0  5.4   4:05.48 httpd
     25538 apachewp  20   0 3057m 831m  51m S  0.0  5.2   0:26.23 httpd
     25530 apachewp  20   0 2933m 704m  53m S  0.0  4.4   0:19.28 httpd
     25485 apachewp  20   0 2777m 548m  48m S  0.0  3.4   0:11.02 httpd
     25497 apachewp  20   0 2757m 498m  45m S  0.0  3.1   0:11.90 httpd
     25486 apachewp  20   0 2600m 372m  51m S  0.0  2.3   0:07.74 httpd
     24866 apache    20   0 2462m  91m  18m S  0.3  0.6   0:01.87 httpd
     24924 apache    20   0 2369m  80m 8436 S  0.0  0.5   0:02.98 httpd
     24952 apache    20   0 2368m  80m 6476 S  0.0  0.5   0:01.30 httpd

Following is our event reading

   <IfModule mpm_event_module>
   StartServers             15
   MinSpareThreads         25
   MaxSpareThreads         75
   ThreadsPerChild         25
   MaxRequestWorkers      100
   MaxConnectionsPerChild   0

Memory utlization was not so high when used with prefork module but it increased with event module . PHP version is 7.2 . Please suggest workarounds to reduce the memory footprint.

I can directly access Host A ( 10.100.64.112) from my HomePC but cannot directly access Host B (172.88.3.31) . To access Host B , I need to first ssh to Host A and then to Host B .

To directly access Host B , I tried to set up an SSH tunnel using local port forwarding with the below command on Host A

 Host A # ssh  [email protected] -L 4420:172.88.3.31:22

Now I tried to validate whether the tunnel is working or not

 Host A # ssh 10.100.64.112 -p 4420
 [email protected]'s password:

It works fine and it takes me to Host B

But If I access directly the tunnel from my HOmePC , it does not work

    $  ssh 10.100.64.112 -p 4122
    ssh: connect to host 10.100.64.112 port 4122: Connection refused

On checking with netstat , it shows the following

$ sudo netstat -an | grep 4420
tcp        0      0 127.0.0.1:4420              0.0.0.0:*                   LISTEN
tcp        0      0 ::1:4420                    :::*                        LISTEN

Please suggest so that I connect to Host B directly.

I am getting the following error during directory creation using file resource

Error: Cannot create /data/tomcat/${dirsname}/conf; parent directory /data/tomcat/${dirsname} does not exist
Error: /Stage[main]/App/File[/data/tomcat/${dirsname}/conf]/ensure: change from absent to directory failed: Cannot create /data/tomcat/${dirsname}/conf; parent directory /data/tomcat/${dirsname} does not exist`

Error: Cannot create /data/tomcat/$dirsname/config; parent directory /data/tomcat/$dirsname does not exist
Error: /Stage[main]/App/File[/data/tomcat/$dirsname/config]/ensure: change from absent to directory failed: Cannot create /data/tomcat/$dirsname/config; parent directory /data/tomcat/$dirsname does not exist`

But technically the directories are getting created as can can be seen from the puppet command output

Notice: /Stage[main]/App/File[/data/tomcat/$dirsname/conf/]/ensure: created
Notice: /Stage[main]/App/File[/data/tomcat/jacplus8/conf/context.xml]/ensure: defined content as '{md5}ba798c2de6dd554035c05352891a0d07'
Notice: /Stage[main]/App/File[/data/tomcat/jacplus8/conf/server.xml]/ensure: defined content as '{md5}6298e53557ead6855cef9bcbb082a074'`

Notice: /Stage[main]/App/File[/data/tomcat/$dirsname/config/]/ensure: created
Notice: /Stage[main]/App/File[/data/tomcat/jacplus8/config/runtime.settings]/ensure: defined content as '{md5}2e2596f0f9107c8cfc783da3d5004dae'`

I checked the directory structure , the directories are available but not sure why it is showing the error messages. My manifests rule are as follows

class app {
$dirsname=myapp8

file { 'directory':
     path => "/data/tomcat/${dirsname}/",
     ensure => "directory",
     owner => "root",
     group => "root",
    }

 file { '/data/tomcat/${dirsname}/conf':
    ensure => "directory",
    owner => "root",
    group => "root",
    require => File['directory'],
  }

 file { '/data/tomcat/$dirsname/config':
    ensure => "directory",
    owner => "root",
    group => "root",
    require => File['directory'],
   } 

  file { '/data/tomcat/$dirsname/conf/':
    path => "/data/tomcat/$dirsname/conf/",
    ensure => "present",
    recurse => "true",
    source => "puppet:///modules/conf/"
   }

  file { '/data/tomcat/$dirsname/config/':
    path => "/data/tomcat/$dirsname/config/",
    ensure => "present",
    recurse => "true",
    source => "puppet:///modules/config/"
   }
  }

I am trying to create multiple directories and then copy files to each of the directories . For that I have created the following resource

$dirs=myapp

$appdirs = [  "/data/tomcat/$dirs/conf", "/data/tomcat/$dirs/config" ]


 file { $appdirs:
 ensure => "directory",
 owner => "root",
 group => "root",
}

file { "Copy Directory":
path => "/data/tomcat/$dirs/conf",
ensure => "present",
recurse => "true",
source => "puppet:///modules/tomcat8/conf/"
 }
}

But I am getting the error already defined as shown below

 Error: Could not retrieve catalog from remote server: Error 500 on SERVER: Server Error: 
 Evaluation Error: Error while evaluating a Resource Statement, Cannot alias File[Copy Directory] to
 ["/data/tomcat/jacplus8/conf"] at /etc/puppetlabs/code/environments/production/manifests/classes/app.pp:20; 
 resource ["File", "/data/tomcat/jacplus8/conf"] already declared at /etc/puppetlabs/code/environments/production
 /manifests/classes/app.pp:14 at /etc/puppetlabs/code/environments/production/manifests/classes/app.pp:20:1
 on node Node-003.example.com

I need to copy files using the second resource file { "Copy Directory":} to the destination directory after it is created using the first resource file { $appdirs:} but it is giving me already defined error although the resource names are different . I am looking for workarounds so that I can create the directories and then copy files in it .

I am trying to create multiple directories using the following manifests

class app {
$dirs = app8
$appdirs = ['/data/tomcat/app8/conf', '/data/tomcat/app8/config', '/data/tomcat/app8/libs', '/data/tomcat/app8/deploy', '/data/tomcat   /app8/webapps', '/data/tomcat/app8/temp', '/data/tomcat/app8/work']
file { "/data/tomcat/$dirs":
      path => "/data/tomcat/$dirs",
      ensure => directory,
      owner => root,
      group => root,
    }
file { "$appdirs":
  path => '$appdirs',
  ensure => directory,
  owner  => root,
  group  => root,
  mode   =>  0644,
  }
}

But when I execute it is failing with the below error

 # puppet agent --verbose --no-daemonize --onetime
 Info: Using configured environment 'production'
Info: Retrieving pluginfacts
Info: Retrieving plugin
Info: Caching catalog for node-003.wiley.com
Error: Failed to apply catalog: Parameter path failed on File[[/data/tomcat/app8/conf, /data/tomcat/app8/config, /data/tomcat/app8/libs, /data/tomcat/app8/deploy, /data/tomcat/app8/webapps, /data/tomcat/app8/temp, /data/tomcat/app8/work]]: File paths must be fully qualified, not '$appdirs' at /etc/puppetlabs/code/environments/production/manifests/classes/app.pp:15

Please suggest how to resolve the issue

We had following below iptables rules that exist in our web front-end boxes to prevent IP Spoofing:

-A INPUT -s 255.0.0.0/8 -j LOG --log-prefix "Spoofed source IP"
-A INPUT -s 255.0.0.0/8 -j DROP
-A INPUT -s 0.0.0.0/8 -j LOG --log-prefix "Spoofed source IP"
-A INPUT -s 0.0.0.0/8 -j DROP

We want to add below rules now to further harden IP Spoofing prevention

 -A INPUT -s 224.0.0.0/3 -j LOG --log-prefix "Spoofed source IP"
 -A INPUT -s 255.0.0.0/8 -j DROP
 -A INPUT –s 169.254.0.0/16 -j LOG --log-prefix "Spoofed source IP"
 -A INPUT -s 169.254.0.0/16 -j DROP
 -A INPUT –s 240.0.0.0/5 -j LOG --log-prefix "Spoofed source IP"
 -A INPUT -s 240.0.0.0/5 -j DROP

Do you suggest adding above rules in a production box running Apache httpd as a reverse proxy? This production box is behind a F5 load balancer.

Also, do we need to enable the below kernel parameters for the above rules to work effectively?

           net.ipv4.conf.all.rp_filter=1
           net.ipv4.conf.all.log_martians=1
           net.ipv4.conf.default.log_martians=1

We have configured 12 servers with Puppet/MCollective . All the servers are rspodning to mco ping . But we had a patch update for OS and after that one of the node is not responding.

        $ sudo tail -f mcollective.log
           I, [2016-11-28T17:47:53.493341 #25669]  INFO -- : activemq.rb:129:in `on_connectfail' TCP Connection to stomp+ssl://mcollective@jw-node1-x1:61614 failed on attempt 4620
       I, [2016-11-28T17:48:23.495747 #25669]  INFO -- : activemq.rb:139:in `on_ssl_connecting' Establishing SSL session with stomp+ssl://mcollective@node1-x1:61614
       E, [2016-11-28T17:48:23.635431 #25669] ERROR -- : activemq.rb:149:in `on_ssl_connectfail' SSL session creation with stomp+ssl://mcollective@node1-x1:61614 failed: hostname "node1-x1" does not match the server certificate

Puppet version is as follows:

    $ /opt/puppetlabs/bin/puppet --version
      4.5.2

I have compared they keys and cerrtifactes which has been exchanged between nodes, but they are same in terms of permission , size and contents of key file.

Please suggest what went wrong after reboot.

We are migrating to later versions of puppet.

New Version:

# puppet --version
4.5.2

Existing Version:

#puppet-3.8.7

Our existing site.pp is as follows:

#A default site.pp to do a quick test run
import "../classes/*"
import "../nodes/*"
   file { 'testfile':
       path => '/home/test/testfile',
       ensure => present,
       mode => 0755,
       content => "A test file to check a different manifestdir" ,
      }
 Exec { path => ["/bin" , "/sbini/", "/usr/bin" , "/usr/sbin/"]  }

Now since we are migrating to Puppet 4, i think below import function to include multiple pp files will not work here in Puppet 4

import "../classes/*"
import "../nodes/*" 

If I am not wrong then I can copy all manifests under classes and nodes directory to the below manifest directory

/apps/puppetlabs/code/environments/production/manifests/site.pp          

Please suggest how to update the manifests to the later versions of Puppet or there is no compatibility issues ?

My Puppet Server Environment setting is as follows

[master]
vardir = /opt/puppetlabs/server/data/puppetserver
logdir = /data/log/puppetlabs/puppetserver
rundir = /var/run/puppetlabs/puppetserver
pidfile = /var/run/puppetlabs/puppetserver/puppetserver.pid
codedir = /data/puppetlabs/code

#  puppet master --configprint manifest
/data/puppetlabs/code/environments/production/manifests

On Agents it is as follows

server = jw-host
environment = production

But --configprint is showing a different location on agent nodes

# puppet master --configprint manifest
/etc/puppetlabs/code/environments/production/manifests

It is not getting changed to /data because of which my manifests are not getting applied .

Versions are as follows

 # puppetserver  --version
  puppetserver version: 2.4.0
 # puppet  --version
  4.5.2

What is the issue here ?

We are using a shared Puppet Server to manage both QA and DEV environments . The default configurations Puppet Server uses is Production environment

# puppet master --configprint all | grep production

environment = production
manifest = /data/puppetlabs/code/environments/production/manifests
modulepath = /data/puppetlabs/code/environments/production/modules:/data/puppetlabs/code/modules:/opt/puppetlabs/puppet/modules

I want to enable two more environments DEV and QA. For this I will copy the above directories and rename to qa and development environment respectively on Puppet Server. What configuration items to enable in puppet.conf to enable the new environments in Puppet Server. If I create multiple entries as in example above will it work . Please suggest .

I did a Puppet/MCollective setup using four nodes.

  # mco ping
  Node-010.test.com                   time=107.58 ms
  Node-003.test.com                   time=110.68 ms
  Node-009.test.com                   time=114.42 ms
  Node-002.test.com                   time=116.49 ms

#mco puppet runonce works fine but is not applying the manifests .

  # mco puppet runonce
  * [ ============================================================> ] 4 / 4
  Finished processing 4 / 4 hosts in 151.61 ms

My manifests configuration is as follows:

   # puppet master --configprint manifest
   /etc/puppetlabs/code/environments/production/manifests

My Site.pp is as follows

  # cat /etc/puppetlabs/code/environments/production/manifests/site.pp
node default {
}
node 'Node-002.test.com'{
file {"/tmp/helloworld.txt":
      ensure => file,
      owner  => 'root',
      group  => 'root',
      mode   => '0644',
      content =>" Hi ",
   }
  }

Site.pp is not getting applied .

From the log file , it seems it is reading from cache and applying for nodes for which no rule exists .

  ` # tail -f /var/log/puppetlabs/puppetserver/puppetserver.log
2016-06-21 20:21:06,355 INFO  [qtp1367105977-65] [puppet-server] Puppet    Compiled catalog for Node-009.test.com in environment qa in 0.04 seconds
2016-06-21 20:21:08,223 INFO  [qtp1367105977-65] [puppet-server] Puppet Caching node for Node-002.test.com
2016-06-21 20:21:08,618 INFO  [qtp1367105977-69] [puppet-server] Puppet Caching node for Node-002.test.com 

Am i hitting some bug or missing some configuration item ?

I have written the following snippet below using user resource in puppet to create unix users .

   node 'node2.example.com','node3.example.com'{
  user {
  'askar':
  ensure  => 'present',
  managehome => 'true',
  comment => 'man Home',
  home    => '/home/askar',
  shell   => '/bin/bash',
  expiry  => '2016-03-22',
  password => '$1$cs1j/t.D$4Q2Ocr0pulyNTUx/',
  password_min_age => '30',
  password_max_age => '60',
 }
exec {
'chage':
    path => '/usr/bin/',
    command => 'chage -d 0 askar',
  } 
}

It works fine for me, but the issue for me, I need to create 10 to 20 users. How can i reuse the above snippet to create 10 users. I am new to puppet , so pointers will be of great help.

I am trying to configure IPMI in my HP ProLiant Servers. Configuration looks successful , but somewhat confused with the 'Administrator' user account .

Following are some information on IPMI in one of my server:

# ipmitool user list 2
ID  Name             Callin  Link Auth  IPMI Msg   Channel Priv Limit
1   Administrator    true    false      true       ADMINISTRATOR
2   admin             true    false      true       USER
3   (Empty User)     true    false      false      NO ACCESS
4   (Empty User)     true    false      false      NO ACCESS
______________________________

]# ipmitool channel getciphers ipmi 2 
ID   IANA    Auth Alg        Integrity Alg   Confidentiality Alg
0    N/A     none            none            none           
1    N/A     hmac_sha1       none            none           
2    N/A     hmac_sha1       hmac_sha1_96    none           
3    N/A     hmac_sha1       hmac_sha1_96    aes_cbc_128    

The confusion is regarding the password for Administrator Account. Do I need to set the password for Administrator account or by default if any password is Set ? For HP Servers that supports ILO , i think it comes with A random, eight-character, alphanumeric string which is printed on a little cardboard card attached to the server . Is this is the password for Administrator Account ? .

Also One more clarification needed regarding the 'Administrator' account is it disabled by default for logging to the IPMI enabled device remotely. I am getting the following error.
=============================
#ipmitool -I lanplus -H 192.168.2.153 -U Administrator sel elist Password:
Error: Unable to establish IPMI v2 / RMCP+ session
Get SEL Info command failed
============================
But if I try to connect using the 'admin' user that I created it works fine
======================
# ipmitool -I lanplus -H 192.168.2.153 -U admin -L USER sel elist
Password:
SEL has no entries
======================
Please clarify .

I am trying to configure Ethernet bonding for our servers . Both the Ethernet interface in my server are connected to two different network switches physically.

Looks like my Ethernet bonding configuring is OK , but I am seeing packet loss after configuring bonding . Sometimes the packet loss is around 90% .

My /proc/net/bonding/bond0 looks like this

[root@ ~]# cat /proc/net/bonding/bond0 

Ethernet Channel Bonding Driver: v3.6.0 (September 26, 2009) 

Bonding Mode: adaptive load balancing 
Primary Slave: None 
Currently Active Slave: eth0 
MII Status: up 
MII Polling Interval (ms): 0 
Up Delay (ms): 0 
Down Delay (ms): 0 

Slave Interface: eth0 
MII Status: up 
Speed: 1000 Mbps 
Duplex: full 
Link Failure Count: 0 
Permanent HW addr: e4:e1:5b:d0:11:10 
Slave queue ID: 0 

Slave Interface: eth1 
MII Status: up 
Speed: 1000 Mbps 
Duplex: full 
Link Failure Count: 0 
Permanent HW addr: e4:e1:5b:d0:11:14 
Slave queue ID: 0 

[root@portaldb1 ~]# cat /sys/class/net/bond0/bonding/mode 
balance-alb 6 

# cat /sys/class/net/bond0/bonding/miimon 
0 

OS I tried are RHEL 6.2 and CentOS 5.5 , seeing the packet drops in both OS

What I am missing in the configuration ? I tried connecting both the interface to the same switch , but still seeing the packets drops . Any help/suggestions to resolve the issue will be hughly appreciated .