Questions[devops](server)

I’m quite new to devops/ci/cd, so bear with me.

Currently, I’m getting a puppet manifest configuration set up for docker image verification. Let me try to layout it out:

I have a puppet master server with the following configuration:

• Puppetmaster version: 5.5.10-4ubuntu3
• Both of the servers are linux 20.04
• All hosted on AWS

/etc/puppet/code/environments/production/manifests/site.pp

node default {
    include 'docker'
    docker::image { 'jorgemauriciodev/ubuntu-dockerfile-dev-v1': }
}

Every time I try a new verification command in the .pp, I restart the puppet master server with the following command: sudo systemctl restart puppet-master

I’ve installed a module called garethr-docker on both of my servers – the agent and the master.

On the slave / agent server, I have this image docker image present and running: jorgemauriciodev/ubuntu-dockerfile-dev-v1 For the first step, I just want to check if the image exists in my slave server. Later on, I’ll figure out to make sure it’s running or built with the Dockerfile.

On the slave / agent server, I run: sudo puppet agent –test

And I receive the following error message:

Error: Could not retrieve catalog from remote server: Error 500 on SERVER: Server Error: Evaluation Error: Error while evaluating a Function Call, Could not find class ::docker for ip-123-123-123-123.us-east-2.compute.internal (file: /etc/puppet/code/environments/production/manifests/site.pp, line: 2, column: 5) on node ip-123-123-123-123.us-east-2.compute.internal Warning: Not using cache on failed catalog Error: Could not retrieve catalog; skipping run

Anyone has any idea on what I’m getting wrong with the puppet manifest file?

Edit 1:

I did some changes that were suggested. Now, my file has a diferent name and changed the content: /etc/puppet/code/environments/production/manifests/init.pp

include 'docker'
class { 'docker':
  version => 'latest',
}
docker::image { 'jorgemauriciodev/ubuntu-dockerfile-dev-v1': }

Still returns me that same error message.

Here´s the full message:

/usr/lib/ruby/vendor_ruby/puppet/util.rb:461: warning: URI.escape is obsolete
/usr/lib/ruby/vendor_ruby/puppet/util.rb:461: warning: URI.escape is obsolete
/usr/lib/ruby/vendor_ruby/puppet/util.rb:461: warning: URI.escape is obsolete
/usr/lib/ruby/vendor_ruby/puppet/util.rb:461: warning: URI.escape is obsolete
/usr/lib/ruby/vendor_ruby/puppet/util.rb:315: warning: deprecated Object#=~ is called on Puppet::Transaction::Report; it always returns nil
/usr/lib/ruby/vendor_ruby/puppet/util.rb:461: warning: URI.escape is obsolete
/usr/lib/ruby/vendor_ruby/puppet/util.rb:461: warning: URI.escape is obsolete
/usr/lib/ruby/vendor_ruby/puppet/util.rb:461: warning: URI.escape is obsolete
/usr/lib/ruby/vendor_ruby/puppet/util.rb:315: warning: deprecated Object#=~ is called on Puppet::Transaction::Report; it always returns nil
/usr/lib/ruby/vendor_ruby/puppet/util.rb:461: warning: URI.escape is obsolete
Info: Using configured environment 'production'
Info: Retrieving pluginfacts
/usr/lib/ruby/vendor_ruby/puppet/util.rb:461: warning: URI.escape is obsolete
/usr/lib/ruby/vendor_ruby/puppet/util.rb:315: warning: deprecated Object#=~ is called on Puppet::Transaction::Report; it always returns nil
/usr/lib/ruby/vendor_ruby/puppet/util.rb:461: warning: URI.escape is obsolete
/usr/lib/ruby/vendor_ruby/puppet/indirector/request.rb:272: warning: URI.unescape is obsolete
/usr/lib/ruby/vendor_ruby/puppet/util.rb:461: warning: URI.escape is obsolete
Info: Retrieving plugin
/usr/lib/ruby/vendor_ruby/puppet/util.rb:461: warning: URI.escape is obsolete
/usr/lib/ruby/vendor_ruby/puppet/util.rb:315: warning: deprecated Object#=~ is called on Puppet::Transaction::Report; it always returns nil
/usr/lib/ruby/vendor_ruby/puppet/util.rb:461: warning: URI.escape is obsolete
/usr/lib/ruby/vendor_ruby/puppet/indirector/request.rb:272: warning: URI.unescape is obsolete
/usr/lib/ruby/vendor_ruby/puppet/util.rb:461: warning: URI.escape is obsolete
Info: Retrieving locales
/usr/lib/ruby/vendor_ruby/puppet/util.rb:461: warning: URI.escape is obsolete
/usr/lib/ruby/vendor_ruby/puppet/util.rb:315: warning: deprecated Object#=~ is called on Puppet::Transaction::Report; it always returns nil
/usr/lib/ruby/vendor_ruby/puppet/util.rb:461: warning: URI.escape is obsolete
/usr/lib/ruby/vendor_ruby/puppet/indirector/request.rb:272: warning: URI.unescape is obsolete
/usr/lib/ruby/vendor_ruby/puppet/util.rb:461: warning: URI.escape is obsolete
/usr/lib/ruby/vendor_ruby/puppet/util.rb:461: warning: URI.escape is obsolete
Error: Could not retrieve catalog from remote server: Error 500 on SERVER: Server Error: Evaluation Error: Error while evaluating a Function Call, Could not find class ::docker for ip-172-31-21-116.us-east-2.compute.internal (file: /etc/puppet/code/environments/production/manifests/init.pp, line: 1, column: 1) on node ip-172-31-21-116.us-east-2.compute.internal
Warning: Not using cache on failed catalog
Error: Could not retrieve catalog; skipping run
/usr/lib/ruby/vendor_ruby/puppet/file_system/uniquefile.rb:126: warning: $SAFE will become a normal global variable in Ruby 3.0
/usr/lib/ruby/vendor_ruby/puppet/util.rb:461: warning: URI.escape is obsolete
/usr/lib/ruby/vendor_ruby/puppet/file_system/uniquefile.rb:126: warning: $SAFE will become a normal global variable in Ruby 3.0

Edit 2:

I installed further modules on master server and slave/agent server with this command:

sudo puppet module install puppetlabs-docker --version 4.1.2

And it´s still returning me an error message, but now seems that it´s a different one. So, probably a syntax issue now.

Here´s the important part of the message.

Error: Could not retrieve catalog from remote server: Error 500 on SERVER: Server Error: Evaluation Error: Error while evaluating a Resource Statement, Duplicate declaration: Class[Docker] is already declared; cannot redeclare (file: /etc/puppet/code/environments/production/manifests/init.pp, line: 3) (file: /etc/puppet/code/environments/production/manifests/init.pp, line: 3, column: 1) on node ip-123-123-123-123.us-east-2.compute.internal
Warning: Not using cache on failed catalog
Error: Could not retrieve catalog; skipping run

Besides this message, it´s still returning me those warnings, but the list is much bigger. However, I don´t think it´s a concern at this time.

Given this block of pseudo code from Terraform:

resource "null_resource" {
    provisioner "local-exec" {
        command = "echo hello"
        interpreter = local.os == "Windows" ? ["PowerShell", "-Command"] : ["bash"]
}

How would I get this to run in Powershell on Windows and bash on Linux?

I saw few links where I can tag my docker image using ${Build.SourceVersion} in azure devops pipeline.

But it is using the complete ID of the commit.

But I want to use only the short ID.

I mean this (2cc7968) instead of this (2cc79689fc29ad69698d3062688e2a650da62b8e)

How to get this?

My pipeline:

# Deploy to Azure Kubernetes Service
# Build and push image to Azure Container Registry; Deploy to Azure Kubernetes Service
# https://docs.microsoft.com/azure/devops/pipelines/languages/docker

trigger:
  - master

resources:
  - repo: self

variables:
  # Container registry service connection established during pipeline creation
  dockerRegistryServiceConnection: "685f0716-8b46-436e-8d2a-3d0ff987fce9"
  imageRepository: "azuredevopssampleapp"
  containerRegistry: "aksdevopsacrtesting.azurecr.io"
  dockerfilePath: "**/Dockerfile"
  tag: "$(Build.BuildId)"
  imagePullSecret: "aksdevopsacrtesting458647f2-auth"

  # Agent VM image name
  vmImageName: "ubuntu-latest"

stages:
  - stage: Build
    displayName: Build stage
    jobs:
      - job: Build
        displayName: Build
        pool:
          vmImage: $(vmImageName)
        steps:
          - task: Docker@2
            displayName: Build and push an image to container registry
            inputs:
              command: buildAndPush
              repository: $(imageRepository)
              dockerfile: $(dockerfilePath)
              containerRegistry: $(dockerRegistryServiceConnection)
              tags: |
                $(tag)
          - upload: pipeline_content/manifests
            artifact: manifests

I need to get only the nearest subdirectory and file name from the file path as follows.

Ex:-

(1)E:\Dump\DumpText-1.txt - Dump\DumpText-1.txt

(2)E:\Dump\SubDump1\DumpText-1.txt - SubDump1\DumpText-1.txt

(3)E:\Dump\SubDump3\Sub_SubDump1\DumpText-1.txt -Sub_SubDump1\DumpText-1.txt

My PowerShell Script as follows:

Get-ChildItem $directoryPath -Recurse | ForEach-Object -Process {if (!$_.PSIsContainer) {$_.FullName; $_.FullName -replace "^.*?\\"; " "}}

I am running this template in cloud formation to create auto scaling in ubuntu-18.04 machine and it will rollback the all instances automatically that is created by this template and giving such error like.

Received 0 SUCCESS signal(s) out of 1. Unable to satisfy 100% MinSuccessfulInstancesPercent requirement The following resource(s) failed to create: [WebServerGroup]. . Rollback requested by user.

{
    "AWSTemplateFormatVersion": "2010-09-09",

    "Parameters": {
        "VpcId": {
            "Type": "AWS::EC2::VPC::Id",
            "Description": "VpcId of your existing Virtual Private Cloud (VPC)",
            "ConstraintDescription": "must be the VPC Id of an existing Virtual Private Cloud."
        },
        "Subnets": {
            "Type": "List<AWS::EC2::Subnet::Id>",
            "Description": "The list of SubnetIds in your Virtual Private Cloud (VPC)"
        },
        "InstanceType": {
            "Description": "WebServer EC2 instance type",
            "Type": "String",
            "Default": "t2.small",
            "AllowedValues": [
                "t1.micro",
                "t2.nano",
                "t2.micro",
                "t2.small",
                "t2.medium",
                "t2.large",
                "m1.small",
                "m1.medium",
                "cg1.4xlarge"
            ],
            "ConstraintDescription": "must be a valid EC2 instance type."
        },
"WebServerCapacity": {
      "Default": "2",
      "Description": "The initial number of WebServer instances",
      "Type": "Number",
      "MinValue": "1",
      "MaxValue": "10",
      "ConstraintDescription": "must be between 1 and 10 EC2 instances."
    },
        "KeyName": {
            "Description": "The EC2 Key Pair to allow SSH access to the instances",
            "Type": "AWS::EC2::KeyPair::KeyName",
            "ConstraintDescription": "must be the name of an existing EC2 KeyPair."
        },
        "SSHLocation": {
            "Description": "The IP address range that can be used to SSH to the EC2 instances",
            "Type": "String",
            "MinLength": "9",
            "MaxLength": "18",
            "Default": "0.0.0.0/0",
            "AllowedPattern": "(\\d{1,3})\\.(\\d{1,3})\\.(\\d{1,3})\\.(\\d{1,3})/(\\d{1,2})",
            "ConstraintDescription": "must be a valid IP CIDR range of the form x.x.x.x/x."
        }
    },

    "Resources": {
      "WebServerScaleUpPolicy": {
            "Type": "AWS::AutoScaling::ScalingPolicy",
            "Properties": {
                "AdjustmentType": "ChangeInCapacity",
                "AutoScalingGroupName": {
                    "Ref": "WebServerGroup"
                },
                "Cooldown": "60",
                "ScalingAdjustment": 1
            }
        },
        "WebServerScaleDownPolicy": {
            "Type": "AWS::AutoScaling::ScalingPolicy",
            "Properties": {
                "AdjustmentType": "ChangeInCapacity",
                "AutoScalingGroupName": {
                    "Ref": "WebServerGroup"
                },
                "Cooldown": "60",
                "ScalingAdjustment": -1
            }
        },
        "CPUAlarmHigh": {
            "Type": "AWS::CloudWatch::Alarm",
            "Properties": {
                "AlarmDescription": "Scale-up if CPU > 70% for 5 minutes",
                "MetricName": "CPUUtilization",
                "Namespace": "AWS/EC2",
                "Statistic": "Average",
                "Period": 300,
                "EvaluationPeriods": 2,
                "Threshold": 70,
                "AlarmActions": [{
                    "Ref": "WebServerScaleUpPolicy"
                }],
                "Dimensions": [{
                    "Name": "AutoScalingGroupName",
                    "Value": {
                        "Ref": "WebServerGroup"
                    }
                }],
                "ComparisonOperator": "GreaterThanThreshold"
            }
        },
        "CPUAlarmLow": {
            "Type": "AWS::CloudWatch::Alarm",
            "Properties": {
                "AlarmDescription": "Scale-down if CPU < 40% for 5 minutes",
                "MetricName": "CPUUtilization",
                "Namespace": "AWS/EC2",
                "Statistic": "Average",
                "Period": 300,
                "EvaluationPeriods": 2,
                "Threshold": 40,
                "AlarmActions": [{
                    "Ref": "WebServerScaleDownPolicy"
                }],
                "Dimensions": [{
                    "Name": "AutoScalingGroupName",
                    "Value": {
                        "Ref": "WebServerGroup"
                    }
                }],
                "ComparisonOperator": "LessThanThreshold"
            }
        },
           "ApplicationLoadBalancer": {
            "Type": "AWS::ElasticLoadBalancingV2::LoadBalancer",
            "Properties": {
            "Name": "elb-test",
            "Scheme": "internet-facing",
            "IpAddressType": "ipv4",
            "Type": "application",
                "Subnets": {
                    "Ref": "Subnets"
                }
            }
        },
        "ALBListener": {
            "Type": "AWS::ElasticLoadBalancingV2::Listener",
            "Properties": {
                "DefaultActions": [{
                    "Type": "forward",
                    "TargetGroupArn": {
                        "Ref": "ALBTargetGroup"
                    }
                }],
                "LoadBalancerArn": {
                    "Ref": "ApplicationLoadBalancer"
                },
                "Port": 80,
                "Protocol": "HTTP"
            }
        },
        "ALBTargetGroup": {
            "Type": "AWS::ElasticLoadBalancingV2::TargetGroup",
            "Properties": {
                "Name": "ELB-Group",
                "HealthCheckIntervalSeconds": 30,
                "HealthCheckTimeoutSeconds": 5,
                "HealthyThresholdCount": 3,
                "Port": 80,
                "Protocol": "HTTP",
                "TargetType": "instance",
                "UnhealthyThresholdCount": 5,
                "VpcId": {
                    "Ref": "VpcId"
                }
            }
        },
        "WebServerGroup": {
            "Type": "AWS::AutoScaling::AutoScalingGroup",
            "Properties": {
                "VPCZoneIdentifier": {
                    "Ref": "Subnets"
                },
                "HealthCheckGracePeriod": 300,
                "LaunchConfigurationName": {
                    "Ref": "LaunchConfig"
                },
                "MinSize": "1",
                "MaxSize": "8",
                    "DesiredCapacity": {
                  "Ref": "WebServerCapacity"
                },
                "TargetGroupARNs": [{
                    "Ref": "ALBTargetGroup"
                }]
            },
      "CreationPolicy": {
        "ResourceSignal": {
          "Timeout": "PT5M",
          "Count": {
            "Ref": "WebServerCapacity"
          }
        }
      },
      "UpdatePolicy": {
        "AutoScalingRollingUpdate": {
          "MinInstancesInService": 1,
          "MaxBatchSize": 1,
          "PauseTime": "PT5M",
          "WaitOnResourceSignals": true
        }
      }
        },
        "LaunchConfig": {
            "Type": "AWS::AutoScaling::LaunchConfiguration",
            "Metadata" : {
            "Comment" : "Install a simple application",
            "AWS::CloudFormation::Init" : {
              "full_install" : [ "install_and_enable_cfn_hup" ]
          },
          "install_and_enable_cfn_hup" : {
            "files" : {
                        "/etc/cfn/cfn-hup.conf" : {
                          "content" : { "Fn::Join" : ["", [
                            "[main]\n",
                            "stack=", { "Ref" : "AWS::StackId" }, "\n",
                            "region=", { "Ref" : "AWS::Region" }, "\n"
                          ]]},
                          "mode"    : "000400",
                          "owner"   : "root",
                          "group"   : "root"
                        },
                        "/etc/cfn/hooks.d/cfn-auto-reloader.conf" : {
                          "content": { "Fn::Join" : ["", [
                            "[cfn-auto-reloader-hook]\n",
                            "triggers=post.update\n",
                            "path=Resources.EC2Instance.Metadata.AWS::CloudFormation::Init\n",
                            "action=/opt/aws/bin/cfn-init -v ",
                            "         --stack ", { "Ref" : "AWS::StackName" },
                            "         --resource LaunchConfig",
                            "         --configsets full_install ",
                            "         --region ", { "Ref" : "AWS::Region" }, "\n",
                            "runas=root\n"
                          ]]}
                        },
                       "/lib/systemd/system/cfn-hup.service": {
                            "content": { "Fn::Join" : ["", [
                            "[Unit]\n",
                            "Description=cfn-hup daemon\n\n",
                            "[Service]\n",
                            "Type=simple\n",
                            "ExecStart=/opt/aws/bin/cfn-hup\n", 
                            "Restart=always\n\n",
                            "[Install]\n",
                            "WantedBy=multi-user.target"]]}
                             }
                  },  
            "commands" : {
                  "01enable_cfn_hup" : {
                      "command" : "systemctl enable cfn-hup.service"
                  },
                  "02start_cfn_hup" : {
                      "command" : "systemctl start cfn-hup.service"
                  }
              }
           }
        },
            "Properties": {
                "KeyName": {
                    "Ref": "KeyName"
                },
                "ImageId": "ami-00932e4c143f3fdf0",
                "SecurityGroups": [{
                    "Ref": "InstanceSecurityGroup"
                }],
                "InstanceType": {
                    "Ref": "InstanceType"
                },
                "UserData"       : { "Fn::Base64" : { "Fn::Join" : ["", [
             "#!/bin/bash -xe\n",
             "apt-get update -y\n",
             "apt-get install -y python-setuptools\n",
             "mkdir -p /opt/aws/bin\n",
             "python /usr/lib/python2.7/dist-packages/easy_install.py --script-dir /opt/aws/bin https://s3.amazonaws.com/cloudformation-examples/aws-cfn-bootstrap-latest.tar.gz\n",
              "/opt/aws/bin/cfn-init -v ",
             "         --stack ", { "Ref" : "AWS::StackName" },
             "         --resource LaunchConfig",
             "         --configsets full_install ",
             "         --region ", { "Ref" : "AWS::Region" }, "\n",

             "/opt/aws/bin/cfn-signal -e $? ",
             "         --stack ", { "Ref" : "AWS::StackName" },
             "         --resource WebServerGroup",
             "         --region ", { "Ref" : "AWS::Region" }, "\n"
            ]]}}}
            },
        
        
        "InstanceSecurityGroup": {
            "Type": "AWS::EC2::SecurityGroup",
            "Properties": {
                "GroupDescription": "Enable SSH access and HTTP from the load balancer only",
                "SecurityGroupIngress": [{
                        "IpProtocol": "tcp",
                        "FromPort": 22,
                        "ToPort": 22,
                        "CidrIp": {
                            "Ref": "SSHLocation"
                        }
                    },
                    {
                        "IpProtocol": "tcp",
                        "FromPort": 80,
                        "ToPort": 80,
                        "SourceSecurityGroupId": {
                            "Fn::Select": [
                                0,
                                {
                                    "Fn::GetAtt": [
                                        "ApplicationLoadBalancer",
                                        "SecurityGroups"
                                    ]
                                }
                            ]
                        }
                    }
                ],
                "VpcId": {
                    "Ref": "VpcId"
                }
            }
        }
    },
    "Outputs": {
        "URL": {
            "Description": "The URL of the website",
            "Value": {
                "Fn::Join": [
                    "",
                    [
                        "http://",
                        {
                            "Fn::GetAtt": [
                                "ApplicationLoadBalancer",
                                "DNSName"
                            ]
                        }
                    ]
                ]
            }
        }
    }
}

So please let me know if there is any issue with this template or this CFN command that i am running with this