emmdee's questions

Ansible 2.8.1

In my playbook tasks/ dir:

main.yml
dev.yml

In main.yml I have a block like this:

- include_tasks: dev.yml
  when: ec2_tag_env == 'dev'

It works fine

However, if I try to call a specific task within dev.yml using a tag. It won't qualify the task during the run

For example, this tagged task within dev.yml:

- name: Pull the latest image
  docker_image:
    name: "{{ dev_image }}"
    source: pull
  tags:
    - container

When I run the playbook with -t container it will NOT qualify since the include_tasks step doesn't have that tag.

Adding the tag to include_tasks will of course fix the problem, but I would then need to keep track of tags as they get added to sub-tasks and add them here as well:

- include_tasks: dev.yml
  when: ec2_tag_env == 'dev'
  tags:
    - container

Questions

• Is it possible to have Ansible just "know" what tasks are within the include_tasks block and pull the applicable tags?

• What's the best practice for achieving this goal?

What I would prefer to not have to do:

• Put everything main.yml. I have so many tasks in this playbook I really want to keep them organized in files.
• Tag all my include_tasks blocks with all its sub-tags manually. Sounds like a nightmare to manage.

When I am running sed operations on a file, for example:

sed 's/this/that/' /my/file.txt

The output is displayed on stdout.

However, when I apply the changes in place with -i there is no stdout (no output at all to screen).

sed -i 's/this/that/' /my/file.txt

How can I still output to screen when I'm using the -i trigger?

I have the following scenario.

I have a backup extract, this exists in a local folder ./app_backup. Example directory structure inside this:

./app_backup/plugins/plugin1
./app_backup/plugins/plugin2
./app_backup/plugins/plugin3
./app_backup/src/a
./app_backup/src/b
./app_backup/assets/mine.jpg
./app_backup/.../lots/more/...

I also have a local dir ./plugins that holds local plugins I'm actively developing. It looks something like this:

./plugins/pluginX
./plugins/pluginY

This whole structure needs to get mounted into a container. I'm doing this via compose.

(if there is a naming conflict, the folder in ./plugins should take precedence over the one in the backup)

I need these to all mount together to a main plugin dir INSIDE the container once it spins up, the paths will be something like this desired result inside the container:

/app/main/plugins/plugin1
/app/main/plugins/plugin2
/app/main/plugins/plugin3
/app/main/plugins/pluginX
/app/main/plugins/pluginY

Naturally, the compose file for the container can simply do this:

volumes:
  - ./app_backup/:/app/main/
  - ./plugins/pluginX:/app/main/plugins/pluginX
  - ./plugins/pluginY:/app/main/plugins/pluginY

However, I do not always know the quantity or name of the local plugins, it can be dynamic so I can't explicitly define each plugin subdir in the compose file

For example, one developer could have ./plugins/pluginX and another could have 10 different ones of different names such as ./plugins/pluginABC.

Therefore I need to just mount the entire ./plugins/ dir since we will never know explicitly which dirs are inside it.

Something like this:

volumes:
  - ./app_backup:/app/main
  - ./plugins:/app/main/plugins

This inherently creates another issue, if I mount ./plugins then the entire /app/main/plugins (that came from app_backup) gets overwritten!

So right now I'm stuck making the devs update the compose file for every single plugin they are developing. This is a pain point as they're constantly needing to edit their compose file!

Please let me know if what I'm trying to do is even possible, and if not, what's some suggested course of actions to ease up this workflow. Thanks!

If the volume mount allowed for "subdirs-only" such as ./plugins/* I think it would solve my problem but I don't think it works like that. I hope I'm wrong.

EDIT:

Ugly workaround:

1. Each plugin has its own compose file. The only thing this compose file does is handle the individual mount path.
2. Find occurrences of: ./plugins/*/docker-compose.yml
3. Include those files with docker-compose -f when starting the service.

I would really like to not have to handle a compose file for each service, but this at least accomplishes the goal.

Using nginx rate limiting, using this rule as an example:

limit_req_zone $request_limiting_zone_key zone=request_limiting_search:20m rate=30r/m;

This will set 30 requests per minute. However what if I wanted to stack a second rule, so the logic is "30 requests per minute, or 1 request per second. Whichever comes first"

The reason for this is we still want a cap of 30/min but there are some weird attempts that we're seeing a spam of 10x requests in a 1-2 second period. This causes some issues when requests come through that fast. So we would like to limit users to 1 request a second, AND a cap of 30 per minute.

I read the doc reference but didn't see much around stacking different rate rules.

Is this possible? If so, what would be the syntax?

At a maximum, a VPC network ACL can have 40 rules applied.

I have a list of over 50 IP addresses that I need to explicitly block access to in our systems, over any port and any protocol. This is an ideal purpose for an ACL, but the limit is hindering me completing this task.

Of course, I can do this in IPTables on each host, but I want to block any and all traffic to all components in the VPC (to ELB's for example). Furthermore it's far more ideal to manage these rules in one place rather than on each and every host.

I am hoping there is some way I am not understanding doing this at the system/platform level. Security groups are explicit allow, with no deny action, so they won't do the trick.

My goal is to get a simple "# of requests per day" value against a Route 53 hosted zone.

I see no straightforward way to do this.

I have created the query logging as explained here: https://docs.aws.amazon.com/Route53/latest/DeveloperGuide/query-logs.html

However this is extensive logging and it's split per global edge server. All I want is "domain example.com was queried 40,000 times in the last 24 hours" and similar metrics.

Is this possible? The logging seems overkill and I'll have to do quite a bit of parsing through all the subdirs to get that type of sum.

I have the following conditional in an Ansible task:

when: ec2_tag_Name == 'testhost01'

It works fine, however I would like to match a wildcard on the ec2_tag_Name field.

So something like this

when: ec2_tag_Name == 'testhost*'

The goal is to match anything like testhostx testhost12 testhostABC etc etc just anything matching testhost at the start of the string.

Is this possible? Can't seem to get it working.

I have kind of a weird setup that I need to run a command as a docker container in a sh block in a Jenkinsfile.

Issue that I'm facing is specifically around the awk command used to trim the output.

Here is the command that WORKS FINE when ran directly on a bash shell:

OPFILENAME=$(docker run -t \
-e AWS_SECRET_ACCESS_KEY='<omitted>' \
-e AWS_ACCESS_KEY_ID='<omitted>' \
-e AWS_DEFAULT_REGION='us-east-1' \
mydockerimage:0.1 \
bash -c "aws s3 ls my-bucket-name/dir/ | sort | tail -n 1 | awk '{print \$4}' ") && \
echo $OPFILENAME

So I need to run this exact same thing on a remote host through a Jenkins pipeline, here is the general syntax:

pipeline {              
 agent any

 environment {
  BUILDHOST = 'buildhost.example.com' 
  SSHCMD = "ssh -o StrictHostKeyChecking=no jenkins@${env.BUILDHOST}"
 }


stages {
  stage('Get filename from s3') {
    steps { 
      sshagent ( ['ssh_config']) {
        sh """${SSHCMD} '''
            OPFILENAME=\$(sudo docker run -t \
            -e AWS_SECRET_ACCESS_KEY='<omitted>' \
            -e AWS_ACCESS_KEY_ID='<omitted>' \
            -e AWS_DEFAULT_REGION='us-east-1' \
            mydockerimage:0.1 \
            bash -c "aws s3 ls my-bucket-name/dir/ | sort | tail -n 1 | awk '{print \$4}') && \
            echo \$OPFILENAME
            '''
          """
       }
      }
     }
    }
}

Here is the error that Jenkins throws:

...bash -c "aws s3 ls my-bucket-name/dir/ | sort | tail -n 1 | awk {print' '}") && echo $OPFILENAME "
                        '
bash: -c: line 1: unexpected EOF while looking for matching `"'
bash: -c: line 3: syntax error: unexpected end of file

Notice how it transformed the awk command as such: awk {print' '}

Trying with various changes: ...| sort | tail -n 1 | awk \'{print \$4}\'") results in the exact same error.

Tried like this: awk "'{print \$4}'" and it doesn't throw an error but in the logs it shows like this: awk "{print' '}" thus the awk trimming desired doesn't take place. Grr!

So I know there is something wrong with the quoting, and since I am using triple-quotes for multi-line commands it's dirtying it up even more!

I have referenced this gist, trying to make sense of how to do this: https://gist.github.com/Faheetah/e11bd0315c34ed32e681616e41279ef4 but still running into issue after issue.

Caveats: - Command must run in an sshagent block and on a remote host. Commands cannot be ran in the local Jenkins workspace.

I have existing infrastructure in Terraform and have been using it for a while. Recently I had swapped the AWS credentials of my local laptop (the creds stored in ~/.aws/credentials) and it stopped working until I re-set those credentials back.

The problem is that I'm declaring the creds in the Terraform source itself but it doesn't seem to be using them at all.

terraform {
  backend "s3" {
    bucket = "example_tf_states"
    key    = "global/vpc/us_east_1/example_state.tfstate"
    encrypt = true
    region     = "us-east-1"
  }
}

provider "aws" {
  access_key = "${var.access_key}"
  secret_key = "${var.secret_key}"
  region = "${var.region}"
}



variable "access_key" {
  default = "<hidden_for_stack_exchange_post>"
}

variable "secret_key" {
  default = "<hidden_for_stack_exchange_post>"
}

variable "region" {
  default = "us-east-1"
}

The access ID permissions are 100% good. I am using the same account ID and secret key both for the aws configure settings that go into ~/.aws/credentials as I am in the above Terraform variable declarations.

Everything works fine as long as the creds are in ~/.aws/credentials but as soon as the OS level credentials are gone (ie rm ~/.aws/credentials) I get the following when trying to run Terraform operations, such as terraform plan:

Failed to load backend:
Error configuring the backend "s3": No valid credential sources found for AWS Provider.
  Please see https://terraform.io/docs/providers/aws/index.html for more information on
  providing credentials for the AWS Provider

Please update the configuration in your Terraform files to fix this error.
If you'd like to update the configuration interactively without storing
the values in your configuration, run "terraform init".

If I re-populate the ~/.aws/credentials by running aws configure it will work fine again.

I'm not understanding -- if my provider setting is explicitly declaring the credentials to use inside the Terraform source code, why does my OS-level AWS configuration matter at all?

How can I make Terraform only use the creds defined in my Terraform configuration and ignore what's in my OS user profile?

Edit, it's Terraform v0.11.7

Edit: Please note that I'm trying to solve the issue on why the statically declared creds are not being utilized in the provider declaration. Not looking for alternative methods or workarounds. Thanks.

I need to trigger a Jenkins job from another job and pass various values to it for later conditional logic. It seems the Parameterized Trigger Plugin will do exactly what I need. (https://plugins.jenkins.io/parameterized-trigger)

However this plugin I'm not seeing anything regarding pipeline syntax both on the doc as well as in the syntax builder.

Most all plugins I've used allow use within pipeline scripts. (Jenkinsfiles). They rarely have documentation for syntax but I can typically just use the Syntax builder inside Jenkins to get the right pipeline syntax to use the plugin.

Is it possible to use this plugin within a pipeline (multi-branch pipeline to be specific)?

If not, then is there any alternative ways to do what I need to do from a Jenkins pipeline? (Trigger another job while passing some value/parameter to it).

I currently have a WORKING SFTP login, using a private key for login and the user is chroot'ed into their home directory.

Goal: Keep the user chroot but allow WRITE access to the relative chroot directory, without having to specific any path or cd anywhere.

In other words, when the sftp user logs in, I don't want them to have to cd to another path in order to upload a file.

Due to the chroot directory requiring full root ownership (due to chroot design), I'm not certain if this is even possible.

In /etc/passwd:

sftpuser:x:1006:1006:,,,:/home/sftpuser:/bin/false

The users login pubkey is in:

/home/sftpuser/.ssh/authorized_keys

Chroot rules in sshd_confg:

Match User sftpuser
    ChrootDirectory /home/sftpuser
    ForceCommand internal-sftp
    AllowTCPForwarding no
    X11Forwarding no

Home dir permissions:

# ls -l /home/ |grep sftpuser
drwxr-xr-x 5 root      root     4096 May  4 11:24 sftpuser

# ls -l /home/sftpuser/
total 4
drwxrw-r-x 3 sftpuser sftpuser 4096 May  4 11:23 sftp_share

Example current workflow:

Connected to sftp.example.com.
sftp> ls
sftp_share

sftp> put testfile.txt
Uploading testfile.txt to /testfile.txt
remote open("/testfile.txt"): Permission denied

sftp> cd sftp_share

sftp> put testfile.txt
Uploading testfile.txt to /sftp_share/testfile.txt
testfile.txt

Desired workflow:

Connected to sftp.example.com.
sftp> put testfile.txt
Uploading testfile.txt to /testfile.txt
testfile.txt

Is there any way to allow the upload (put) to the default directory, without having to chdir first?

I do have flexibility to move these directories around. For example, the sftp chroot dir doesn't have to be in the users home directory, I can even change around the users home dir (but the user must still be able to use authorized_keys to login).

Please note: I do understand many SFTP clients, and the command line SFTP client allows for defining a relative path at login. That is out of scope for this question, I'm desiring this config be done server-side and the client simply just needs to login.

I'm trying to find the history of container restarts.

Of course the STATUS field on a docker ps will show the current uptime.

However if I have a container with a restart policy such as restart: unless-stopped and it's gone through several restarts - How can I check that restart/uptime history?

If the docker engine doesn't natively track this - is there a known good method to handle this?

Following the official documentation (https://docs.docker.com/registry/spec/api/#deleting-an-image) I have been able to successfully delete an image. As expected, after deleting, the image can no longer be pulled nor its manifest called via API.

I feel like I've got the hard part done, however the problem is that the repo is still listed under /v2/_catalog after the deletion is finished. I'm trying to fully purge the registry.

Here is my registry compose file:

registry:
  image: registry:2.5.2
  container_name: registry-test
  ports:
    - 5007:5000
  environment:
    REGISTRY_STORAGE: s3
    REGISTRY_HTTP_TLS_CERTIFICATE: /etc/cert.crt
    REGISTRY_HTTP_TLS_KEY: /etc/cert.key
    REGISTRY_STORAGE_S3_ACCESSKEY: ******
    REGISTRY_STORAGE_S3_SECRETKEY: ******
    REGISTRY_STORAGE_S3_REGION: us-west-1
    REGISTRY_STORAGE_S3_BUCKET: ******
    REGISTRY_STORAGE_S3_SECURE: "true"
    REGISTRY_STORAGE_DELETE_ENABLED: "true"
  volumes:
    - /dockerdata/volumes/registry-test/etc/cert.crt:/etc/cert.crt
    - /dockerdata/volumes/registry-test/etc/cert.key:/etc/cert.key
  restart: unless-stopped

Here is the high-level method on what I did to delete the image:

1. Gather image digest: HEAD https://myprivateregistry:5001/v2/myimage/manifests/mytag with "Accept: application/vnd.docker.distribution.manifest.v2+json" added to the header on the call

2. The call returns header key Docker-Content-Digest with a value such as sha256:b57z31xyz0f616e65f106b424f4ef29185fbd80833255d79dabc73b8eb873bd

3. Using that value from step 2, run the delete call: DELETE https://myprivateregistry:5001/v2/myimage/manifests/sha256:b57z31xyz0f616e65f106b424f4ef29185fbd80833255d79dabc73b8eb873bd

4. Registry API returns 202 Accepted

5. Run garbage collection manually: registry garbage-collect /etc/docker/registry/config.yml

6. Garbage collector deletes the associated blobs from disk (log omitted here, but it successfully deletes the blobs)

At this point I can confirm the blobs are completely deleted from disk and I can no longer call image details (like in step 1 above) so I thought I was done.

However, when running: /v2/_catalog my associated repo still lists (even though there are no images within it)! Obviously it cannot pull or be used, but how can I fully remove that repo from that list now that it has no images associated with it?

I don't see anywhere how to properly remove this on the API documentation page. Perhaps I'm missing it somewhere?

EDIT -

I wanted to add some more info on how the registry looks before and after the above deletion takes place.

Before the Delete Operation Above:

docker/registry/v2/repositories/myimage/_manifests/revisions/...
docker/registry/v2/repositories/myimage/_manifests/tags/... 
docker/registry/v2/repositories/myimage/_layers/sha256/... (5 layers listed)
docker/registry/v2/blobs/sha256/...

After the Delete Operation Above:

docker/registry/v2/repositories/myimage/_layers/sha256/... (5 layers listed)

So the only thing left over is the _layers directory with the same 5x layers listed. This seems to be the reason why it's still listed on _catalog

When I delete the myimage folder (from docker/registry/v2/repositories/myimage) then the repository is no longer shown in the _catalog

This seems to be a method to purge it from the _catalog listing. However - what if an image has 2x tags, yet only 1 is deleted - is there a reason to delete anything from _layers in that case? How would that be handled with multiple versions of an image? Obviously I can't just clobber the _layers directory as the final method since, in the real world, there will be many tagged versions of an image. So this needs to be done intelligently.

I am simply finding it hard to find any documentation on the maintenance/upkeep of the Docker registry nor the schema for the _layers subdirectory and why the garbage collector doesn't clean up that directory the same it does with manifests and blobs.

Would like to forcefully kick a specific user connection.

OS is Ubuntu 16, OVPN Server is OpenVPN 2.3.10

I can see them connected in /etc/openvpn/openvpn-status.log which lists out current client status. And their persistent connection pool is saved to /etc/openvpn/ipp.txt

I can kick all users by simply cycling the openvpn daemon service openvpn restart however I want to kick ONE single user.

I've tried openvpn --help and man openvpn as well as searched google but not seeing anything.

OS is Ubuntu 16, Server is OpenVPN 2.3.10

Any time I authenticate a system with LDAP/AD I like to have at least one "local" admin that can get in even if Directory Services go down/have an issue. Essentially a "backdoor" just in case the directory service has an issue.

Is this possible in Jenkins? I can only see the option to enable 1 authentication method at a time.

Issue:

In passing a variable declared within Jenkinsfile to a sh command that ssh's and executes on a remote host, the variable contents are not retained on the remote host.

Built-in Jenkins variables retain fine both locally and on the remote host. The variable I've defined works fine locally but doesn't translate on the remote host.

Although this issue references Docker, this is actually 100% Jenkins pipeline-based as it could apply to any example with or without Docker.

Background:

I'm trying to dynamically create an image name based on the current build tag and put that name into a variable.

Then I pass that variable into the sh step which remotes over to a Docker host and runs a build step with the name defined.

Snippet of applicable parts of Jenkinsfile...

// The below stage just echo's out some stuff into a file dockerimgname.jenkins.out
stage ('Construct Img name') {
  sh '''echo ${BUILD_TAG} | awk '{print tolower($0)}' | sed 's/jenkins-//' > dockerimgname.jenkins.out'''
}

// This stage reads that file from previous stage and puts the value into variable.
// The variable is echo'd locally which works perfectly. Then ssh is called to execute command on remote host. That's where the variable value doesn't work.
stage ('Build Target Container') {
  def jobBaseName = readFile 'dockerimgname.jenkins.out'
  echo "${jobBaseName}"
  sh 'ssh -i ~/ssh_keys/key.key user@somehost "cd /dockerdata/build/${BUILD_TAG} && docker build -t localrepo/${jobBaseName}:${BUILD_NUMBER} ."'
}

Normally I would assume it doesn't have the variables since it's a remote host... However it is odd that ${BUILD_NUMBER} and ${BUILD_TAG} translate and work fine on the remote host. Why doesn't ${jobBaseName}?? (it comes out empty/null on remote host).

Firstly, I am using these as a base guide: http://docs.aws.amazon.com/AWSEC2/latest/WindowsGuide/ami-create-standard.html http://reduktor.net/2014/09/autoscale-ec2-instances-ad-aws/

Secondly, our region doesn't support the AWS directory service

Scenario

I am looking to create a "base AMI" for a set of application servers we have. I may need to deploy 1 or more new servers based off of this image.

The instances are originially created from EC2 Windows Server 2008 R2 Datacenter base AMI (Created by Amazon)

The applications themselves are static, licensing is completely redistributable, and the config doesn't need to change on one machine vs another.

The ONLY thing that is different from one machine to the other is the hostname and network config. Network config is handled by EC2, so the hostname is really the only thing that needs to change from once instance to another

The machines are part of Active Directory, and have specific OU Group Policy rules applied to the machines. They will all join the same OU.

Goal:

My goal is to have a base AMI. When this AMI is launched it auto-joins the domain OR is already joined to the domain.

The applications that are ran from the machine REQUIRE domain accounts to run the Windows Services. So I can't have an image that's not joined to the domain.

An idea I had (Will test this tomorrow):

• Create a default/generic hostname "server-xy", join the domain and create my AMI. Power down the source VM then deploy from AMI. Once it boots, rename it to its final hostname "server-02". Then deploy another from AMI, rename it, rinse and repeat -- sound viable? EDIT: this didn't work since renaming it invalidates the original hostname at the domain level. Seems like what I would like to do might not be feasible.

I see this similar question was asked before, however it was either not answered or it pertained 100% to Windows workarounds.

This is specifically Linux and I am trying to restore a Clonezilla image to a 4TB hard drive.

I have used this image many times before on other disks, however due to MBR restrictions on this larger drive, it needs to be GPT so I cannot restore the disk image.

The exact error that Clonezilla states is "Error: Destination disk size is 4.00TB, which is larger than the MBR partition table entry maximum 2TiB. You have to use GUID partition table format (GPT)"

I understand "what" the problem is but I do not know the steps to resolve. I'm going to try cloning the master machine using Macrium reflect and restoring, hopefully it can migrate the partition structure automatically.

NOTE: This drive is blank, therefore I have no concern over data loss. Experimentation is fine as I have nothing to lose on the drive.

EDIT/UPDATE:

So it seems Clonezilla actually allows the process to continue, however it will write everything as-is to MBR and the disk will be seen as 2TB.

With this in place, the question then changes to converting the MBR to GPT on the OS/boot volume (yes I am booting into the system that I am working with, data loss is not a problem since I can just restore from image if something breaks, which is has many times so far in my trials)

I have opened the disk with gdisk and ran the conversion to GPT. This was succesful, however this kills the GRUB boot partition and indeed the system does not boot after conversion.

I am following this: https://wiki.archlinux.org/index.php/GUID_Partition_Table#Convert_from_MBR_to_GPT

What I need help with now: So right now I seem to have restored my Clonezilla image to the new disk in MBR format and have done and in-place conversion to GPT using gdisk.

I am now looking for guidance on creating the GRUB partition and re-installing GRUB so that the system can boot after conversion.

Once I have this all laid out I can format it to a full step by step answer with all the components together to help others in the future.

Thanks!!

This question is about KMS activation over a different network from the one my workstation is connected to. The software I am dealing with is Windows 7 Enterprise and MS Office 2013

Scenario

I manage a LAN for my company. We have our own internal AD and DNS/DHCP servers. However, Our physical workstations are owned and managed by a Corporation and they handle licensing of the workstation software and manage the KMS servers.

I can physically plug the workstations into the CORP LAN and Domain and force activation, but this is basically the worst case scenario.

My Goal:

My Goal is to create a constant static route to kms.corp.com through a router that is connected to both networks. I want the workstations to then have DNS records that automatically point to the proper host for KMS activation.

I drew a diagram of the scenario I would like to have:

Now creating the routes is easy, what I don't understand is the DNS settings I need to create in order to allow the machines to properly access the KMS activation server.

Any help is appreciated, thanks.

I am very sorry if this has been answered already. I found this post: DHCP Reservation However I wasn't sure if that was related to Windows or Linux DHCP server and I wanted to clearly state my configuration for an accurate yes/no before I make any changes.

I have a Windows DHCP server with the following config:

• Scope = 10.10.5.0/24
• Address pool = 10.10.5.10 - 10.10.5.254

My goal is to limit that address pool to 10.10.5.240 - 10.10.5.254 since non-reserved DHCP clients are very uncommon on this subnet.

However, I have MANY address reservations already configured based on MAC address across the whole /24 subnet and they cannot be affected.

Will adjusting the Address Pool affect my reservations?