mdarwin's questions

We are deploying a lambda using CloudFormation SAM templates.

We would like to package the lambda into an S3 bucket, then deploy the AWS::Serverless::Function in multiple regions.

However, lambda code must reside in the same region as where it's being deployed.

AWS have documented how to work around this problem, essentially by creating another lambda CopyZips to copy the zip file to each region where you want it deployed, and a new s3 bucket LambdaZipsBucket in each region to put it in. Then your lambda function looks like this:

MyFunction:
    DependsOn: CopyZips
    Type: AWS::Lambda::Function
    Properties:
      Code:
        S3Bucket: !Ref 'LambdaZipsBucket'
        S3Key: !Sub '${QSS3KeyPrefix}functions/packages/MyFunction/lambda.zip'

The problem is, that relies on knowing the exact path to the zip file. In our case we are using SAM template substitution, so we never specify the S3 bucket or S3 key - instead our Function looks like this:

  MyFunction:
    Type: 'AWS::Serverless::Function'
    Properties:
      CodeUri: ./MyCompany.Service.Broker.Public

The S3 bucket is not referenced at all in the CloudFormation template. And the lambda, which you can see by browsing the bucket, has a name like :

ApiFunction-CodeUri-2342873t823t482346-97346583746583745.zip

This is because there are lots of similar zips, created at build time. The Serverless template model will decode this, and if we look in the CloudFormation console, in the template tab, and select "view processed template", we can see the S3Bucket and S3Key fields populated correctly.

The build is done in Azure devops, using the AmazonWebServices.aws-vsts-tools.LambdaNETCoreDeploy.LambdaNETCoreDeploy@1 task, where we specify the bucket name and region, but again, not the name of the zip file.

Given that I have no access in the template to the exact path to the lambda zip file, how am I supposed to automate the copy of the zip file to a local bucket?

UPDATE:

I tried adding the following line in Properties, just before CodeUri:

      Code:
        S3Bucket: !Ref 'LambdaZipsBucket'

However, I got the following error on deployment:

Resource with id [ApiFunction] is invalid. property Code not defined for resource of type AWS::Serverless::Function 

I have an AWS VPC with two subnets: Public and Private. I can access the internet (via a NAT Gateway) from the public subnet, but not from the private subnet, and I can't work out what's wrong.

• Both subnets have NACLs which allow all traffic in and out.
• Both EC2 instances I'm using for my testing (one in each subnet) have the exact same security groups: (allow https out, ssh in)

I have enabled flow logs to try and see what the problem is. Here is what I see when connecting to youtube from the box in the public subnet:

216.58.210.206 10.0.129.171 443 40682 6 9 4760 1596476113 1596476173 ACCEPT OK

10.0.129.171 216.58.210.206 40682 443 6 11 1218 1596476113 1596476173 ACCEPT OK

Here we can see the outbound request, and the response, both allowed to pass through.

When repeating the same test from a box in the private subnet, here's what I see:

10.0.40.146 216.58.210.206 46398 443 6 2 120 1596469555 1596469612 ACCEPT OK

The above line shows the https request from my private subnet EC2 instance to https://youtube.com - the traffic is allowed. No response is logged.

Because I can ssh from public to private subnet, I can deduce that the route is present and correct. So where's the response? Why is the traffic blocked?

Security groups are identical between the two boxes, and they allow tcp on 443 to 0.0.0.0/0

Interestingly, the route from the public subnet to the internet points to the internet gateway, but the route from the private subnet points to the NAT Gateway.

How do I debug this further?

I have a VPC with 3 subnets:

1. Public (FE layer, inbound internet access only)
2. Protected (app layer, outbound internet access only)
3. Private (DB layer, no internet access at all)

The public subnet NACL blocks all inbound from the internet on any ports other than 443.

The private subnet NACL blocks all in and out from the internet.

The NACL for the protected subnet has to allow:

• outbound on 443 to 0.0.0.0/0 for outbound internet https requests
• inbound from 0.0.0.0/0 on 1024-65535 for responses to those requests

The protected subnet routes to the internet through a NAT Gateway, which lives in the public subnet.

The problem is this: outbound traffic from the protected subnet to the internet only works if the NACL for the public subnet is opened to allow inbound on ports 1024 - 65535. This is presumably because the traffic goes through the public subnet (since that's where the NAT Gateway lives).

So I can open up the NACL for the public subnet (inbound from 0.0.0.0/0 on 1024-65535), but that makes it less secure; in fact if I'm opening all those ports, I may as well just not bother with a NACL at all (obviously I am also using security groups).

Am I missing something?

I'm deploying an application to AWS using the CloudFormation plugin for Azure DevOps.

It works really well, and will automatically update things that need changing, leaving other things alone. I've been working on it for a few days, and it works fine.

However I just saw the following error when committing a small change:

Test for existence of change set Dev-TV-API-ChangeSet-Release-42 returned error: 'Stack [Dev-TV-API-NoAuth] does not exist'.

That looked a bit strange, so I thought maybe it's having trouble working out what needs to change. So I decided to delete the stack manually, and re-run the pipeline. Now I get exactly the same error. The stack shows up in AWS with status REVIEW_IN_PROGRESS - and a single event.

Any ideas what could be going on here?

Full log:

2020-07-30T14:17:27.5715771Z Test for existence of change set Dev-TV-API-ChangeSet-Release-42 returned error: 'Stack [Dev-TV-API-NoAuth] does not exist'.
2020-07-30T14:17:27.5718700Z Loading template file from 'D:\a\r1\a\_TV CI\drop\package.yml'
2020-07-30T14:17:27.5724733Z Loading template parameters file 'D:\a\r1\a\_TV CI\drop\api.parameters.dev.json'
2020-07-30T14:17:27.5731008Z Successfully loaded template parameters
2020-07-30T14:17:27.5733302Z Setting capability CAPABILITY_IAM for stack
2020-07-30T14:17:27.5733902Z Setting capability CAPABILITY_NAMED_IAM for stack
2020-07-30T14:17:27.5734434Z Setting capability CAPABILITY_AUTO_EXPAND for stack
2020-07-30T14:17:27.5735777Z Creating CREATE type change set Dev-TV-API-ChangeSet-Release-42
2020-07-30T14:17:27.8132789Z Waiting for change set Dev-TV-API-ChangeSet-Release-42 to be validated for stack Dev-TV-API-NoAuth
2020-07-30T14:17:58.3348363Z Change set creation request failed with error: 'Stack Dev-TV-API-NoAuth failed to create successfully from the change set Dev-TV-API-ChangeSet-Release-42. Error: 'Resource is not in the state changeSetCreateComplete'' Error: Stack Dev-TV-API-NoAuth failed to create successfully from the change set Dev-TV-API-ChangeSet-Release-42. Error: 'Resource is not in the state changeSetCreateComplete'
2020-07-30T14:17:58.3355531Z     at .<anonymous> (D:\a\_tasks\CloudFormationCreateOrUpdateStack_7ef7cdfa-aa45-42c5-93c8-d7603643dd99\1.7.0\CloudFormationCreateOrUpdateStack.js:2:4039841)
2020-07-30T14:17:58.3356313Z     at next (native)
2020-07-30T14:17:58.3357470Z     at o (D:\a\_tasks\CloudFormationCreateOrUpdateStack_7ef7cdfa-aa45-42c5-93c8-d7603643dd99\1.7.0\CloudFormationCreateOrUpdateStack.js:2:4028532)
2020-07-30T14:17:58.3359409Z     at process._tickDomainCallback (internal/process/next_tick.js:135:7)
2020-07-30T14:17:58.3489430Z ##[error]Error: Stack Dev-TV-API-NoAuth failed to create successfully from the change set Dev-TV-API-ChangeSet-Release-42. Error: 'Resource is not in the state changeSetCreateComplete'
2020-07-30T14:17:58.3772097Z ##[section]Finishing: Create/Update Stack: Dev-TV-API-NoAuth

I'm trying to reference a security group inside a CloudFormation template.

the name of the group is !Sub '${EnvironmentName}-SG-Private, where EnvironmentName is a template parameter.

However, the DBInstance AWS type requires the GroupId of the security group, not the group name. So I'm trying to use GetAtt to retrieve it, but without any luck:

Fn::GetAtt: [$(Fn::Sub:[${EnvironmentName}-SG-Private]), GroupId]

This and various other permutations all caused errors.

Reading the docs, it says

For the Fn::GetAtt logical resource name, you cannot use functions. You must specify a string that is a resource's logical ID.

If I've read that right, the problem cannot be solved the way I'm trying to do so. So how am I supposed to reference this security group? Do I need to export its GroupId at group creation time?

I'm designing an architecture for a system (built on AWS) which will have multiple different production environments, in different zones.

Initially I had thought that it would be a good idea to use 1 VPC per environment, with another Operations and Maintenance (OAM) VPC providing a bastion host and used for ssh connections to the environment VPCs (which can then block all inbound ssh traffic from the public internet). Each environment (env) VPC would have VPC peering to the OAM VPC.

The problem with this approach is that if I want to access all env VPCs from the OAM VPC, they must each have different CIDR ranges. This has two implications:

1. I need to keep track of which environment has which CIDR range
2. I will be boxing myself in for future scalability - either the CIDR ranges are too few and too large, in which case I might reach the limit of number of environments, or too small and too numerous, in which case I will limit the number of things I can put in each environment.

The alternative approach is to keep all the env VPCs completely isolated from each other, which means that each one can have identical CIDR ranges. To me this seems like an advantage, because identical environments means easier maintenance and fewer human errors. Plus we can fit more things in each environment. But to access them I would have to add a bastion host in each one, which is a) less secure and b) wasteful.

Is there a best practice on how these two conflicting demands (security and conformity) should be reconciled?