Dustin Oprea's questions

As of a few months ago, Terraform will fail 10% of the time while, apparently, pushing state to the backend (which is in S3). I'll have to cleanup the cruft left behind, run it again, and it'll pass. It was working fine for a couple of years before this started. The provider version hasn't changed. The environment hasn't changed. Any thoughts on what might be causing/exacerbating this?

module.task-definition.aws_ecs_task_definition.task-definition-default: Destroying... [id=workflow-api-production]
module.task-definition.aws_ecs_task_definition.task-definition-default: Destruction complete after 0s
module.task-definition.aws_ecs_task_definition.task-definition-default: Creating...
module.task-definition.aws_ecs_task_definition.task-definition-default: Creation complete after 0s [id=workflow-api-production]
module.load-balancer.module.service.aws_ecs_service.default: Modifying... [id=arn:aws:ecs:us-east-1:326764833890:service/internal-webserver-ssl/production-workflow-api]
module.load-balancer.module.service.aws_ecs_service.default: Modifications complete after 1s [id=arn:aws:ecs:us-east-1:326764833890:service/internal-webserver-ssl/production-workflow-api]
╷
│ Error: Failed to save state
│ 
│ Error saving state: failed to upload state: operation error S3: PutObject, failed to rewind transport stream for retry, request stream is not seekable
╵
╷
│ Error: Failed to persist state to backend
│ 
│ The error shown above has prevented Terraform from writing the updated state to the configured backend. To allow for recovery, the state has been written to the file "errored.tfstate" in the
│ current working directory.
│ 
│ Running "terraform apply" again at this point will create a forked state, making it harder to recover.
│ 
│ To retry writing this state, use the following command:
│     terraform state push errored.tfstate

My Terraform Git repository ended-up having some unreconciliable data error and I couldn't push a recent change. I exported my commit to a patch, recloned, applied the patch, and then realized I just blew-away all of my local state. I used AWS CLI to sync-down the whole tree of state files, copied the one into/as the .terraform/terraform.tfstate of the path that I wanted to apply, ran init, and am now getting a version error.

This is the exact same running environment that produced the state files in the first place. This is also the only machine that apply changes so everything should still be in sync (not that it relates to the immediate problem, but it won't cause any additional issues once we get past this).

Commands that I've tried running:

$ terraform init
$ terraform init -upgrade
$ terraform init -upgrade -migrate-state

The inevitable, unwavering output:

Initializing the backend...
╷
│ Error: Failed to load state: Terraform 1.3.9 does not support state version 4, please update.

I can't seem to find any posts or write-ups anywhere about how to restore from remote state, and yet I'm guessing that almost everyone has had to restore from remote state at some point. I can't understand it.

What should be the process to restore from remote? Why would I have version issues if the environment, project config, and project state haven't changed?

Thank you. Any help is appreciated.

There are no issues connecting to it from any other environment (local works fine, as does from the Cloud), and the security group is currently totally open. An actual PostgreSQL instance, hosted in AWS, had no issue. It's just an Aurora thing. I've created several instances with different configurations. SSL authentication is turned off, and SSL itself is configured to be optional.

"Sorry, we encountered an error and were unable to complete your request."

The subscription listing (aws sns list-subscriptions-by-topic; the UI, too) shows a "Deleted" for a subscription that was probably unsubscribed.

{
    "Subscriptions": [
        {
            "SubscriptionArn": "Deleted",
            "Owner": "326764833890",
            "Protocol": "email",
            "Endpoint": "[email protected]",
            "TopicArn": "arn:aws:sns:us-east-1:326764833890:my-sns"
        },
        {
            "SubscriptionArn": "arn:aws:sns:us-east-1:326764833890:my-sns:65d23ee0-57e1-4ba5-86e7-2913bef8c9bf",
            "Owner": "326764833890",
            "Protocol": "email",
            "Endpoint": "[email protected]",
            "TopicArn": "arn:aws:sns:us-east-1:326764833890:my-sns"
        }
    ]
}

So, there's nothing to use to delete. When deleting via the UI, it shows an error complaining about the ARN being only one part rather than six.

Is there any way of overriding the working-directory from the task-definition default when starting a task from the CLI? It seems like a fairly easy thing and a potentially important thing, but, if it is available, it looks like it's going to be very obscure, in that it's not advertised in the CLI documentation or UI.

The AWS console appears to kick you out after 24-hours, and I'd like to increase it slightly. This has nothing to do with SSO. Is this a fundamental requirement or can it be tweaked somewhere?

When you create a cluster, you declare an instance count and that number of EC2 instances are created. When you even tweak the entries in the associated security-group, ECS will re-fill all of its instances in order to adopt the security changes.

There appears to be no way to tweak the number of instances, which is a big deal since you'll need to dump all of your existing scheduled tasks and services in order to create (or delete and recreate) a cluster to have the updated count. I don't understand why this (or other settings, such a instance type) can't be modified since even tweaking security settings will already reprovision the cluster and this would ostensibly have a similar response.

Am I missing something?

When creating an EC2-mode ECS cluster, you must assign/create a security group:

However, there appears to be no way to retrieve the ARN/name of the security group afterward.

UI:

CLI:

$ aws ecs describe-clusters --clusters extraction
{
    "clusters": [
        {
            "clusterArn": "arn:aws:ecs:us-east-1:326764833890:cluster/extraction",
            "clusterName": "extraction",
            "status": "ACTIVE",
            "registeredContainerInstancesCount": 0,
            "runningTasksCount": 0,
            "pendingTasksCount": 0,
            "activeServicesCount": 0,
            "statistics": [],
            "tags": [],
            "settings": [
                {
                    "name": "containerInsights",
                    "value": "disabled"
                }
            ],
            "capacityProviders": [
                "FARGATE_SPOT",
                "FARGATE"
            ],
            "defaultCapacityProviderStrategy": []
        }
    ],
    "failures": []
}

Assume there are no services that can be expected:

As this is a cluster semantic, I would assume there is a cluster solution to inspecting this information. That said, I'm close to assuming that this information is not actually [exposed] in ECS and might only, actually, be found by looking at the actual instances in EC2.

Note that it's interesting that the "attributes" under the "ECS Instances" tab shows a machine class but, yet, not the security group:

I have a BIND9 server. I can query it and telnet to it from one system. It will not respond at all when this is attempted from another system on the same subnet/VLAN. It behaves as if the packets are being dropped.

With just a simple telnet, I can see, with tcpdump, that the packets are being received from this system, but I do not understand why the server never responds. I have already restarted the non-working client system. To minimize interference, I have also disabled that client's local iptables.

For reference, the ACL of the BIND9 server is "any" and iptables is permitting UDP/TCP on 53 from any source. iptables is totally disabled on the non-working client system (by way of UFW).

The server is Ubuntu 16.04 . The working client system is 17.04 . The non-working client system is 14.04 .

Thoughts?

In TFS 2015 the "publish" task in a build definition may either push to the TFS server or to a fileshare/UNC path. Unfortunately, deleted builds aren't automatically pruned if you're using a fileshare. However, I'm unable to determine where TFS hosts the published builds when you're using the "server" option.

I do not have direct access to the TFS server, just the agent hosts. However, I have a sandbox environment whose local TFS admin doesn't appear to expose any configuration for where this path might be, either.

Does anyone have more information than I do? Where are these builds hosted in TFS? Where is this configured?

So, in order to prevent console-UI (newt, snack, dialog) prompts during the dpkg post-install, you have to do something like:

debconf-set-selections <<< 'mysql-server mysql-server/root_password password password'
debconf-set-selections <<< 'mysql-server mysql-server/root_password_again password verify_password'
apt-get -y install mysql-server

• Is there a rule for this?
• How do we discover the prompt names/descriptors for other packages that invoke the console UI in order to prevent them?
• Would we likely have to dig through the source-code to figure out how the prompts are invoked?
• Is there a simple way to just choose all defaults (effectively clicking on whatever the default button would be and/or taking the correct text/options)?

Thanks.

NOTE

It looks like setting this variable is the solution for taking the defaults:

DEBIAN_FRONTEND=noninteractive

I'm using Openswan with ipsec and ipsec keeps complaining about the shared-key not being present. I'm running Ubuntu 14.04 .

I'm just experimenting on a couple of internal systems since I'm new to this.

Output:

root@ip-10-1-1-4:/etc# ipsec auto --up L2TP-PSK
104 "L2TP-PSK" #10: STATE_MAIN_I1: initiate
003 "L2TP-PSK" #10: received Vendor ID payload [Openswan (this version) 2.6.38 ]
003 "L2TP-PSK" #10: received Vendor ID payload [Dead Peer Detection]
003 "L2TP-PSK" #10: received Vendor ID payload [RFC 3947] method set to=115 
003 "L2TP-PSK" #10: Can't authenticate: no preshared key found for `10.1.1.4' and `10.1.1.36'.  Attribute OAKLEY_AUTHENTICATION_METHOD
003 "L2TP-PSK" #10: no acceptable Oakley Transform
214 "L2TP-PSK" #10: STATE_MAIN_I1: NO_PROPOSAL_CHOSEN

Server:

10.1.1.36 %any : PSK "69EA16F2C529E74A7D1B0FE99E69F6BDCD3E44"

Client:

%any 10.1.1.36 : PSK "69EA16F2C529E74A7D1B0FE99E69F6BDCD3E44"

I know that it doesn't necessarily matter which sides of the conversation are actually designated as "left" and "right" since it's detected automatically and I don't know if/how these relate to which of the two addresses is which in the two lines above. However, I've tried switching both values and using all IPs (so that there are no "%any" values). I'm also not sure which side of the conversation is originating the complaint.

Can someone with more experience give me a hint about where to do from here :) ?

Thanks. This is, of course, a little obscure for a newcomer.

It seems like this should be easy. I run "ssh-keygen -l -f " (the default seems to be SHA1). All of the examples show it printing the hex-formatted digest with a little extra, harmless information. Yet, on my Mac I'm getting a useless, opaque string.

I should mention that the "-E" parameter works on Mac (10.10) but is unavailable in Ubuntu (14.04). Keep that in mind if it doesn't work for you.

$ ssh-keygen -l -E md5 -f dustin.pem
2048 MD5:29:ed:da:d3:5a:8c:78:4f:62:d3:fd:0c:77:5b:6d:d9 dustin.pem.pub (RSA)

$ ssh-keygen -l -E sha1 -f dustin.pem
2048 SHA1:x2ENPL+vzVdlgkIyu0tAhVQ+H4U dustin.pem.pub (RSA)

$ ssh-keygen -l -E sha256 -f dustin.pem
2048 SHA256:agJs/axI8QPzet/eoPMDxLSf37fd1bgsMX4Di0gqMy4 dustin.pem.pub (RSA)

Both of the attempted SHA algorithms return opaque, but different, ASCII strings.

What am I missing? How do I get a SHA1 hex-digest without having to copy my key to a Linux system to do it?

Thanks.

When I build a self-signed certificate for an internal CA, should I make the issuer DN match the DN in the request, should they be different, or is there no convention for one or the other?

Am I correct in assuming that there's protection against the modification of both the subject and the extension information in a request or X509 certificate?

This signature is just another element embedded in the ASN.1 encoding?

It seems like OpenSSL is broken when trying to read back subjectAltName/otherName/UTF8 values that were written by itself:

The relevant openssl.cnf configuration (using an official but random OID):

[alt_names]
DNS.1   = www.foo.com
DNS.2   = www.bar.org
IP.1    = 192.168.1.1
IP.2    = 192.168.69.144
email = email@me

otherName = 1.3.6.1.4.1.1;UTF8:some other identifier

A sample from the dump of a CSR generated from this config:

            TLS Web Server Authentication
        X509v3 Subject Alternative Name: 
            DNS:www.foo.com, DNS:www.bar.org, IP Address:192.168.1.1, IP Address:192.168.69.144, email:email@me, othername:<unsupported>
Signature Algorithm: sha1WithRSAEncryption
    6f:4a:1d:8f:43:7e:4d:d1:0c:7e:05:9d:1f:f0:98:b1:69:cf:

Can someone indicate whether I'm doing something wrong? This is driving me nuts.

I'm trying to start supervisord with Upstart. The Upstart job is very simple and traditional (start on, stop on, author description, respawn, exec).

I've tried running supervisord in the foreground ("-n") and omitting the "expect" stanza from the Upstart job. The start and stop both hang.

I've tried running supervisord as a daemon (its default mode), and setting "expect" to both "fork" and "daemon" (and "stop", in desperate attempt). start and stop both hang.

The Upstart log doesn't display any entries from my attempts.

The only way that I can get it to start is to write the job and restart the system. It always starts properly. But, the command always blocks when I try to manually stop and start it.

I'm very familiar with Upstart infrastructure and configuration. Does anyone have any advice?

Is there any way to measure performance on a FUSE-based mount that is mounted via long-running executable (no device is used, nor /etc/fstab entry added)?

Although it seems relatively easy to build an ECDSA hash programmatically, I've exhaustively searched for how to do it via command-line (in order to validate my results), and have found no examples of how to do it via the command-line.

Does anyone have experience with this?

"ssh-keygen -F " seems to return one host-key for a given hostname (first matched?), and "ssh-keyscan " seems to return all matched, but the order is not clear.

What's the specific difference between the two?