I have at home two CoreOS stable v1122.2.0 machines. both of them are registered at discovery.etcd.io properly. (i see both of them registered at https://ip:2380 when i browse to the discovery url).

the first server is 10.79.218.2, and the 2nd is 10.79.218.3.

I'm trying to configure etcd for tls. so I created the required certificates and I configured etcd2 for tls. all the configuration (of the first server with local ip of 10.79.218.2) is in the cloud-config.yml file:

 #cloud-config

 write_files:
  - path: "/etc/ssl/etcd/ca.pem"
    permissions: "0600"
    owner: "etcd:etcd"
    content: |
      -----BEGIN CERTIFICATE-----
      ...
      -----END CERTIFICATE-----
  - path: "/etc/ssl/etcd/server1.pem"
    permissions: "0600"
    owner: "etcd:etcd"
    content: |
      -----BEGIN CERTIFICATE-----
      ...
      -----END CERTIFICATE-----
  - path: "/etc/ssl/etcd/server1-key.pem"
    permissions: "0600"
    owner: "etcd:etcd"
    content: |
      -----BEGIN EC PRIVATE KEY-----
      ...
      -----END EC PRIVATE KEY-----
  - path: "/etc/kubernetes/cni/net.d/10-calico.conf"
    content: |
      {
          "name": "calico",
          "type": "flannel",
          "delegate": {
              "type": "calico",
              "etcd_endpoints": "https://10.79.218.2:2379,https://10.79.218.3:2379",
              "log_level": "none",
              "log_level_stderr": "info",
              "hostname": "10.79.218.2",
              "policy": {
                  "type": "k8s",
                  "k8s_api_root": "http://127.0.0.1:8080/api/v1/"
              }
          }
      }
  - path: "/etc/kubernetes/manifests/policy-controller.yaml"
    content: |
     apiVersion: v1
      kind: Pod
      metadata:
        name: calico-policy-controller
        namespace: calico-system
      spec:
        hostNetwork: true
        containers:
          # The Calico policy controller.
          - name: k8s-policy-controller
            image: calico/kube-policy-controller:v0.2.0
            env:
              - name: ETCD_ENDPOINTS
                value: "https://10.79.218.2:2379,http://10.79.218.3:2379"
              - name: K8S_API
                value: "http://127.0.0.1:8080"
              - name: LEADER_ELECTION
                value: "true"
          # Leader election container used by the policy controller.
          - name: leader-elector
            image: quay.io/calico/leader-elector:v0.1.0
            imagePullPolicy: IfNotPresent
            args:
              - "--election=calico-policy-election"
              - "--election-namespace=calico-system"
              - "--http=127.0.0.1:4040"

  - path: "/etc/kubernetes/manifests/kube-scheduler.yaml"
    content: |
      apiVersion: v1
      kind: Pod
      metadata:
        name: kube-scheduler
        namespace: kube-system
      spec:
        hostNetwork: true
        containers:
        - name: kube-scheduler
          image: quay.io/coreos/hyperkube:v1.3.6_coreos.0
          command:
          - /hyperkube
          - scheduler
          - --master=http://127.0.0.1:8080
          - --leader-elect=true
          livenessProbe:
            httpGet:
              host: 127.0.0.1
              path: /healthz
              port: 10251
            initialDelaySeconds: 15
            timeoutSeconds: 1
  - path: "/etc/kubernetes/manifests/kube-controller-manager.yaml"
    content: |
      apiVersion: v1
      kind: Pod
      metadata:
        name: kube-controller-manager
        namespace: kube-system
      spec:
        hostNetwork: true
        containers:
        - name: kube-controller-manager
          image: quay.io/coreos/hyperkube:v1.3.6_coreos.0
          command:
          - /hyperkube
          - controller-manager
          - --master=http://127.0.0.1:8080
          - --leader-elect=true
          - --service-account-private-key-file=/etc/kubernetes/ssl/apiserver-key.pem
          - --root-ca-file=/etc/kubernetes/ssl/ca.pem
          livenessProbe:
            httpGet:
              host: 127.0.0.1
              path: /healthz
              port: 10252
            initialDelaySeconds: 15
            timeoutSeconds: 1
          volumeMounts:
          - mountPath: /etc/kubernetes/ssl
            name: ssl-certs-kubernetes
            readOnly: true
          - mountPath: /etc/ssl/certs
            name: ssl-certs-host
            readOnly: true
        volumes:
        - hostPath:
            path: /etc/kubernetes/ssl
          name: ssl-certs-kubernetes
        - hostPath:
            path: /usr/share/ca-certificates
          name: ssl-certs-host
  - path: "/etc/kubernetes/manifests/kube-proxy.yaml"
    content: |
      apiVersion: v1
      kind: Pod
      metadata:
        name: kube-proxy
        namespace: kube-system
      spec:
        hostNetwork: true
        containers:
        - name: kube-proxy
          image: quay.io/coreos/hyperkube:v1.3.6_coreos.0
          command:
          - /hyperkube
          - proxy
          - --master=http://127.0.0.1:8080
          - --proxy-mode=iptables
          securityContext:
            privileged: true
          volumeMounts:
          - mountPath: /etc/ssl/certs
            name: ssl-certs-host
            readOnly: true
        volumes:
        - hostPath:
            path: /usr/share/ca-certificates
          name: ssl-certs-host
  - path: "/etc/kubernetes/manifests/kube-apiserver.yaml"
    content: |
      apiVersion: v1
      kind: Pod
      metadata:
        name: kube-apiserver
        namespace: kube-system
      spec:
        hostNetwork: true
        containers:
        - name: kube-apiserver
          image: quay.io/coreos/hyperkube:v1.3.6_coreos.0
          command:
          - /hyperkube
          - apiserver
          - --bind-address=0.0.0.0
          - --etcd-servers=https://10.79.218.2:2379,https://10.79.218.3:2379
          - --allow-privileged=true
          - --service-cluster-ip-range=10.0.0.0/24
          - --secure-port=443
          - --advertise-address=10.79.218.2
          - --admission-control=NamespaceLifecycle,LimitRanger,ServiceAccount,ResourceQuota
          - --tls-cert-file=/etc/kubernetes/ssl/apiserver.pem
          - --tls-private-key-file=/etc/kubernetes/ssl/apiserver-key.pem
          - --client-ca-file=/etc/kubernetes/ssl/ca.pem
          - --service-account-key-file=/etc/kubernetes/ssl/apiserver-key.pem
          - --runtime-config=extensions/v1beta1=true,extensions/v1beta1/networkpolicies=true
          ports:
          - containerPort: 443
            hostPort: 443
            name: https
          - containerPort: 8080
            hostPort: 8080
            name: local
          volumeMounts:
          - mountPath: /etc/kubernetes/ssl
            name: ssl-certs-kubernetes
            readOnly: true
          - mountPath: /etc/ssl/certs
            name: ssl-certs-host
            readOnly: true
        volumes:
        - hostPath:
            path: /etc/kubernetes/ssl
          name: ssl-certs-kubernetes
        - hostPath:
            path: /usr/share/ca-certificates
          name: ssl-certs-host
  - path: "/etc/flannel/options.env"
    content: |
     FLANNELD_IFACE=10.79.218.2
     FLANNELD_ETCD_ENDPOINTS=https://10.79.218.2:2379,https://10.79.218.3:2379
  - path: "/etc/kubernetes/ssl/ca.pem"
    permissions: "0600"
    owner: "root:root"
    content: |
      -----BEGIN CERTIFICATE-----
      ...
      -----END CERTIFICATE-----
  - path: "/etc/kubernetes/ssl/apiserver.pem"
    permissions: "0600"
    owner: "root:root"
    content: |
      -----BEGIN CERTIFICATE-----
      ...
      -----END CERTIFICATE-----
  - path: "/etc/kubernetes/ssl/apiserver-key.pem"
    permissions: "0600"
    owner: "root:root"
    content: |
      -----BEGIN RSA PRIVATE KEY-----
      ...
      -----END RSA PRIVATE KEY-----
 ssh_authorized_keys:
          - "ssh-rsa ... ufk@ufk-osx-music"
 coreos:
   update:
     reboot-strategy: etcd-lock
   etcd2:
     discovery: https://discovery.etcd.io/...
    # multi-region and multi-cloud deployments need to use $public_ipv4
     advertise-client-urls: https://10.79.218.2:2379
     initial-advertise-peer-urls: https://10.79.218.2:2380
     # listen on both the official ports and the legacy ports
     # legacy ports can be omitted if your application doesn't depend on them
     listen-client-urls: https://0.0.0.0:2379
     listen-peer-urls: https://10.79.218.2:2380
   units:
     - name: 00-enp4s0.network
       runtime: true
       content: |
        [Match]
        Name=enp4s0

        [Network]
        Address=10.79.218.2/24
        Gateway=10.79.218.232
        DNS=8.8.8.8
     - name: var-lib-rkt.mount
       enable: true
       command: start
       content: |
         [Mount]
         What=/dev/disk/by-uuid/23cc3284-e121-4448-b755-b74b5a022251
         Where=/var/lib/rkt
         Type=btrfs
         Options=loop,discard
     - name: etcd2.service
       command: start
       drop-ins:
        - name: 30-certs.conf
          content: |
           [Service]
           Environment="ETCD_CERT_FILE=/etc/ssl/etcd/server1.pem"
           Environment="ETCD_KEY_FILE=/etc/ssl/etcd/server1-key.pem"
           Environment="ETCD_TRUSTED_CA_FILE=/etc/ssl/etcd/ca.pem"
           Environment="ETCD_CLIENT_CERT_AUTH=true"
           Environment="ETCD_PEER_CERT_FILE=/etc/ssl/etcd/server1.pem"
           Environment="ETCD_PEER_KEY_FILE=/etc/ssl/etcd/server1-key.pem"
           Environment="ETCD_PEER_TRUSTED_CA_FILE=/etc/ssl/etcd/ca.pem"
           Environment="ETCD_PEER_CLIENT_CERT_AUTH=true"
     - name: flanneld.service
       command: start
       drop-ins:
        - name: 50-network-config.conf
          content: |
           [Service]
           ExecStartPre=/usr/bin/etcdctl set /coreos.com/network/config '{"Network":"10.1.0.0/16", "Backend": {"Type": "vxlan"}}'
        - name: 40-ExecStartPre-symlink.conf
          content: |
           [Service]
           ExecStartPre=/usr/bin/ln -sf /etc/flannel/options.env /run/flannel/options.env
     - name: kubelet.service
       enable: false
       content: |
        [Service]
        ExecStartPre=/usr/bin/mkdir -p /etc/kubernetes/manifests

        Environment=KUBELET_VERSION=v1.3.6_coreos.0
        ExecStart=/usr/lib/coreos/kubelet-wrapper \
           --api-servers=http://127.0.0.1:8080 \
           --network-plugin-dir=/etc/kubernetes/cni/net.d \
           --network-plugin=cni \
           --register-schedulable=false \
           --allow-privileged=true \
           --config=/etc/kubernetes/manifests \
           --hostname-override=10.79.218.2 \
           --cluster-dns=8.8.8.8 \
           --cluster-domain=cluster.local
        Restart=always
        RestartSec=10
        [Install]
        WantedBy=multi-user.target
     - name: calico-node.service
       enable: false
       content: |
        [Unit]
        Description=Calico per-host agent
        Requires=network-online.target
        After=network-online.target

        [Service]
        Slice=machine.slice
        Environment=CALICO_DISABLE_FILE_LOGGING=true
        Environment=HOSTNAME=10.79.218.2
        Environment=IP=${ADVERTISE_IP}
        Environment=FELIX_FELIXHOSTNAME=10.79.218.2
        Environment=CALICO_NETWORKING=false
        Environment=NO_DEFAULT_POOLS=true
        Environment=ETCD_ENDPOINTS=https://10.79.218.2:2379,https://10.79.218.3:2379
        ExecStart=/usr/bin/rkt run --inherit-env --stage1-from-dir=stage1-fly.aci \
        --volume=modules,kind=host,source=/lib/modules,readOnly=false \
        --mount=volume=modules,target=/lib/modules \
        --trust-keys-from-https quay.io/calico/node:v0.19.0

        KillMode=mixed
        Restart=always
        TimeoutStartSec=0

        [Install]
        WantedBy=multi-user.target

• i removed important stuff (like keys) and replaced them with ...
• i have kubelet and calico configured but disabled for now till i'll get etcd2 up and running with tls.

so I'm trying to run etcdctl from my mac pro laptop to list the etcd2 members (while being connected to the same local network) using the command

./etcdctl --endpoints="https://10.79.218.2:2379,https://10.79.218.3:2379" --key-file="./cfssl/client-key.pem" --cert-file="./cfssl/client.pem" --ca-file="./cfssl/ca.pem"  member list

and I get in response

Try '--no-sync' if you want to access non-published client endpoints(https://10.79.218.2:2379,https://10.79.218.3:2379).
Error:  client: no endpoints available

running the same command with --no-sync results the following:

Failed to get leader:  client: etcd cluster is unavailable or misconfigured

I created the certificates as described in https://github.com/coreos/docs/blob/master/os/generate-self-signed-certificates.md

what am I missing ?

thanks

update

so i found out about https://github.com/coreos/etcd/tree/master/hack/tls-setup. and I re-created the certificates using this script, and I configured my coreos etcd2 servers with these scripts.

it created the following certificates: ca-key.pem ca.csr ca.pem etcd1-key.pem etcd1.csr etcd1.pem etcd2-key.pem etcd2.csr etcd2.pem etcd3-key.pem etcd3.csr etcd3.pem proxy1-key.pem proxy1.csr proxy1.pem

the problem is that I don't see any client certificate.. so I don't quite understand what certificates to pass as parameters for etcdctl to work from my desktop. any ideas ?

update 2

so i found how to use etcdctl with tls.. i think. but still I get errors. I executed the following command:

/etcdctl --endpoint=https://10.79.218.2:2379 --ca-file=./certs/ca.pem --cert-file=./certs/etcd1.pem --key-file=./certs/etcd1-key.pem --no-sync ls

but I get the error

Error:  client: etcd cluster is unavailable or misconfigured
error #0: x509: cannot validate certificate for 10.79.218.2 because it doesn't contain any IP SANs