Home / ubuntu / Questions / 926021
In Process
Mirto Busico
Asked: 2017-06-17 01:30:03 +0800 CST

Juju bootstrap of LXD fails behind a proxy

  • 772

I'm trying to bootstrap juju on a virtualbox machine with ubuntu 16.04 using lxd cloud obtainig the error

ERROR Get https://10.229.0.1:8443/1.0: Gateway Time-out

Snap says:

sysop@ubuntu01:~$ sudo snap list
[sudo] password for sysop: 
Name  Version  Rev   Developer  Notes
core  16-2     1689  canonical  -
juju  2.2.0    1956  canonical  classic
lxd   2.14     2417  canonical  -

LXC is working correctly and I have a container delivered. lxc list says:

sysop@ubuntu01:~$ lxc list --verbose
+-------+---------+--------------------+------+------------+-----------+
| NAME  |  STATE  |        IPV4        | IPV6 |    TYPE    | SNAPSHOTS |
+-------+---------+--------------------+------+------------+-----------+
| primo | RUNNING | 10.229.0.73 (eth0) |      | PERSISTENT | 0         |
+-------+---------+--------------------+------+------------+-----------+

I'm behind a proxy that requires authentication so in lxc info I have:

sysop@ubuntu01:~$ lxc info
config:
   core.https_address: '[::]'
   core.proxy_http: http://user:password@proxy-address:8080/
   core.proxy_https: http://user:password@proxy-address:8080/
   core.proxy_ignore_hosts: localhost

The output of juju bootstrap is:

    sysop@ubuntu01:~$ juju bootstrap localhost lxd1 --debug
    09:34:53 INFO  juju.cmd supercommand.go:63 running juju [2.2.0 gc go1.8]
    09:34:53 DEBUG juju.cmd supercommand.go:64   args: []string{"/snap/juju/1956/bin/juju", "bootstrap", "localhost", "lxd1", "--debug"}
    09:34:53 DEBUG juju.tools.lxdclient client.go:230 connecting to LXD remote "local": "unix:///var/snap/lxd/common/lxd/unix.socket"
    09:34:54 INFO  juju.tools.lxdclient client.go:284 using LXD API version "1.0"
    09:34:54 INFO  juju.tools.lxdclient client.go:333 LXD "default" profile uses network bridge "lxdbr0"
    09:34:54 INFO  cmd provider.go:179 Resolved LXD host address on bridge lxdbr0: 10.229.0.1:8443
    09:34:54 INFO  cmd cloudcredential.go:51 no credentials found, checking environment
    09:34:54 DEBUG juju.tools.lxdclient client.go:230 connecting to LXD remote "local": "unix:///var/snap/lxd/common/lxd/unix.socket"
    09:34:54 INFO  juju.tools.lxdclient client.go:284 using LXD API version "1.0"
    09:34:54 INFO  juju.tools.lxdclient client.go:333 LXD "default" profile uses network bridge "lxdbr0"
    09:34:54 DEBUG juju.cmd.juju.commands bootstrap.go:803 authenticating with region "" and credential "localhost" (LXD credential "localhost")
    09:34:54 DEBUG juju.cmd.juju.commands bootstrap.go:915 provider attrs: map[]
    09:34:54 INFO  cmd authkeys.go:114 Adding contents of "/home/sysop/.local/share/juju/ssh/juju_id_rsa.pub" to authorized-keys
    09:34:54 INFO  cmd authkeys.go:114 Adding contents of "/home/sysop/.ssh/id_rsa.pub" to authorized-keys
    09:34:54 DEBUG juju.cmd.juju.commands bootstrap.go:971 preparing controller with config: map[net-bond-reconfigure-delay:17 disable-network-management:false authorized-keys:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCpLnUIrvGmeOMKaH0KLltB94Hn20c4ARM6/vSW6PguUr04+EzU2q9s/GFiIq7VjC0aWxQydDJG/4HqThf0VX5nB9h5sYBLIaq8LoMemw/FtsZrnQ2En3u9kngvcYkRmQEcyCzdHmhzVKG1JAE3NYz67aCaa2OAWZw7fDcXP78KsYDuSv8eTFr4qPoGGi2ayG/NeXkM8WgS8fgQI9emOn9zFVrHIo7Ah2Wa9ucUJjXQvNA+6hZjM2DnxeLV4gdVLxLBvxF1Vj9MHqArPBUZ4ZEFGcdXYoPyny2S6iT3vf3fbCaU785JNweCQwZ0oxRXCDYWXPhGY4zVwzUK1ykojOrh juju-client-key
    ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCjq011k7jLFei5RL7m07dDwEHA4hoKI9AN/WlfjH+ThrUErSV4vG3SRL3jClUldirTTLeT3XwwiGMpLAfxtamI/cbFjuqCY7nHJ+zgQ+GfmgpMfulYxJgz9D1oGdL8+Mn3EfG5V4pfDe1uTiOkmKkChh53rWTznAuDQYbFaXlbQdsVbUDRjmhuAjKnPqX/81WKeB4XedNAsycsqhBmK2ITx22ff2PwkHaegsGUzbXcRVY8i47gq96vCU3gLb64fUkTA9miEAovtpalsr13jSbzTrLfwtExVbpew0EAb4de5BqQATqOtexD6CNzkxEzmHp7am0c8MdzMjBFpkCIN29/ sysop@ubuntu01
    automatically-retry-hooks:true agent-metadata-url: type:lxd ignore-machine-addresses:false proxy-ssh:false image-stream:released enable-os-upgrade:true apt-ftp-proxy: ftp-proxy: http-proxy: transmit-vendor-metrics:true name:controller apt-https-proxy: firewall-mode:instance logforward-enabled:false default-series:xenial max-status-history-age:336h ssl-hostname-verification:true no-proxy:127.0.0.1,localhost,::1 apt-mirror: https-proxy: max-status-history-size:5G apt-no-proxy: resource-tags: apt-http-proxy: logging-config: agent-stream:released uuid:2168b6e8-f8b8-4edb-8982-191e4b407901 test-mode:false provisioner-harvest-mode:destroyed image-metadata-url: development:false enable-os-refresh-update:true]
    09:34:54 DEBUG juju.tools.lxdclient client.go:230 connecting to LXD remote "remote": "10.229.0.1:8443"
    ERROR Get https://10.229.0.1:8443/1.0: Gateway Time-out
    09:34:54 DEBUG cmd supercommand.go:459 error stack: 
    Get https://10.229.0.1:8443/1.0: Gateway Time-out
    github.com/juju/juju/tools/lxdclient/client.go:278: 
    github.com/juju/juju/tools/lxdclient/client.go:132: 
    github.com/juju/juju/provider/lxd/environ_raw.go:92: 
    github.com/juju/juju/provider/lxd/environ.go:67: 
    github.com/juju/juju/provider/lxd/provider.go:57: 
    github.com/juju/juju/environs/bootstrap/prepare.go:160: 
    github.com/juju/juju/environs/bootstrap/prepare.go:99: 
    github.com/juju/juju/cmd/juju/commands/bootstrap.go:450:

What can I do to bootstrap JUJU?

UPDATE:

Tried everything suggested in this thread: Juju 2.0 proxy for bootstrap

--model-defaults is undefined for juju 2.2.0

--config is simply ignored

sysop@ubuntu01:~$ juju bootstrap lxd --model-defaults=config.yaml ERROR flag provided but not defined: --model-defaults sysop@ubuntu01:~$ juju bootstrap lxd --config=config.yaml ERROR Get https://10.229.0.1:8443/1.0: Gateway Time-out

I don't know what else to try. Any suggestion?

One step ahead.

Setting the core.http_address to the lxdbr0 address and core.proxy_ignore_hosts to the same address the bootstrap process starts and create the lxc controller container but end in error saying:

09:30:29 ERROR juju.cmd.juju.commands bootstrap.go:492 unable to contact api server after 1 attempts: unable to connect to API: malformed HTTP response "\x15\x03\x01\x00\x02\x02\x16"

Then the process ends cleaning up the created container.

What can I do?

Lxc info says:

    sysop@ubuntu01:~$ lxc info
    config:
    core.https_address: 10.229.0.1:9000
    core.https_allowed_credentials: "true"
    core.https_allowed_headers: Origin, X-Requested-With, Content-Type, Accept
    core.https_allowed_methods: GET, POST, PUT, DELETE, OPTIONS
    core.https_allowed_origin: https://localhost:8000
    core.proxy_http: http://username:password@proxy:8080/
    core.proxy_https: http://username:password@proxy:8080/
    core.proxy_ignore_hosts: 10.229.0.1

config.yaml contains:

    default-series: xenial
    no-proxy: 10.229.0.1
    apt-http-proxy: http://username:password@proxy:8080
    apt-https-proxy: http://username:password@proxy:8080
    apt-ftp-proxy: http://username:password@proxy:8080
    http-proxy: http://username:password@proxy:8080
    https-proxy: http://username:password@proxy:8080
    ftp-proxy: http://username:password@proxy:8080

and the bootstrap log is:

    sysop@ubuntu01:~$ juju bootstrap localhost --config config.yaml --debug
    09:09:10 INFO  juju.cmd supercommand.go:63 running juju [2.2.0 gc go1.8]
    09:09:10 DEBUG juju.cmd supercommand.go:64   args: []string{"/snap/juju/1956/bin/juju", "bootstrap", "localhost", "--config", "config.yaml", "--debug"}
    09:09:10 DEBUG juju.tools.lxdclient client.go:230 connecting to LXD remote "local": "unix:///var/snap/lxd/common/lxd/unix.socket"
    09:09:10 INFO  juju.tools.lxdclient client.go:284 using LXD API version "1.0"
    09:09:10 INFO  juju.tools.lxdclient client.go:333 LXD "default" profile uses network bridge "lxdbr0"
    09:09:10 INFO  cmd provider.go:179 Resolved LXD host address on bridge lxdbr0: 10.229.0.1:9000
    09:09:10 INFO  cmd cloudcredential.go:51 no credentials found, checking environment
    09:09:10 DEBUG juju.tools.lxdclient client.go:230 connecting to LXD remote "local": "unix:///var/snap/lxd/common/lxd/unix.socket"
    09:09:10 INFO  juju.tools.lxdclient client.go:284 using LXD API version "1.0"
    09:09:10 INFO  juju.tools.lxdclient client.go:333 LXD "default" profile uses network bridge "lxdbr0"
    09:09:10 DEBUG juju.cmd.juju.commands bootstrap.go:803 authenticating with region "" and credential "localhost" (LXD credential "localhost")
    09:09:10 DEBUG juju.cmd.juju.commands bootstrap.go:915 provider attrs: map[]
    09:09:11 INFO  cmd authkeys.go:114 Adding contents of "/home/sysop/.local/share/juju/ssh/juju_id_rsa.pub" to authorized-keys
    09:09:11 INFO  cmd authkeys.go:114 Adding contents of "/home/sysop/.ssh/id_rsa.pub" to authorized-keys
    09:09:11 DEBUG juju.cmd.juju.commands bootstrap.go:971 preparing controller with config: map[logging-config: authorized-keys:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCpLnUIrvGmeOMKaH0KLltB94Hn20c4ARM6/vSW6PguUr04+EzU2q9s/GFiIq7VjC0aWxQydDJG/4HqThf0VX5nB9h5sYBLIaq8LoMemw/FtsZrnQ2En3u9kngvcYkRmQEcyCzdHmhzVKG1JAE3NYz67aCaa2OAWZw7fDcXP78KsYDuSv8eTFr4qPoGGi2ayG/NeXkM8WgS8fgQI9emOn9zFVrHIo7Ah2Wa9ucUJjXQvNA+6hZjM2DnxeLV4gdVLxLBvxF1Vj9MHqArPBUZ4ZEFGcdXYoPyny2S6iT3vf3fbCaU785JNweCQwZ0oxRXCDYWXPhGY4zVwzUK1ykojOrh juju-client-key
    ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCjq011k7jLFei5RL7m07dDwEHA4hoKI9AN/WlfjH+ThrUErSV4vG3SRL3jClUldirTTLeT3XwwiGMpLAfxtamI/cbFjuqCY7nHJ+zgQ+GfmgpMfulYxJgz9D1oGdL8+Mn3EfG5V4pfDe1uTiOkmKkChh53rWTznAuDQYbFaXlbQdsVbUDRjmhuAjKnPqX/81WKeB4XedNAsycsqhBmK2ITx22ff2PwkHaegsGUzbXcRVY8i47gq96vCU3gLb64fUkTA9miEAovtpalsr13jSbzTrLfwtExVbpew0EAb4de5BqQATqOtexD6CNzkxEzmHp7am0c8MdzMjBFpkCIN29/ sysop@ubuntu01
    max-status-history-age:336h enable-os-upgrade:true test-mode:false firewall-mode:instance image-metadata-url: image-stream:released ftp-proxy:http://username:password@proxy:8080 agent-stream:released name:controller development:false agent-metadata-url: resource-tags: transmit-vendor-metrics:true http-proxy:http://username:password@proxy:8080 max-status-history-size:5G apt-no-proxy: apt-mirror: net-bond-reconfigure-delay:17 apt-http-proxy:http://username:password@proxy:8080 default-series:xenial type:lxd logforward-enabled:false enable-os-refresh-update:true apt-https-proxy:http://username:password@proxy:8080 provisioner-harvest-mode:destroyed uuid:dc69de65-706c-4839-8456-b3afc7dce7f9 proxy-ssh:false automatically-retry-hooks:true ssl-hostname-verification:true no-proxy:10.229.0.1 apt-ftp-proxy:http://username:password@proxy:8080 disable-network-management:false https-proxy:http://username:password@proxy:8080 ignore-machine-addresses:false]
    09:09:11 DEBUG juju.tools.lxdclient client.go:230 connecting to LXD remote "remote": "10.229.0.1:9000"
    09:09:11 INFO  juju.tools.lxdclient client.go:284 using LXD API version "1.0"
    09:09:11 INFO  cmd bootstrap.go:481 Creating Juju controller "localhost-localhost" on localhost/localhost
    09:09:11 INFO  juju.cmd.juju.commands bootstrap.go:536 combined bootstrap constraints: 
    09:09:11 DEBUG juju.environs.bootstrap bootstrap.go:199 model "controller" supports service/machine networks: false
    09:09:11 DEBUG juju.environs.bootstrap bootstrap.go:201 network management by juju enabled: true
    09:09:11 INFO  cmd bootstrap.go:233 Loading image metadata
    09:09:11 INFO  cmd bootstrap.go:296 Looking for packaged Juju agent version 2.2.0 for amd64
    09:09:11 INFO  juju.environs.bootstrap tools.go:72 looking for bootstrap agent binaries: version=2.2.0
    09:09:11 INFO  juju.environs.tools tools.go:101 finding agent binaries in stream "released"
    09:09:11 INFO  juju.environs.tools tools.go:103 reading agent binaries with major.minor version 2.2
    09:09:11 INFO  juju.environs.tools tools.go:111 filtering agent binaries by version: 2.2.0
    09:09:11 INFO  juju.environs.tools tools.go:117 filtering agent binaries by architecture: amd64
    09:09:11 DEBUG juju.environs.tools urls.go:109 trying datasource "keystone catalog"
    09:09:12 DEBUG juju.environs.simplestreams simplestreams.go:683 using default candidate for content id "com.ubuntu.juju:released:tools" are {20161007 mirrors:1.0 content-download streams/v1/cpc-mirrors.sjson []}
    09:09:14 INFO  juju.environs.bootstrap tools.go:74 found 15 packaged agent binaries
    09:09:14 INFO  cmd bootstrap.go:357 Starting new instance for initial controller
    09:09:14 INFO  cmd environ.go:152 To configure your system to better support LXD containers, please see: https://github.com/lxc/lxd/blob/master/doc/production-setup.md
    Launching controller instance(s) on localhost/localhost...
    09:09:14 DEBUG juju.provider.lxd environ_broker.go:40 StartInstance: "0", xenial
    09:09:14 DEBUG juju.cloudconfig.instancecfg instancecfg.go:825 Setting numa ctl preference to false
    09:09:14 DEBUG juju.environs imagemetadata.go:112 obtained image datasource "default cloud images"
    09:09:14 DEBUG juju.environs imagemetadata.go:112 obtained image datasource "default ubuntu cloud images"
    09:09:14 DEBUG juju.provider.lxd environ_broker.go:112 LXD requires https://, using: https://cloud-images.ubuntu.com/releases/
    09:09:14 DEBUG juju.tools.lxdclient client.go:230 connecting to LXD remote "default cloud images": "https://streams.canonical.com/juju/images/releases/"
    09:09:14 INFO  juju.tools.lxdclient client_image.go:166 no image for juju/xenial/amd64 found in https://streams.canonical.com/juju/images/releases/
    09:09:14 DEBUG juju.tools.lxdclient client.go:230 connecting to LXD remote "default ubuntu cloud images": "https://cloud-images.ubuntu.com/releases/"
    09:09:15 INFO  juju.tools.lxdclient client_image.go:170 found image from https://cloud-images.ubuntu.com/releases/ for juju/xenial/amd64 = 7a7ff654cbd8f5f09bec03aa19d8d7d92649127d18659036a963b1ea63f90d25
    09:09:17 INFO  juju.tools.lxdclient client_image.go:182 copying image for juju/xenial/amd64 from https://cloud-images.ubuntu.com/releases/: metadata: 100% (1.62GB/s)
    09:09:20 INFO  juju.tools.lxdclient client_image.go:182 copying image for juju/xenial/amd64 from https://cloud-images.ubuntu.com/releases/: rootfs: 1% (548.63kB/s)
    [...snip...]
    09:15:13 INFO  juju.tools.lxdclient client_image.go:182 copying image for juju/xenial/amd64 from https://cloud-images.ubuntu.com/releases/: rootfs: 99% (436.00kB/s)
    09:15:16 INFO  juju.tools.lxdclient client_image.go:182 copying image for juju/xenial/amd64 from https://cloud-images.ubuntu.com/releases/: rootfs: 100% (435.60kB/s)
    09:15:19 DEBUG juju.tools.lxdclient client_image.go:174 dropped 0 progress messagess/: rootfs: 100% (435.60kB/s)
    09:15:19 DEBUG juju.service discovery.go:63 discovered init system "systemd" from series "xenial"               
    09:15:19 DEBUG juju.provider.lxd environ_broker.go:230 LXD user data; 3981 bytes
    09:15:19 INFO  juju.provider.lxd environ_broker.go:211 starting instance "juju-dce7f9-0" (image "juju/xenial/amd64")...
    09:16:08 INFO  juju.provider.lxd environ_broker.go:56 started instance "juju-dce7f9-0"                          
    - juju-dce7f9-0 (arch=amd64)          
    09:16:09 INFO  juju.environs.bootstrap bootstrap.go:601 newest version: 2.2.0
    09:16:09 INFO  juju.environs.bootstrap bootstrap.go:616 picked bootstrap agent binary version: 2.2.0
    09:16:09 INFO  juju.environs.bootstrap bootstrap.go:393 Installing Juju agent on bootstrap instance
    09:16:10 INFO  cmd bootstrap.go:480 Fetching Juju GUI 2.7.3
    09:16:10 DEBUG juju.cloudconfig.instancecfg instancecfg.go:825 Setting numa ctl preference to false
    Waiting for address
    Attempting to connect to 10.229.0.159:22
    09:16:36 DEBUG juju.provider.common bootstrap.go:497 connection attempt for 10.229.0.159 failed: /var/lib/juju/nonce.txt does not exist
    09:16:41 INFO  juju.cloudconfig userdatacfg_unix.go:410 Fetching agent: curl -sSfw 'tools from %{url_effective} downloaded: HTTP %{http_code}; time %{time_total}s; size %{size_download} bytes; speed %{speed_download} bytes/s ' --retry 10 -o $bin/tools.tar.gz <[https://streams.canonical.com/juju/tools/agent/2.2.0/juju-2.2.0-ubuntu-amd64.tgz]>
    09:20:29 INFO  cmd bootstrap.go:420 Bootstrap agent now started
    09:20:30 INFO  juju.juju api.go:308 API endpoints changed from [] to [10.229.0.159:17070]
    09:20:31 INFO  cmd controller.go:82 Contacting Juju controller at 10.229.0.159 to verify accessibility...
    09:20:31 INFO  juju.juju api.go:67 connecting to API addresses: [10.229.0.159:17070]
    09:30:29 DEBUG juju.api apiclient.go:806 error dialing websocket: malformed HTTP response "\x15\x03\x01\x00\x02\x02\x16"
    09:30:29 ERROR juju.cmd.juju.commands bootstrap.go:492 unable to contact api server after 1 attempts: unable to connect to API: malformed HTTP response "\x15\x03\x01\x00\x02\x02\x16"
    09:30:29 DEBUG juju.cmd.juju.commands bootstrap.go:493 (error details: [{github.com/juju/juju/cmd/juju/common/controller.go:115: unable to contact api server after 1 attempts} {github.com/juju/juju/cmd/juju/common/controller.go:39: } {github.com/juju/juju/juju/api.go:72: } {github.com/juju/juju/api/apiclient.go:190: } {github.com/juju/juju/api/apiclient.go:596: } {github.com/juju/juju/api/apiclient.go:711: } {github.com/juju/juju/api/apiclient.go:807: unable to connect to API} {github.com/juju/juju/api/apiclient.go:831: } {malformed HTTP response "\x15\x03\x01\x00\x02\x02\x16"}])
    09:30:29 DEBUG juju.cmd.juju.commands bootstrap.go:1078 cleaning up after failed bootstrap
    09:30:29 INFO  juju.provider.common destroy.go:20 destroying model "controller"
    09:30:29 INFO  juju.provider.common destroy.go:31 destroying instances
    09:30:41 INFO  juju.provider.common destroy.go:51 destroying storage
    09:30:41 DEBUG juju.provider.lxd environ.go:193 instances: []
    09:30:41 INFO  cmd supercommand.go:465 command finished
    sysop@ubuntu
juju

1 Answers

  1. It would appear that the Juju controller for created in LXD (10.229.0.159) isn't being reached directly from the client but instead going through the proxy. Try using lxc network edit to decrease the DHCP range to around 10-15 addresses then update config.yaml put in a range of IP addresses in for the no-proxy.

    Proxies have always been trickier than necessary in Juju. I'll open a bug to add something like proxy-juju-network flag to the config. If set to True then all network traffic to bootstrap and machines launched by Juju will go through the proxy settings, if set to False (default) then any address in the Juju model (bootstrap and deployed machines) will automatically be added to the no-proxy list.

    • 0