Home / server / Questions / 1143548
Accepted
motorbass
Asked: 2023-09-12 22:29:04 +0800 CST

AWX error X509 using custom EE image with pyopenssl

  • 772

I'm currently setting up an AWX platform hosted on K8s cluster to get a proper UI + features for multi-user purpose.

Context :
I created an EE image pushed on a Nexus repository that AWX use in order to have all the ansible galaxy collections that Ansible projects need (nutanix.ncp, community.hashi_vault, community.windows and ansible.windows) + pip modules (ansible-pylibssh, hvac, paramiko, pexpect, pykerberos, pywinrm, cryptography, pyopenssl).

EE image creation and push/pull is ok. If needed I may share the requirements.yml and requirements.txt files.

Here's the execution-environment.yml for reference :

---
version: 1

build_arg_defaults:
  EE_BASE_IMAGE: 'quay.io/ansible/awx-ee:latest'

dependencies:
  galaxy: requirements.yml
  python: requirements.txt

additional_build_steps:
  prepend: |
    RUN python3 -m pip install --upgrade pip
    RUN pip3 install --upgrade pip setuptools
    RUN whoami
    RUN cat /etc/os-release

  append:
  - RUN ls -la /etc

Then, when I setup a project on AWX using this EE image, it fails on X509_V_FLAG_CB_ISSUER_CHECK error :

{
  "module_stdout": "",
  "module_stderr": "Traceback (most recent call last):\n  File \"/root/.ansible/tmp/ansible-tmp-1694504214.889407-61-240849331705059/AnsiballZ_ntnx_subnets_info.py\", line 107, in <module>\n    _ansiballz_main()\n  File \"/root/.ansible/tmp/ansible-tmp-1694504214.889407-61-240849331705059/AnsiballZ_ntnx_subnets_info.py\", line 99, in _ansiballz_main\n    invoke_module(zipped_mod, temp_path, ANSIBALLZ_PARAMS)\n  File \"/root/.ansible/tmp/ansible-tmp-1694504214.889407-61-240849331705059/AnsiballZ_ntnx_subnets_info.py\", line 47, in invoke_module\n    runpy.run_module(mod_name='ansible_collections.nutanix.ncp.plugins.modules.ntnx_subnets_info', init_globals=dict(_module_fqn='ansible_collections.nutanix.ncp.plugins.modules.ntnx_subnets_info', _modlib_path=modlib_path),\n  File \"/usr/lib64/python3.8/runpy.py\", line 207, in run_module\n    return _run_module_code(code, init_globals, run_name, mod_spec)\n  File \"/usr/lib64/python3.8/runpy.py\", line 97, in _run_module_code\n    _run_code(code, mod_globals, init_globals,\n  File \"/usr/lib64/python3.8/runpy.py\", line 87, in _run_code\n    exec(code, run_globals)\n  File \"/tmp/ansible_ntnx_subnets_info_payload_mpaf5bgi/ansible_ntnx_subnets_info_payload.zip/ansible_collections/nutanix/ncp/plugins/modules/ntnx_subnets_info.py\", line 188, in <module>\n  File \"<frozen importlib._bootstrap>\", line 991, in _find_and_load\n  File \"<frozen importlib._bootstrap>\", line 975, in _find_and_load_unlocked\n  File \"<frozen importlib._bootstrap>\", line 655, in _load_unlocked\n  File \"<frozen importlib._bootstrap>\", line 618, in _load_backward_compatible\n  File \"<frozen zipimport>\", line 259, in load_module\n  File \"/tmp/ansible_ntnx_subnets_info_payload_mpaf5bgi/ansible_ntnx_subnets_info_payload.zip/ansible_collections/nutanix/ncp/plugins/module_utils/prism/subnets.py\", line 9, in <module>\n  File \"<frozen importlib._bootstrap>\", line 991, in _find_and_load\n  File \"<frozen importlib._bootstrap>\", line 975, in _find_and_load_unlocked\n  File \"<frozen importlib._bootstrap>\", line 655, in _load_unlocked\n  File \"<frozen importlib._bootstrap>\", line 618, in _load_backward_compatible\n  File \"<frozen zipimport>\", line 259, in load_module\n  File \"/tmp/ansible_ntnx_subnets_info_payload_mpaf5bgi/ansible_ntnx_subnets_info_payload.zip/ansible_collections/nutanix/ncp/plugins/module_utils/prism/clusters.py\", line 7, in <module>\n  File \"<frozen importlib._bootstrap>\", line 991, in _find_and_load\n  File \"<frozen importlib._bootstrap>\", line 975, in _find_and_load_unlocked\n  File \"<frozen importlib._bootstrap>\", line 655, in _load_unlocked\n  File \"<frozen importlib._bootstrap>\", line 618, in _load_backward_compatible\n  File \"<frozen zipimport>\", line 259, in load_module\n  File \"/tmp/ansible_ntnx_subnets_info_payload_mpaf5bgi/ansible_ntnx_subnets_info_payload.zip/ansible_collections/nutanix/ncp/plugins/module_utils/prism/prism.py\", line 5, in <module>\n  File \"<frozen importlib._bootstrap>\", line 991, in _find_and_load\n  File \"<frozen importlib._bootstrap>\", line 975, in _find_and_load_unlocked\n  File \"<frozen importlib._bootstrap>\", line 655, in _load_unlocked\n  File \"<frozen importlib._bootstrap>\", line 618, in _load_backward_compatible\n  File \"<frozen zipimport>\", line 259, in load_module\n  File \"/tmp/ansible_ntnx_subnets_info_payload_mpaf5bgi/ansible_ntnx_subnets_info_payload.zip/ansible_collections/nutanix/ncp/plugins/module_utils/entity.py\", line 13, in <module>\n  File \"<frozen importlib._bootstrap>\", line 991, in _find_and_load\n  File \"<frozen importlib._bootstrap>\", line 975, in _find_and_load_unlocked\n  File \"<frozen importlib._bootstrap>\", line 655, in _load_unlocked\n  File \"<frozen importlib._bootstrap>\", line 618, in _load_backward_compatible\n  File \"<frozen zipimport>\", line 259, in load_module\n  File \"/tmp/ansible_ntnx_subnets_info_payload_mpaf5bgi/ansible_ntnx_subnets_info_payload.zip/ansible/module_utils/urls.py\", line 115, in <module>\n  File \"/usr/lib/python3.8/site-packages/urllib3/contrib/pyopenssl.py\", line 46, in <module>\n    import OpenSSL.SSL\n  File \"/usr/local/lib/python3.8/site-packages/OpenSSL/__init__.py\", line 8, in <module>\n    from OpenSSL import crypto, SSL\n  File \"/usr/local/lib/python3.8/site-packages/OpenSSL/crypto.py\", line 1517, in <module>\n    class X509StoreFlags(object):\n  File \"/usr/local/lib/python3.8/site-packages/OpenSSL/crypto.py\", line 1537, in X509StoreFlags\n    CB_ISSUER_CHECK = _lib.X509_V_FLAG_CB_ISSUER_CHECK\nAttributeError: module 'lib' has no attribute 'X509_V_FLAG_CB_ISSUER_CHECK'\n",
  "exception": "Traceback (most recent call last):\n  File \"/root/.ansible/tmp/ansible-tmp-1694504214.889407-61-240849331705059/AnsiballZ_ntnx_subnets_info.py\", line 107, in <module>\n    _ansiballz_main()\n  File \"/root/.ansible/tmp/ansible-tmp-1694504214.889407-61-240849331705059/AnsiballZ_ntnx_subnets_info.py\", line 99, in _ansiballz_main\n    invoke_module(zipped_mod, temp_path, ANSIBALLZ_PARAMS)\n  File \"/root/.ansible/tmp/ansible-tmp-1694504214.889407-61-240849331705059/AnsiballZ_ntnx_subnets_info.py\", line 47, in invoke_module\n    runpy.run_module(mod_name='ansible_collections.nutanix.ncp.plugins.modules.ntnx_subnets_info', init_globals=dict(_module_fqn='ansible_collections.nutanix.ncp.plugins.modules.ntnx_subnets_info', _modlib_path=modlib_path),\n  File \"/usr/lib64/python3.8/runpy.py\", line 207, in run_module\n    return _run_module_code(code, init_globals, run_name, mod_spec)\n  File \"/usr/lib64/python3.8/runpy.py\", line 97, in _run_module_code\n    _run_code(code, mod_globals, init_globals,\n  File \"/usr/lib64/python3.8/runpy.py\", line 87, in _run_code\n    exec(code, run_globals)\n  File \"/tmp/ansible_ntnx_subnets_info_payload_mpaf5bgi/ansible_ntnx_subnets_info_payload.zip/ansible_collections/nutanix/ncp/plugins/modules/ntnx_subnets_info.py\", line 188, in <module>\n  File \"<frozen importlib._bootstrap>\", line 991, in _find_and_load\n  File \"<frozen importlib._bootstrap>\", line 975, in _find_and_load_unlocked\n  File \"<frozen importlib._bootstrap>\", line 655, in _load_unlocked\n  File \"<frozen importlib._bootstrap>\", line 618, in _load_backward_compatible\n  File \"<frozen zipimport>\", line 259, in load_module\n  File \"/tmp/ansible_ntnx_subnets_info_payload_mpaf5bgi/ansible_ntnx_subnets_info_payload.zip/ansible_collections/nutanix/ncp/plugins/module_utils/prism/subnets.py\", line 9, in <module>\n  File \"<frozen importlib._bootstrap>\", line 991, in _find_and_load\n  File \"<frozen importlib._bootstrap>\", line 975, in _find_and_load_unlocked\n  File \"<frozen importlib._bootstrap>\", line 655, in _load_unlocked\n  File \"<frozen importlib._bootstrap>\", line 618, in _load_backward_compatible\n  File \"<frozen zipimport>\", line 259, in load_module\n  File \"/tmp/ansible_ntnx_subnets_info_payload_mpaf5bgi/ansible_ntnx_subnets_info_payload.zip/ansible_collections/nutanix/ncp/plugins/module_utils/prism/clusters.py\", line 7, in <module>\n  File \"<frozen importlib._bootstrap>\", line 991, in _find_and_load\n  File \"<frozen importlib._bootstrap>\", line 975, in _find_and_load_unlocked\n  File \"<frozen importlib._bootstrap>\", line 655, in _load_unlocked\n  File \"<frozen importlib._bootstrap>\", line 618, in _load_backward_compatible\n  File \"<frozen zipimport>\", line 259, in load_module\n  File \"/tmp/ansible_ntnx_subnets_info_payload_mpaf5bgi/ansible_ntnx_subnets_info_payload.zip/ansible_collections/nutanix/ncp/plugins/module_utils/prism/prism.py\", line 5, in <module>\n  File \"<frozen importlib._bootstrap>\", line 991, in _find_and_load\n  File \"<frozen importlib._bootstrap>\", line 975, in _find_and_load_unlocked\n  File \"<frozen importlib._bootstrap>\", line 655, in _load_unlocked\n  File \"<frozen importlib._bootstrap>\", line 618, in _load_backward_compatible\n  File \"<frozen zipimport>\", line 259, in load_module\n  File \"/tmp/ansible_ntnx_subnets_info_payload_mpaf5bgi/ansible_ntnx_subnets_info_payload.zip/ansible_collections/nutanix/ncp/plugins/module_utils/entity.py\", line 13, in <module>\n  File \"<frozen importlib._bootstrap>\", line 991, in _find_and_load\n  File \"<frozen importlib._bootstrap>\", line 975, in _find_and_load_unlocked\n  File \"<frozen importlib._bootstrap>\", line 655, in _load_unlocked\n  File \"<frozen importlib._bootstrap>\", line 618, in _load_backward_compatible\n  File \"<frozen zipimport>\", line 259, in load_module\n  File \"/tmp/ansible_ntnx_subnets_info_payload_mpaf5bgi/ansible_ntnx_subnets_info_payload.zip/ansible/module_utils/urls.py\", line 115, in <module>\n  File \"/usr/lib/python3.8/site-packages/urllib3/contrib/pyopenssl.py\", line 46, in <module>\n    import OpenSSL.SSL\n  File \"/usr/local/lib/python3.8/site-packages/OpenSSL/__init__.py\", line 8, in <module>\n    from OpenSSL import crypto, SSL\n  File \"/usr/local/lib/python3.8/site-packages/OpenSSL/crypto.py\", line 1517, in <module>\n    class X509StoreFlags(object):\n  File \"/usr/local/lib/python3.8/site-packages/OpenSSL/crypto.py\", line 1537, in X509StoreFlags\n    CB_ISSUER_CHECK = _lib.X509_V_FLAG_CB_ISSUER_CHECK\nAttributeError: module 'lib' has no attribute 'X509_V_FLAG_CB_ISSUER_CHECK'\n",
  "msg": "MODULE FAILURE\nSee stdout/stderr for the exact error",
  "rc": 1,
  "_ansible_no_log": false,
  "changed": false
}

Tests : I've tried the following tests to try to resolve it :

  • get the latest version of cryptography and pyopenssl
  • downgrade cryptography to 36.0.2/37.0.0 and pyopenssl version to 22.0.0 as seen on some stackoverflow posts
  • RUN pip3 upgrade on the additional_build_steps > prepend block in image creation

In addition, there's something i don't understand :
I saw that the error log mention the file \"/usr/lib/python3.8/site-packages/urllib3/contrib/pyopenssl.py But when i run a shell the docker image there's no /usr/lib/python3.8 but python3.9 folder.

Questions: When I log to AWX pods, i found out that only awx-operator-controller-manager pod that got that path python3.8 path. So what's the relation between AWX pods/ EE image when trying to run a project ?

About the pyopenssl, what other tests/solution may I test ?

Thanks !

ansible

1 Answers

  1. Best Answer

    Finally it was my mistake... After re-creating the EE image, clean my repository and push it again, it works (every requirements is up-to-date)

    I think I push/pull the wrong image while testing..anyways, everything's fine !

    • 0