Home / server / Questions / 1143713
Accepted
motorbass
Asked: 2023-09-14 22:29:07 +0800 CST

How to troubleshoot API requests sent by Ansible module?

  • 772

Here's the context :
The following playbook (simplified to one task for this stackoverflow topic and for ease of use) got different results depends on where I launch it. It uses the nutanix.ncp galaxy collection (tested with version 1.9.0, 1.8.0, 1.7.0...)

The ntnx_subnets_info method is called to retrieve all the list of existing subnet on a prism central instance, and is filtered through the name parameter to retrieve only details of a specific VLAN.

---
- name: test-get-subnet-info
  hosts: localhost

  vars:
    nutanix_host: "{{ XXXXXXX }}"
    nutanix_username: "{{ XXXXXXX }}"
    nutanix_password: "XXXXXXX "

  collections:
    - nutanix.ncp
  module_defaults:
    group/nutanix.ncp.ntnx:
      nutanix_host: "{{ XXXXXXX }}"
      nutanix_username: "{{ XXXXXXX }}"
      nutanix_password: "XXXXXXX "

  tasks:
  - name: Retrieve subnet info
    ntnx_subnets_info:
      filter:
        name: "my-VLAN"

On a Debian 11 server, this task run smoothly as you can see below enter image description here

Whereas on my custom AWX EE (tested both with docker then on K8s) I got the following error which isn't too explicit Failed to convert API response to json
enter image description here

Troubleshoot steps:

  • try to downgrade nutanix.ncp collection from 1.9.0 to 1.8.0 and 1.7.0 => still the same results (works on debian, not from docker)
  • compare ansible version => both environment run on ansible core 2.15.4
  • compare python version => docker image environment is on 3.9.17 and debian server on 3.9.2
  • launch manual curl requests from both environment => everything works on both environment
  • launch the playbook with -vvvvvv option and compare differences on the log => nutanix.ncp is not so chatty, i don't get any other errors as you can see on the screenshot beyond.

Questions

  • Is there a way to "analyze" Ansible API requests ? like a wireshark/fiddler for Ansible ?
  • how can I go further to troubleshoot and fix this issue ? as it works on one side but not on the others i may compare some stuff and hopefully find a difference ?

Thanks

docker

1 Answers

  1. Best Answer

    After many tests and investigations, just found entity.py in nutanix.ncp collection files. It was the file that is responsible of msg "Failed to convert API response to json"

    • One function sends the error message when URL response received code is >300. I went into a wireshark capture and figured out there was an error in the network flows : enter image description here

    • Then, it bring me to check connection from my docker image / K8s pod to my Prism Central.

      The command openssl s_client -connect fqdn_prism_central:9440 shows the error message

    “ Verify return code: 20 (unable to get local issuer certificate)"

    • Finally, I just update certificate chain on my docker image (through my Dockerfile) and everything is now fine in AWX.

      In my case, as AWX EE image is based on official awx-ee (https://quay.io/repository/ansible/awx-ee?tab=tags&tag=latest), I add the following steps to my dockerfile (note : path and command may differs if you're using something else than CentOS image) :

    COPY ./certificate_chain.pem /etc/pki/ca-trust/source/anchors/certificate_chain.pem
RUN chmod 644 /etc/pki/ca-trust/source/anchors/certificate_chain.pem && update-ca-trust extract

    Pretty tough, but it works great ! :)

    • 1