prep-node: Encountered an error while preparing the provided nodes as Kubernetes nodes. Code: 2

[svanschalkwyk]

Python 3.6.
Error seems to be print which may have to be print() as per Py3.

"print json.load(sys.stdin)"

Trying fix below:
# check for flags (optional parameters)
if [ $# -eq 4 -a "${4}" == "k8s" ]; then
  role_filter="python -c 'import sys, json; print (json.load(sys.stdin)[\"extensions\"][\"ip_address\"][\"status\"])'"
else
  role_filter="python -c 'import sys, json; print( json.load(sys.stdin)[\"role_status\"]')"
fi

$ pf9ctl cluster prep-node -u steph -p @xxxxx -i localhost -i 192.168.6.131
Preparing nodes  [####################################]  100%          
Encountered an error while preparing the provided nodes as Kubernetes nodes. Code: 2, output log: /home/steph/pf9/log/node_provision_2020_09_04-17_58_12.log

Failed Ansible task:

  File \"<string>\", line 1", "    import sys, json; print json.load(sys.stdin)[\"extensions\"][\"ip_address\"][\"status\"]", "                               ^", "SyntaxError: invalid syntax", "(23) Failed writing body", "  File \"<string>\", line 1", "    import sys, json; print json.load(sys.stdin)[\"extensions\"][\"ip_address\"][\"status\"]", "                               ^", "SyntaxError: invalid syntax", "(23) Failed writing body", "  File \"<string>\", line 1", "    import sys, json; print json.load(sys.stdin)[\"extensions\"][\"ip_address\"][\"status\"]", "                               ^", "SyntaxError: invalid syntax", "(23) Failed writing body", "  File \"<string>\", line 1", "    import sys, json; print json.load(sys.stdin)[\"extensions\"][\"ip_address\"][\"status\"]", "                               ^", "SyntaxError: invalid syntax", "(23) Failed writing body"], "stdout": "[ waiting for pf9-hostagent to complete convergence ]\n--> TIMEOUT = 1200 seconds\n--> flag_k8s=0\nASSERT: *** TIMEOUT EXCEEDED ***\n", "stdout_lines": ["[ waiting for pf9-hostagent to complete convergence ]", "--> TIMEOUT = 1200 seconds", "--> flag_k8s=0", "ASSERT: *** TIMEOUT EXCEEDED ***"]}
fatal: [192.168.6.131]: FAILED! => {"changed": false, "connection": "close", "content": "{\"message\":\"RoleUpdateConflict: Role cannot be updated in the current state: Cannot add role pf9-kube to host 247f5db1-81a6-41cf-9f12-4309ad7cba89 in the current state.\"}", "content_length": "170", "content_type": "application/json; charset=utf-8", "date": "Fri, 04 Sep 2020 18:18:41 GMT", "elapsed": 0, "json": {"message": "RoleUpdateConflict: Role cannot be updated in the current state: Cannot add role pf9-kube to host 247f5db1-81a6-41cf-9f12-4309ad7cba89 in the current state."}, "msg": "Status code was 409 and not [200]: HTTP Error 409: Conflict", "redirected": false, "server": "nginx/1.15.3", "status": 409, "strict_transport_security": "max-age=15724800; includeSubDomains", "url": "https://pmkft-1598358826-11591.platform9.io/resmgr/v1/hosts/247f5db1-81a6-41cf-9f12-4309ad7cba89/roles/pf9-kube"}
192.168.6.131              : ok=25   changed=7    unreachable=0    failed=1    skipped=20   rescued=0    ignored=0   
localhost                  : ok=21   changed=3    unreachable=0    failed=1    skipped=20   rescued=0    ignored=0   

[mattroark]

The offending lines are:

• express-cli/pf9/express/roles/wait-for-convergence/files/wait_for_agent_convergence.sh

  Line 28 in 2e3ce05

  role_filter="python -c 'import sys, json; print json.load(sys.stdin)[\"extensions\"][\"ip_address\"][\"status\"]'"

• express-cli/pf9/express/roles/wait-for-convergence/files/wait_for_agent_convergence.sh

  Line 30 in 2e3ce05

  role_filter="python -c 'import sys, json; print json.load(sys.stdin)[\"role_status\"]'"

Indeed, I do think adding parentheses would solve this conflict and allow it to be backwards compatible as per PEP-3105.

[mattroark]

A similar issue was reported on our Community Forum: https://community.platform9.com/t/not-being-able-to-prep-node-from-bareos/71/4

As noted there, I do not believe this should impact the actual host convergence itself nor the operation of the monitoring script overall. The timeout appears to be asserted regardless, which would normally indicate that the host has failed to or is stuck converging.

In this situation, it would likely be helpful to know the output of /var/log/pf9/hostagent.log on the node attempting to be added in order to understand if this is the case or not. If it is failing, it may be for a reason which we'll want to handle separately.