Argthtjtr

when Ansible playbook tries to connect to the (Tomcat server) VM machine which is in the same subnet, it gives the following error:

Step1: ssh command the one which ansible playbook uses:

ssh -C -vvv -o ControlMaster=auto -o ControlPersist=60s 

-o KbdInteractiveAuthentication=no -o PreferredAuthentications=publickey

-o PasswordAuthentication=no -o User=ec2-user -o ConnectTimeout=10

-o ControlPath=/var/lib/jenkins/.ansible/cp/ansible-ssh-%h-%p-%r -tt 172.31.1.128 '( umask 22 && mkdir -p "$( echo $HOME/.ansible/tmp/ansible-tmp-1528657894.57-17447076442650 )" && echo "$( echo $HOME/.ansible/tmp/ansible-tmp-1528657894.57-17447076442650 )" )'

Output of the SSH command with failure message:

OpenSSH_7.2p2 Ubuntu-4ubuntu2.4, OpenSSL 1.0.2g  1 Mar 2016

debug1: Reading configuration data /etc/ssh/ssh_config

debug1: /etc/ssh/ssh_config line 19: Applying options for *

debug1: auto-mux: Trying existing master

debug1: Control socket "/var/lib/jenkins/.ansible/cp/ansible-ssh-172.31.1.128-22-ec2-user" does not exist

debug2: resolving "172.31.1.128" port 22

debug2: ssh_connect_direct: needpriv 0

debug1: Connecting to 172.31.1.128 [172.31.1.128] port 22.

debug2: fd 3 setting O_NONBLOCK

debug1: fd 3 clearing O_NONBLOCK

debug1: Connection established.

debug3: timeout: 10000 ms remain after connect

debug1: permanently_set_uid: 0/0

debug1: key_load_public: No such file or directory

debug1: identity file /root/.ssh/server1.pem type -1

debug1: key_load_public: No such file or directory

debug1: identity file /root/.ssh/server1.pem-cert type -1

debug1: Enabling compatibility mode for protocol 2.0

debug1: Local version string SSH-2.0-OpenSSH_7.2p2 Ubuntu-4ubuntu2.4

debug1: Remote protocol version 2.0, remote software version OpenSSH_7.4

debug1: match: OpenSSH_7.4 pat OpenSSH* compat 0x04000000

debug2: fd 3 setting O_NONBLOCK

debug1: Authenticating to 172.31.1.128:22 as 'ec2-user'

debug3: send packet: type 20

debug1: SSH2_MSG_KEXINIT sent

debug3: receive packet: type 20

debug1: SSH2_MSG_KEXINIT received

debug2: local client KEXINIT proposal

debug2: KEX algorithms: curve25519-sha256@libssh.org,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group-exchange-sha1,diffie-hellman-group14-sha1,ext-info-c

debug2: host key algorithms: ecdsa-sha2-nistp256-cert-v01@openssh.com,ecdsa-sha2-nistp384-cert-v01@openssh.com,ecdsa-sha2-nistp521-cert-v01@openssh.com,ssh-ed25519-cert-v01@openssh.com,ssh-rsa-cert-v01@openssh.com,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521,ssh-ed25519,rsa-sha2-512,rsa-sha2-256,ssh-rsa

debug2: ciphers ctos: chacha20-poly1305@openssh.com,aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm@openssh.com,aes256-gcm@openssh.com,aes128-cbc,aes192-cbc,aes256-cbc,3des-cbc

debug2: ciphers stoc: chacha20-poly1305@openssh.com,aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm@openssh.com,aes256-gcm@openssh.com,aes128-cbc,aes192-cbc,aes256-cbc,3des-cbc

debug2: MACs ctos: umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1

debug2: MACs stoc: umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1

debug2: compression ctos: zlib@openssh.com,zlib,none

debug2: compression stoc: zlib@openssh.com,zlib,none

debug2: languages ctos:

debug2: languages stoc:

debug2: first_kex_follows 0

debug2: reserved 0

debug2: peer server KEXINIT proposal

debug2: KEX algorithms: curve25519-sha256,curve25519-sha256@libssh.org,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group16-sha512,diffie-hellman-group18-sha512,diffie-hellman-group-exchange-sha1,diffie-hellman-group14-sha256,diffie-hellman-group14-sha1,diffie-hellman-group1-sha1

debug2: host key algorithms: ssh-rsa,rsa-sha2-512,rsa-sha2-256,ecdsa-sha2-nistp256,ssh-ed25519

debug2: ciphers ctos: chacha20-poly1305@openssh.com,aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm@openssh.com,aes256-gcm@openssh.com,aes128-cbc,aes192-cbc,aes256-cbc,blowfish-cbc,cast128-cbc,3des-cbc

debug2: ciphers stoc: chacha20-poly1305@openssh.com,aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm@openssh.com,aes256-gcm@openssh.com,aes128-cbc,aes192-cbc,aes256-cbc,blowfish-cbc,cast128-cbc,3des-cbc

debug2: MACs ctos: umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1

debug2: MACs stoc: umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1

debug2: compression ctos: none,zlib@openssh.com

debug2: compression stoc: none,zlib@openssh.com

debug2: languages ctos:

debug2: languages stoc:

debug2: first_kex_follows 0

debug2: reserved 0

debug1: kex: algorithm: curve25519-sha256@libssh.org

debug1: kex: host key algorithm: ecdsa-sha2-nistp256

debug1: kex: server->client cipher: chacha20-poly1305@openssh.com MAC: <implicit> compression: zlib@openssh.com

debug1: kex: client->server cipher: chacha20-poly1305@openssh.com MAC: <implicit> compression: zlib@openssh.com

debug3: send packet: type 30

debug1: expecting SSH2_MSG_KEX_ECDH_REPLY

debug3: receive packet: type 31

debug1: Server host key: ecdsa-sha2-nistp256 SHA256:4/kIIUmDqoKA1xgTfdHsCGgIWIbWUQf+0BAEBMe0cJM

Warning: Permanently added '172.31.1.128' (ECDSA) to the list of known hosts.

debug3: send packet: type 21

debug2: set_newkeys: mode 1

debug1: rekey after 134217728 blocks

debug1: SSH2_MSG_NEWKEYS sent

debug1: expecting SSH2_MSG_NEWKEYS

debug3: receive packet: type 21

debug2: set_newkeys: mode 0

debug1: rekey after 134217728 blocks

debug1: SSH2_MSG_NEWKEYS received

debug2: key: /root/.ssh/server1.pem ((nil))

debug3: send packet: type 5

debug3: receive packet: type 7

debug1: SSH2_MSG_EXT_INFO received

debug1: kex_input_ext_info: server-sig-algs=<rsa-sha2-256,rsa-sha2-512>

debug3: receive packet: type 6

debug2: service_accept: ssh-userauth

debug1: SSH2_MSG_SERVICE_ACCEPT received

debug3: send packet: type 50

debug3: receive packet: type 51

debug1: Authentications that can continue: publickey,gssapi-keyex,gssapi-with-mic

debug3: start over, passed a different list publickey,gssapi-keyex,gssapi-with-mic

debug3: preferred publickey

debug3: authmethod_lookup publickey

debug3: remaining preferred:

debug3: authmethod_is_enabled publickey

debug1: Next authentication method: publickey

debug1: Trying private key: /root/.ssh/server1.pem

debug3: sign_and_send_pubkey: RSA SHA256:tbkErO+dWUV5xSGkdiFYf3Q9P2ZEgBHUr/dCUZMyANM

debug3: send packet: type 50

debug2: we sent a publickey packet, wait for reply

debug3: receive packet: type 52

debug1: Enabling compression at level 6.

debug1: Authentication succeeded (publickey).

Authenticated to 172.31.1.128 ([172.31.1.128]:22).

debug1: setting up multiplex master socket

debug3: muxserver_listen: temporary control path /var/lib/jenkins/.ansible/cp/ansible-ssh-172.31.1.128-22-ec2-user.7se8dQdicjV8JSGk

debug2: fd 4 setting O_NONBLOCK

debug3: fd 4 is O_NONBLOCK

debug3: fd 4 is O_NONBLOCK

debug1: channel 0: new [/var/lib/jenkins/.ansible/cp/ansible-ssh-172.31.1.128-22-ec2-user]

debug3: muxserver_listen: mux listener channel 0 fd 4

debug2: fd 3 setting TCP_NODELAY

debug3: ssh_packet_set_tos: set IP_TOS 0x08

debug1: control_persist_detach: backgrounding master process

debug2: control_persist_detach: background process is 3613

debug2: fd 4 setting O_NONBLOCK

debug1: forking to background

debug1: Entering interactive session.

debug1: pledge: id

debug2: set_control_persist_exit_time: schedule exit in 60 seconds

debug1: multiplexing control connection

debug2: fd 5 setting O_NONBLOCK

debug3: fd 5 is O_NONBLOCK

debug1: channel 1: new [mux-control]

debug3: channel_post_mux_listener: new mux channel 1 fd 5

debug3: mux_master_read_cb: channel 1: hello sent

debug2: set_control_persist_exit_time: cancel scheduled exit

debug3: mux_master_read_cb: channel 1 packet type 0x00000001 len 4

debug2: process_mux_master_hello: channel 1 slave version 4

debug2: mux_client_hello_exchange: master version 4

debug3: mux_client_forwards: request forwardings: 0 local, 0 remote

debug3: mux_client_request_session: entering

debug3: mux_client_request_alive: entering

debug3: mux_master_read_cb: channel 1 packet type 0x10000004 len 4

debug2: process_mux_alive_check: channel 1: alive check

debug3: mux_client_request_alive: done pid = 3615

debug3: mux_client_request_session: session request sent

debug3: mux_master_read_cb: channel 1 packet type 0x10000002 len 237

debug2: process_mux_new_session: channel 1: request tty 1, X 0, agent 0, subsys 0, term "xterm", cmd "( umask 22 && mkdir -p "$( echo $HOME/.ansible/tmp/ansible-tmp-1528657894.57-17447076442650 )" && echo "$( echo $HOME/.ansible/tmp/ansible-tmp-1528657894.57-17447076442650 )" )", env 1

debug3: process_mux_new_session: got fds stdin 6, stdout 7, stderr 8

debug1: channel 2: new [client-session]

debug2: process_mux_new_session: channel_new: 2 linked to control channel 1

debug2: channel 2: send open

debug3: send packet: type 90

debug3: receive packet: type 80

debug1: client_input_global_request: rtype hostkeys-00@openssh.com want_reply 0

debug3: receive packet: type 91

debug2: callback start

debug2: client_session2_setup: id 2

debug2: channel 2: request pty-req confirm 1

debug3: send packet: type 98

debug1: Sending environment.

debug1: Sending env LANG = en_US.UTF-8

debug2: channel 2: request env confirm 0

debug3: send packet: type 98

debug1: Sending command: ( umask 22 && mkdir -p "$( echo $HOME/.ansible/tmp/ansible-tmp-1528657894.57-17447076442650 )" && echo "$( echo $HOME/.ansible/tmp/ansible-tmp-1528657894.57-17447076442650 )" )

debug2: channel 2: request exec confirm 1

debug3: send packet: type 98

debug3: mux_session_confirm: sending success reply

debug2: callback done

debug2: channel 2: open confirm rwindow 0 rmax 32768

debug1: mux_client_request_session: master session id: 2

debug3: receive packet: type 99

debug2: channel_input_status_confirm: type 99 id 2

debug2: PTY allocation request accepted on channel 2

debug2: channel 2: rcvd adjust 2097152

debug3: receive packet: type 99

debug2: channel_input_status_confirm: type 99 id 2

debug2: exec request accepted on channel 2

debug3: receive packet: type 98

debug1: client_input_channel_req: channel 2 rtype exit-status reply 0

debug3: mux_exit_message: channel 2: exit message, exitval 0

debug3: receive packet: type 98

debug1: client_input_channel_req: channel 2 rtype eow@openssh.com reply 0

debug2: channel 2: rcvd eow

debug2: channel 2: close_read

debug2: channel 2: input open -> closed

debug3: receive packet: type 96

debug2: channel 2: rcvd eof

debug2: channel 2: output open -> drain

debug3: receive packet: type 97

debug2: channel 2: rcvd close

debug3: channel 2: will not send data after close

/home/ec2-user/.ansible/tmp/ansible-tmp-1528657894.57-17447076442650

debug3: channel 2: will not send data after close

debug2: channel 2: obuf empty

debug2: channel 2: close_write

debug2: channel 2: output drain -> closed

debug2: channel 2: send close

debug3: send packet: type 97

debug2: channel 2: is dead

debug2: channel 2: gc: notify user

debug3: mux_master_session_cleanup_cb: entering for channel 2

debug2: channel 1: rcvd close

debug2: channel 1: output open -> drain

debug2: channel 1: close_read

debug2: channel 1: input open -> closed

debug2: channel 2: gc: user detached

debug2: channel 2: is dead

debug2: channel 2: garbage collecting

debug1: channel 2: free: client-session, nchannels 3

debug3: channel 2: status: The following connections are open:

  #2 client-session (t4 r0 i3/0 o3/0 fd -1/-1 cc -1)



debug2: channel 1: obuf empty

debug2: channel 1: close_write

debug3: mux_client_read_packet: read header failed: Broken pipe

debug2: Received exit status from master 0

Shared connection to 172.31.1.128 closed.

root@ip-172-31-8-75:~/.ssh# debug2: channel 1: output drain -> closed

debug2: channel 1: is dead (local)

debug2: channel 1: gc: notify user

debug3: mux_master_control_cleanup_cb: entering for channel 1

debug2: channel 1: gc: user detached

debug2: channel 1: is dead (local)

debug2: channel 1: garbage collecting

debug1: channel 1: free: mux-control, nchannels 2

debug3: channel 1: status: The following connections are open:



debug2: set_control_persist_exit_time: schedule exit in 60 seconds

debug1: ControlPersist timeout expired

debug3: send packet: type 1

debug1: channel 0: free: /var/lib/jenkins/.ansible/cp/ansible-ssh-172.31.1.128-22-ec2-user, nchannels 1

debug3: channel 0: status: The following connections are open:



debug3: fd 0 is not O_NONBLOCK

debug3: fd 1 is not O_NONBLOCK

Transferred: sent 2572, received 2480 bytes, in 60.3 seconds

Bytes per second: sent 42.7, received 41.1

debug1: Exit status -1

debug1: compress outgoing: raw data 594, compressed 370, factor 0.62

debug1: compress incoming: raw data 370, compressed 594, factor 1.61

Could some one please help me in troubleshooting this and fixing the ssh to work.

Keys to connect to 172.31.1.128 are present in a file called server1.pem and is located at ~/.ssh/server1.pem location which is given to ansible.cfg with reference to
https://github.com/maheshkharwadkar/devops/tree/master/ansible/e2eproject

Step2: Although, a simple command:

ssh ec2-user@172.31.1.128 

works super cool.

However it fails through the playbook execution when above ssh command in step 1 is ran.

This might not be the exact answer you are looking for, but seeing a file transfer happenning, tells me Ansible is trying to install some of it's packages for remote adminstration, and it's not a connectivity issue.

Maybe try without the ControlPath options to see if the SSH multiplexing is the culprit or ensure that a single SSH connection is happening at the time?

I had a similar problem with Ansible, but my problem was that a python library was not available on my host, to allow Ansible remote administration. I ended up specifying to not gather facts on my playbook, then performing commands via the raw module.

- hosts: my_hosts

  remote_user: admin

  vars_files:

  - my_var_file.yml

  vars:

    ansible_ssh_private_key_file: ~/.ssh/mykey.pub

  gather_facts: no