I am trying to connect to a postgreSQL database (hosted on AWS RDS) via an SSH tunnel in R. So far, I have been able to connect using the following methods:
1.---------------------------
Opening the ssh tunnel in my terminal (MacOs) using
ssh -i {key file path} -f -N -L 5432:{db host}:5432 {ssh user}#{ssh host} -v
and then connecting to the database using
psql -hlocalhost -U{db user} -p5432 -dpostgres
2.---------------------------
Opening the ssh tunnel in my terminal and then running the following code in R to connect
conn <- dbConnect(
RPostgres::Postgres(),
dbname = db_name,
user = db_user,
password = db_password,
host = "127.0.0.1",
port = db_port
)
3.---------------------------
This is where the issue occurs. I'm able to connect by opening the ssh tunnel in R (in a background environment) with
tunnel_process <- callr::r_bg(
function(ssh_host, ssh_user, ssh_key, db_host, db_port) {
session <- ssh::ssh_connect(host = glue::glue("{ ssh_user }#{ ssh_host }"),
keyfile = ssh_key,
verbose = 3)
ssh::ssh_tunnel(session = session,
port = db_port,
target = glue::glue("{ db_host }:{ db_port }"))
},
args = list(ssh_host, ssh_user, ssh_key, db_host, db_port),
stdout = nullfile(),
stderr = nullfile()
)
But then I'm unable to use the same "dbConnect" code as above to connect. It only gives me the following error message
Error: could not connect to server: Connection refused
Is the server running on host "127.0.0.1" and accepting
TCP/IP connections on port 5432?
I am however able to connect directly from the terminal using the connection code in part 1. but only if I run psql -hlocalhost -U{db user} -p5432 -dpostgres, then re-run the ssh tunnel code in R, and only THEN enter my password in the terminal. I would appear that every time I try to connect, it closes the SSH tunnel, so I then have to re-launch it before submitting my password.
Question---------------------------
From what I just detailed, it would appear that:
a. My database is reachable since I can easily connect through the terminal
b. My R code works since I'm able to use it to both successfully open the SSH tunnel AND connect to the database. I'm just unable to use both together for some reason.
c. The tunnel I open through R breaks any time I try to connect to the database. This is not the case for the tunnel I open directly in the terminal.
Since I want to be able to do everything directly from R, does anybody here have any ideas on what may be causing the issue?
EDIT---------------------------
Here's the log I get in R when I try to connect to the database, just before it closes the tunnel:
> ssh::ssh_tunnel(session = session,
+ port = db_port,
+ target = glue::glue("{ db_host }:{ db_port }"))
\ Waiting for connetion on port 5432... client connected!
channel_open: Creating a channel 43 with 64000 window and 32768 max packet
ssh_socket_unbuffered_write: Enabling POLLOUT for socket
packet_send2: packet: wrote [len=124,padding=18,comp=105,payload=105]
channel_open: Sent a SSH_MSG_CHANNEL_OPEN type direct-tcpip for channel 43
ssh_packet_socket_callback: packet: read type 80 [len=492,padding=16,comp=475,payload=475]
ssh_packet_process: Dispatching handler for packet type 80
ssh_packet_global_request: Received SSH_MSG_GLOBAL_REQUEST packet
ssh_packet_global_request: UNKNOWN SSH_MSG_GLOBAL_REQUEST hostkeys-00#openssh.com 0
ssh_packet_process: Couldn't do anything with packet type 80
packet_send2: packet: wrote [len=12,padding=6,comp=5,payload=5]
ssh_socket_unbuffered_write: Enabling POLLOUT for socket
ssh_packet_socket_callback: packet: read type 91 [len=28,padding=10,comp=17,payload=17]
ssh_packet_process: Dispatching handler for packet type 91
ssh_packet_channel_open_conf: Received SSH2_MSG_CHANNEL_OPEN_CONFIRMATION
ssh_packet_channel_open_conf: Received a CHANNEL_OPEN_CONFIRMATION for channel 43:0
ssh_packet_channel_open_conf: Remote window : 2097152, maxpacket : 32768
| Tunneled -1 bytes...ssh_socket_unbuffered_write: Enabling POLLOUT for socket
packet_send2: packet: wrote [len=28,padding=10,comp=17,payload=17]
channel_write_common: channel_write wrote 8 bytes
| Tunneled 7 bytes...ssh_packet_socket_callback: packet: read type 94 [len=28,padding=17,comp=10,payload=10]
ssh_packet_process: Dispatching handler for packet type 94
channel_rcv_data: Channel receiving 1 bytes data in 0 (local win=64000 remote win=2097144)
channel_default_bufferize: placing 1 bytes into channel buffer (stderr=0)
channel_rcv_data: Channel windows are now (local win=63999 remote win=2097144)
ssh_socket_unbuffered_write: Enabling POLLOUT for socket
packet_send2: packet: wrote [len=28,padding=18,comp=9,payload=9]
grow_window: growing window (channel 43:0) to 1280000 bytes
ssh_channel_read_timeout: Read (1) buffered : 1 bytes. Window: 1280000
- Tunneled 8 bytes...ssh_socket_unbuffered_write: Enabling POLLOUT for socket
packet_send2: packet: wrote [len=316,padding=17,comp=298,payload=298]
channel_write_common: channel_write wrote 289 bytes
/ Tunneled 297 bytes...ssh_packet_socket_callback: packet: read type 94 [len=3964,padding=12,comp=3951,payload=3951]
ssh_packet_process: Dispatching handler for packet type 94
channel_rcv_data: Channel receiving 3942 bytes data in 0 (local win=1280000 remote win=2096855)
channel_default_bufferize: placing 3942 bytes into channel buffer (stderr=0)
channel_rcv_data: Channel windows are now (local win=1276058 remote win=2096855)
ssh_channel_read_timeout: Read (3942) buffered : 3942 bytes. Window: 1276058
\ Tunneled 4239 bytes...ssh_socket_unbuffered_write: Enabling POLLOUT for socket
packet_send2: packet: wrote [len=156,padding=8,comp=147,payload=147]
channel_write_common: channel_write wrote 138 bytes
- Tunneled 4377 bytes...ssh_packet_socket_callback: packet: read type 94 [len=76,padding=15,comp=60,payload=60]
ssh_packet_process: Dispatching handler for packet type 94
channel_rcv_data: Channel receiving 51 bytes data in 0 (local win=1276058 remote win=2096717)
channel_default_bufferize: placing 51 bytes into channel buffer (stderr=0)
channel_rcv_data: Channel windows are now (local win=1276007 remote win=2096717)
ssh_channel_read_timeout: Read (51) buffered : 51 bytes. Window: 1276007
| Tunneled 4428 bytes...ssh_socket_unbuffered_write: Enabling POLLOUT for socket
packet_send2: packet: wrote [len=140,padding=14,comp=125,payload=125]
channel_write_common: channel_write wrote 116 bytes
\ Tunneled 4544 bytes...ssh_packet_socket_callback: packet: read type 94 [len=60,padding=8,comp=51,payload=51]
ssh_packet_process: Dispatching handler for packet type 94
channel_rcv_data: Channel receiving 42 bytes data in 0 (local win=1276007 remote win=2096601)
channel_default_bufferize: placing 42 bytes into channel buffer (stderr=0)
channel_rcv_data: Channel windows are now (local win=1275965 remote win=2096601)
ssh_channel_read_timeout: Read (42) buffered : 42 bytes. Window: 1275965
/ Tunneled 4586 bytes...ssh_socket_unbuffered_write: Enabling POLLOUT for socket
packet_send2: packet: wrote [len=60,padding=19,comp=40,payload=40]
channel_write_common: channel_write wrote 31 bytes
- Tunneled 4617 bytes...packet_send2: packet: wrote [len=12,padding=6,comp=5,payload=5]
ssh_channel_send_eof: Sent a EOF on client channel (43:0)
ssh_socket_unbuffered_write: Enabling POLLOUT for socket
packet_send2: packet: wrote [len=12,padding=6,comp=5,payload=5]
ssh_channel_close: Sent a close on client channel (43:0)
ssh_socket_unbuffered_write: Enabling POLLOUT for socket
tunnel closed!
For reference, this is what the same log looks like when using the workaround detailed in 3. (re-running the ssh_tunnel right before submitting my password in the terminal):
> ssh::ssh_tunnel(session = session,
+ port = db_port,
+ target = glue::glue("{ db_host }:{ db_port }"))
\ Waiting for connetion on port 5432... client connected!
channel_open: Creating a channel 43 with 64000 window and 32768 max packet
ssh_socket_unbuffered_write: Enabling POLLOUT for socket
packet_send2: packet: wrote [len=124,padding=18,comp=105,payload=105]
channel_open: Sent a SSH_MSG_CHANNEL_OPEN type direct-tcpip for channel 43
ssh_packet_socket_callback: packet: read type 80 [len=492,padding=16,comp=475,payload=475]
ssh_packet_process: Dispatching handler for packet type 80
ssh_packet_global_request: Received SSH_MSG_GLOBAL_REQUEST packet
ssh_packet_global_request: UNKNOWN SSH_MSG_GLOBAL_REQUEST hostkeys-00#openssh.com 0
ssh_packet_process: Couldn't do anything with packet type 80
packet_send2: packet: wrote [len=12,padding=6,comp=5,payload=5]
ssh_socket_unbuffered_write: Enabling POLLOUT for socket
ssh_packet_socket_callback: packet: read type 91 [len=28,padding=10,comp=17,payload=17]
ssh_packet_process: Dispatching handler for packet type 91
ssh_packet_channel_open_conf: Received SSH2_MSG_CHANNEL_OPEN_CONFIRMATION
ssh_packet_channel_open_conf: Received a CHANNEL_OPEN_CONFIRMATION for channel 43:0
ssh_packet_channel_open_conf: Remote window : 2097152, maxpacket : 32768
| Tunneled -1 bytes...ssh_socket_unbuffered_write: Enabling POLLOUT for socket
packet_send2: packet: wrote [len=28,padding=10,comp=17,payload=17]
channel_write_common: channel_write wrote 8 bytes
| Tunneled 7 bytes...ssh_packet_socket_callback: packet: read type 94 [len=28,padding=17,comp=10,payload=10]
ssh_packet_process: Dispatching handler for packet type 94
channel_rcv_data: Channel receiving 1 bytes data in 0 (local win=64000 remote win=2097144)
channel_default_bufferize: placing 1 bytes into channel buffer (stderr=0)
channel_rcv_data: Channel windows are now (local win=63999 remote win=2097144)
ssh_socket_unbuffered_write: Enabling POLLOUT for socket
packet_send2: packet: wrote [len=28,padding=18,comp=9,payload=9]
grow_window: growing window (channel 43:0) to 1280000 bytes
ssh_channel_read_timeout: Read (1) buffered : 1 bytes. Window: 1280000
- Tunneled 8 bytes...ssh_socket_unbuffered_write: Enabling POLLOUT for socket
packet_send2: packet: wrote [len=316,padding=17,comp=298,payload=298]
channel_write_common: channel_write wrote 289 bytes
\ Tunneled 297 bytes...ssh_packet_socket_callback: packet: read type 94 [len=3964,padding=12,comp=3951,payload=3951]
ssh_packet_process: Dispatching handler for packet type 94
channel_rcv_data: Channel receiving 3942 bytes data in 0 (local win=1280000 remote win=2096855)
channel_default_bufferize: placing 3942 bytes into channel buffer (stderr=0)
channel_rcv_data: Channel windows are now (local win=1276058 remote win=2096855)
ssh_channel_read_timeout: Read (3942) buffered : 3942 bytes. Window: 1276058
/ Tunneled 4239 bytes...ssh_socket_unbuffered_write: Enabling POLLOUT for socket
packet_send2: packet: wrote [len=156,padding=8,comp=147,payload=147]
channel_write_common: channel_write wrote 138 bytes
| Tunneled 4377 bytes...ssh_packet_socket_callback: packet: read type 94 [len=76,padding=15,comp=60,payload=60]
ssh_packet_process: Dispatching handler for packet type 94
channel_rcv_data: Channel receiving 51 bytes data in 0 (local win=1276058 remote win=2096717)
channel_default_bufferize: placing 51 bytes into channel buffer (stderr=0)
channel_rcv_data: Channel windows are now (local win=1276007 remote win=2096717)
ssh_channel_read_timeout: Read (51) buffered : 51 bytes. Window: 1276007
- Tunneled 4428 bytes...ssh_socket_unbuffered_write: Enabling POLLOUT for socket
packet_send2: packet: wrote [len=140,padding=14,comp=125,payload=125]
channel_write_common: channel_write wrote 116 bytes
/ Tunneled 4544 bytes...ssh_packet_socket_callback: packet: read type 94 [len=60,padding=8,comp=51,payload=51]
ssh_packet_process: Dispatching handler for packet type 94
channel_rcv_data: Channel receiving 42 bytes data in 0 (local win=1276007 remote win=2096601)
channel_default_bufferize: placing 42 bytes into channel buffer (stderr=0)
channel_rcv_data: Channel windows are now (local win=1275965 remote win=2096601)
ssh_channel_read_timeout: Read (42) buffered : 42 bytes. Window: 1275965
\ Tunneled 4586 bytes...ssh_socket_unbuffered_write: Enabling POLLOUT for socket
packet_send2: packet: wrote [len=92,padding=12,comp=79,payload=79]
channel_write_common: channel_write wrote 70 bytes
- Tunneled 4656 bytes...ssh_packet_socket_callback: packet: read type 94 [len=380,padding=15,comp=364,payload=364]
ssh_packet_process: Dispatching handler for packet type 94
channel_rcv_data: Channel receiving 355 bytes data in 0 (local win=1275965 remote win=2096531)
channel_default_bufferize: placing 355 bytes into channel buffer (stderr=0)
channel_rcv_data: Channel windows are now (local win=1275610 remote win=2096531)
ssh_channel_read_timeout: Read (355) buffered : 355 bytes. Window: 1275610
| Tunneled 5011 bytes...
Finally, here's the log when running ssh_connect:
> session <- ssh::ssh_connect(host = glue::glue("{ ssh_user }#{ ssh_host }"),
+ keyfile = ssh_key,
+ verbose = 3)
ssh_pki_import_privkey_base64: Trying to decode privkey passphrase=false
ssh_connect: libssh 0.8.6 (c) 2003-2018 Aris Adamantiadis, Andreas Schneider and libssh contributors. Distributed under the LGPL, please refer to COPYING file for information about your rights, using threading threads_pthread
ssh_socket_connect: Nonblocking connection socket: 50
ssh_connect: Socket connecting, now waiting for the callbacks to work
ssh_connect: Actual timeout : 10000
ssh_socket_pollcallback: Received POLLOUT in connecting state
socket_callback_connected: Socket connection callback: 1 (0)
ssh_socket_unbuffered_write: Enabling POLLOUT for socket
callback_receive_banner: Received banner: SSH-2.0-OpenSSH_7.4
ssh_client_connection_callback: SSH server banner: SSH-2.0-OpenSSH_7.4
ssh_analyze_banner: Analyzing banner: SSH-2.0-OpenSSH_7.4
ssh_analyze_banner: We are talking to an OpenSSH client version: 7.4 (70400)
ssh_known_hosts_read_entries: Failed to open the known_hosts file '/etc/ssh/ssh_known_hosts': No such file or directory
ssh_client_select_hostkeys: Changing host key method to "ecdsa-sha2-nistp256,ssh-ed25519,ecdsa-sha2-nistp521,ecdsa-sha2-nistp384,ssh-rsa,ssh-dss"
ssh_socket_unbuffered_write: Enabling POLLOUT for socket
packet_send2: packet: wrote [len=644,padding=9,comp=634,payload=634]
ssh_packet_socket_callback: packet: read type 20 [len=1276,padding=10,comp=1265,payload=1265]
ssh_packet_process: Dispatching handler for packet type 20
ssh_kex_select_methods: Negotiated curve25519-sha256,ecdsa-sha2-nistp256,aes256-ctr,aes256-ctr,hmac-sha2-256,hmac-sha2-256,none,none,,
ssh_socket_unbuffered_write: Enabling POLLOUT for socket
packet_send2: packet: wrote [len=44,padding=6,comp=37,payload=37]
ssh_packet_socket_callback: packet: read type 31 [len=260,padding=11,comp=248,payload=248]
ssh_packet_process: Dispatching handler for packet type 31
ssh_packet_dh_reply: Received SSH_KEXDH_REPLY
ssh_socket_unbuffered_write: Enabling POLLOUT for socket
packet_send2: packet: wrote [len=12,padding=10,comp=1,payload=1]
ssh_client_curve25519_reply: SSH_MSG_NEWKEYS sent
ssh_packet_socket_callback: Processing 112 bytes left in socket buffer
ssh_packet_socket_callback: packet: read type 21 [len=12,padding=10,comp=1,payload=1]
ssh_packet_process: Dispatching handler for packet type 21
ssh_packet_newkeys: Received SSH_MSG_NEWKEYS
crypt_set_algorithms2: Set output algorithm to aes256-ctr
crypt_set_algorithms2: Set HMAC output algorithm to hmac-sha2-256
crypt_set_algorithms2: Set input algorithm to aes256-ctr
crypt_set_algorithms2: Set HMAC input algorithm to hmac-sha2-256
ssh_packet_newkeys: Signature verified and valid
ssh_packet_socket_callback: Processing 96 bytes left in socket buffer
ssh_packet_socket_callback: packet: read type 7 [len=60,padding=6,comp=53,payload=53]
ssh_packet_process: Dispatching handler for packet type 7
ssh_packet_ext_info: Received SSH_MSG_EXT_INFO
ssh_packet_ext_info: Follows 1 extensions
ssh_packet_ext_info: Extension: server-sig-algs=<rsa-sha2-256,rsa-sha2-512>
ssh_connect: current state : 7
packet_send2: packet: wrote [len=28,padding=10,comp=17,payload=17]
ssh_service_request: Sent SSH_MSG_SERVICE_REQUEST (service ssh-userauth)
ssh_socket_unbuffered_write: Enabling POLLOUT for socket
ssh_packet_socket_callback: packet: read type 6 [len=28,padding=10,comp=17,payload=17]
ssh_packet_process: Dispatching handler for packet type 6
ssh_packet_service_accept: Received SSH_MSG_SERVICE_ACCEPT
ssh_socket_unbuffered_write: Enabling POLLOUT for socket
packet_send2: packet: wrote [len=44,padding=4,comp=39,payload=39]
ssh_packet_socket_callback: packet: read type 51 [len=60,padding=15,comp=44,payload=44]
ssh_packet_process: Dispatching handler for packet type 51
ssh_packet_userauth_failure: Access denied for 'none'. Authentication that can continue: publickey,gssapi-keyex,gssapi-with-mic
ssh_packet_userauth_failure: Access denied for 'none'. Authentication that can continue: publickey,gssapi-keyex,gssapi-with-mic
ssh_key_algorithm_allowed: Checking rsa-sha2-512 with list <ssh-ed25519,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521,ssh-rsa,rsa-sha2-512,rsa-sha2-256,ssh-dss>
ssh_key_algorithm_allowed: Checking rsa-sha2-512 with list <ssh-ed25519,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521,ssh-rsa,rsa-sha2-512,rsa-sha2-256,ssh-dss>
ssh_key_algorithm_allowed: Checking rsa-sha2-512 with list <ssh-ed25519,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521,ssh-rsa,rsa-sha2-512,rsa-sha2-256,ssh-dss>
ssh_socket_unbuffered_write: Enabling POLLOUT for socket
packet_send2: packet: wrote [len=636,padding=11,comp=624,payload=624]
ssh_packet_socket_callback: packet: read type 52 [len=12,padding=10,comp=1,payload=1]
ssh_packet_process: Dispatching handler for packet type 52
ssh_packet_userauth_success: Authentication successful
I have 4 NodeMCU boards, 3 of them I can use and 1 I can't seem to upload any of my code to. It does boot, and I can get a readout of it on the serial port.
On boot:
ets Jan 8 2013,rst cause:2, boot mode:(3,6)
load 0x40100000, len 1856, room 16
tail 0
chksum 0x63
load 0x3ffe8000, len 776, room 8
tail 0
chksum 0x02
load 0x3ffe8310, len 552, room 8
tail 0
chksum 0x79
csum 0x79
2nd boot version : 1.5
SPI Speed : 40MHz
SPI Mode : DIO
SPI Flash Size & Map: 8Mbit(512KB+512KB)
jump to run user1 # 1000
rf[112] : 03
rf[113] : 00
rf[114] : 01
SDK ver: 1.5.4(baaeaebb) compiled # May 17 2016 19:23:54
phy ver: 972, pp ver: 10.1
I also tried doing a reset with the flash button held down:
ets Jan 8 2013,rst cause:2, boot mode:(1,6)
I don't get any compile errors with my code, and the code works perfectly fine on the other 3 NodeMCU boards. When I try to upload (regardless of boot mode) I see the blue light near the ESP8266 chip blink a couple of times, and then I get the following errors output:
Sketch uses 230061 bytes (22%) of program storage space. Maximum is 1044464 bytes.
Global variables use 34612 bytes (42%) of dynamic memory, leaving 47308 bytes for local variables. Maximum is 81920 bytes.
warning: espcomm_sync failed
error: espcomm_open failed
error: espcomm_upload_mem failed
error: espcomm_upload_mem failed
I've also tried grounding D0 and connecting the EN pin to 3.3v, which changes the boot mode to ets Jan 8 2013,rst cause:2, boot mode:(1,7) and the red onboard LED turns on, but still doesn't allow me to upload code.
ENVIRONMENT (these settings work on all my other NodeMCU Boards and I did confirm this is a 1.0 board)
IDE: Arduino 1.8.0
Board: NodeMCU 1.0 (ESP-12E)
CPU: 80 MHz
Flash Size: 4M (3M SPIFFS)
Upload Speed: 115200