I'm trying to connect to a database that is located in an external server using dplyr's
src_postgres(dbname = NULL, host = NULL, port = NULL, user = NULL,
password = NULL, ...)
So far so good, I've got all the parameters I need to connect to the database. The problem is that the server where the database is located requires an authentication too (username and password).
I tried creating a connection with ?pipe but seems like it only works when trying to extract files from a remote server.
Any clues?
Good news! I do this all the time and it's not hard :)
Two steps:
1. Create SSH key and put on remote server
from https://serverfault.com/posts/241593/edit
Generate ssh keys on your local machine:
$ ssh-keygen -t rsa -b 2048
And press Enter for empty passphrase to result in:
Generating public/private rsa key pair.
Enter file in which to save the key (/home/username/.ssh/id_rsa):
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in /home/username/.ssh/id_rsa.
Your public key has been saved in /home/username/.ssh/id_rsa.pub.
Copy your keys to the target server
ssh-copy-id id#server
Check that this worked with ssh 'id#server', and check folder .ssh/authorized_keys for the ssh keys.
You should know be able to log in with $ ssh id#server
2. Forward your database port to your local machine
You should know be able to use
ssh -fN id#server
to initiative an SSH connection and forward ports on to your local host.
You may need to adjust the -p parameter to ssh to select the correct port.
Once you can successfully forward the port, you should be able to use src_postres() from your local machine to access the remote database.
You can also start your R script with
system("ssh -fN id#server")
or put the command in your .Rprofile
3. (optional)
Also, maybe you don't want your id and server address in your scripts, say, if you were going to give them to a client, or put them on github.
Then, edit or create file (on local machine) .ssh/config with the following content:
Host my_ssh
User id
Hostname server
and then you can just use ssh -fN my_ssh
the best way to do this is by connecting via DBI and then using the open connection with dplyr. For example:
library(DBI)
con <- dbConnect(RPostgres::Postgres())
db_table <- tbl(con, "my_table")
db_table %>%
group_by(one_var) %>%
tally()
Related
I have created the batch to transfer the file using SSH keys, I checked the public and private key mapping on both the servers and it's working fine.
My Windows batch code using SFTP command is as follows:
open sftp://sftp_user#ssh_dest_server -privatekey="D:\directory_full_path\private_key.ppk" -rawsettings TryAgent=0 AuthKI=0 AuthGSSAPI=0
CD "/XYZ_Directory/folder1"
Put "\\full_directory_path\FILE1.zip"
exit
When I execute the batch manually it's executing fine without any issue, but when I execute batch from SQL Job (using different user) then it's shows below error:
Searching for host...
Connecting to host...
Authenticating...
Continue connecting to an unknown server and add its host key to a cache?
The server's host key was not found in the cache. You have no guarantee that the server is the computer you think it is.
The server's RSA key details are:
Algorithm: ssh-rsa 2048
SHA-256: finger_print_key
MD5: zz:xx:yy:xx:yy:xx:yy:xx:yy:xx:yy:xx:yy:zz:zz:00
If you trust this host,
press Yes. To connect without adding host key to the cache,
press No. To abandon the connection press Cancel.
In scripting, you should use a -hostkey switch to configure the expected host key.
(Y)es, (N)o, C(a)ncel (10 s), (C)opy Key, (P)aste key: Cancel
Host key wasn't verified!
Host key fingerprint is ssh-rsa 2048 finger_print_key.
Authentication failed.
I already tried -hostkey WinSCP command but says "unknown command". Suggestions are most welcome.
Something I want to do like this link "WinSCP" through WinSCP command but inside my Windows batch automatically to verify the host.
To verify a host key in WinSCP script, add -hostkey switch to the open command:
open sftp://sftp_user#ssh_dest_server -hostkey=... -privatekey="D:\directory_full_path\private_key.ppk" -rawsettings TryAgent=0 AuthKI=0 AuthGSSAPI=0
See Verifying the host key ... in script in WinSCP documentation. It covers everything you need to know. In particular, where to get the host key value.
Also note that WinSCP GUI can generate a script template even with the -hostkey switch for you.
Also covered in My script works fine when executed manually, but fails or hangs when run by Windows Scheduler, SSIS or other automation service. What am I doing wrong?
I would like to use RStudio for analysis of data on a MySQL instance. This is a AWS RDS MySql instance that is only accessible via a jump box / bastion host. I have the credentials necessary to connect to the jump box, and from the jump box to the RDS instance. What do I need to do be able to query this DB directly from within the RStudio console?
I can connect (using the Terminal tab in RStudio)to the jump box using:
ssh -p 22xx user#ip.add.re.ss
Then I can connect to RDS mysql using:
mysql -u username -p database -h hostname.us-east-1.rds.amazonaws.com
I can connect and do manual mysql commands from within RStudio terminal, but I don't seem to be able to do anything with the DB from the RStudio console.
Sorry for opening a 2yo thread, but for everyone dealing with this issue as I am - I found this thread and it looks like it works (connecting with MySQL via ssh from R Studio).
You should use something which is called port forwarding. Some details
are here
(https://help.ubuntu.com/community/SSH/OpenSSH/PortForwarding) For
example, say you wanted to connect from your laptop to
http://www.ubuntuforums.org using an SSH tunnel. You would use source
port number 8080 (the alternate http port), destination port 80 (the
http port), and destination server www.ubuntuforums.org. :
ssh -L 8080:www.ubuntuforums.org:80 Where should be
replaced by the name of your laptop.
This is done for whole computer so you dont need to do this from r
studio.
Offcourse you need to forward your port to 3036. But you need special
privilige on the server. Because on most hosting you can only connect
from localhost (for example from PHP)
Source: https://www.py4u.net/discuss/881859
I am some issue with connecting the pipes between MySQL Workbench and my Bitnami Wordpress instance on AWS (using ssh). I have read and attempted the docs several times, I don't know what is going on.
I have successfully ssh-ed via the command line using ssh -i my_key_file bitnami#IP-Address.
However, I cannot get it to work for MySQL Workbench.
Field Value
SSH Hostname IP-Address
SSH Username bitnami
SSH Key File <path-to-key-file>
MySQL Hostname 127.0.0.1
MySQL Server Port 3306
MySQL username root
The path to key-file is correct as it works for other AWS instances. And the SSH hostname and SSH username get my access via the terminal - so I would assume these are fine?
When I test the connection, I am prompted for a password. Everywhere online suggests that this password is the admin password used the Wordpress account (accessible via the system logs). I have this password. However, this doesn't get me in.
The attempted connection returns the error: Failed to Connect to MySQL at 127.0.0.1:3306 through SSH tunnel at bitnami#IP-Address with user root
Any help would be appreciated
I have read and attempted the docs several times, I don't know what is going on.
I guess you are referring to this documentation
Could you try to reset the password for the root user as described in the guide below?
https://docs.bitnami.com/aws/components/mysql/#how-to-reset-the-mysql-root-password
Once you have restarted it and you have chosen a new one, use this new password on the Workbench.
I'm looking for a way that will allow me to use an SSH Tunnel to connect to a MySQL Server (as opposed to a file) within R; I'm assuming it'll require a combination of RCurl and RODBC, but I can't seem to get it to work properly.
I came across this post and this post that talk about utilizing SSH to connect to specific files or tables, but I'm hoping to use it as part of a Shiny app that will execute different SQL queries based on input from the user, which would require connecting into the server as opposed to specific files.
I'm assuming the code would look something along these lines x = scp("remote.ssh.host.com", "/home/dir/file.txt", "My.SCP.Passphrase", user="username"), but would I replace the "/home/dir/file.txt" piece with an odbcConnect() statement or replace it with the port number for the specific database I want to access?
Edit: The line I use for a regular odbcConnect() is odbcConnect(dsn, uid = "userid", pwd = "password"). Part of the problem is, I am developing it on Windows, but it will be deployed to a Linux server (handled be someone else) so I'm struggling to figure out what exactly will need to be used in my server.R code for connecting to the database.
Okay, so to test this on Windows, either grab Cygwin, or install OpenSSH so you can run ssh from the command line in Windows, like you would do in Linux.
Once you have ssh running on your Windows box, then try first making a tunnel through SSH. Run this from the command line:
ssh -f <server_user>#<server_ip> -L <unused_local_port>:localhost:<database_remote_port> -N
Obviously, replace everything in '<>' with the appropriate information. It will ask for the password, and remember that this isn't the database password, but the password to the server itself. Notably, the server_ip doesn't have to be the server with the database on it, just any server that is inside the proper subnet and that runs an SSH server, which is pretty much all Linux machines.
Now, setup an ODBC connection, except make the IP localhost, and the port unused_local_port. Now, try connecting to your new ODBC connection in R. If this works you're halfway there.
The next problem is the password, because you will have to enter a password to connect via SSH, but in R you won't be able to input it after a simple system command. So you have to setup some a public/private rsa key pair. Notably, this will make it so that anyone with access to your user/pass on your Windows box will now have automatic access to your server, so be careful. First, generate a SSH key:
ssh-keygen -t rsa
Don't make a passphrase, and save it in the default location. Now, create the directory for your public key on the remote host, and drop your public key in there.
# This creates a directory on the other machine if it wasn't already there. (Type in your password on the remote machine)
ssh <server_user>#<server_ip> mkdir -p .ssh
# This adds your public key to the list of accepted ones:
cat ~/.ssh/id_rsa.pub | ssh <server_user>#<server_ip> 'cat >> .ssh/authorized_keys'
Now try creating your tunnel again from the command line:
ssh -f <server_user>#<server_ip> -L <unused_local_port>:localhost:<database_remote_port> -N
If it doesn't ask you for the password, you have succeeded in creating your keypair. Now you are ready to run your ssh command from the command line. But before you do that, try and kill your ssh command, so you can make sure that R is actually creating the tunnel, and you aren't just reusing an old one. You can do it through Windows Task Manager (Ctrl+Alt+Esc), and just right click and End Process the ssh.exe.
So, just run:
system('ssh -f <server_user>#<server_ip> -L <unused_local_port>:localhost:<database_remote_port> -N')
And then connect to your new tunneled ODBC connection.
I got the connection details of a SFTP server, connected to it with FileZilla, and then successfully downloaded a file from that SFTP.
The only details I had was host, port, user and pass.
Now I'm trying to connect to this same server trough WinSCP .NET assembly (C#)
using(Session session = new WinSCP.Session()) {
session.Open(new SessionOptions() {
Protocol = Protocol.,
HostName = "ftp.*********.be",
UserName ="*****",
Password ="*****"
});
TransferOptions transferOptions = new TransferOptions();
transferOptions.TransferMode = WinSCP.TransferMode.Binary;
TransferOperationResult transferResult;
transferResult = session.GetFiles("/downld/fileonserver.dbf",#"c:\testfolder\localfilename.dbf", false, transferOptions);
Whatever I try here it keeps asking for a key for SSH, but I don't have that key, I generated a 128 bit RSA key somewhere online and put it in the session options like:
SshHostKeyFingerprint = "ssh-rsa 1024 82:09:12:b4:93:92:3a:61:10:90:61:12:b4:XX:XX:XX"
But this just tells me that key is invalid.
I kind of figured out that I maybe need the public/private SSH key from the server to get this to work but I sadly don't have access to this server.
Since FileZilla can connect to it without me entering any KEYS, why can't my C# program do the same?
I'm not an expert when it comes to security related stuff, so please point me in the right direction. I found this thread but I don't have access to .ssh folder on the FTP server and I don't really get where they are going with this.
You are confusing the SSH server public host key verification with the client public key authentication. These are two completely different things. This first involves the public key of the server, while the latter involves your account public key.
Read about SSH Key Pairs to learn the difference.
FileZilla cannot connect without verifying the server's public host key either. On the first connection it always prompts you to accept the key. Once you do, it optionally caches the key and won't prompt you again, unless the key changes.
You have probably forgotten that you got this prompt before or someone else connected to the server before from your machine.
Any SSH (SFTP) client must do the same. You are losing any security had you not verified your server's host key.
You should get the host key fingerprint from your server administrator.
If you had not, you can see it on WinSCP Server and Protocol information dialog.
For details see WinSCP FAQ Where do I get SSH host key fingerprint to authorize the server?
I solved this by just copying the SSH key returned to my FileZilla client into my C# app. I don't know if this is the right thing to do, but at least it got my solution working now.
It was also an SSH-DSS key 2048 key instead of an SSH-RSA 1024, and that's why messing around with the keys kept failing I guess.