I am trying to send text to a server listening on a TCP port with R, and then read the response text from the server.
Quite trivial, i.e., on the BASH for a server listening to port 12345, that is:
> echo "text" | nc localhost 12345
response
The server keeps running and can be queried any time again after this.
But if I try the same thing from within R with socketConnection, I either never get a response, or it is printed and not captured.
I have tried the following:
con <- socketConnection(port=12345)
con <- socketConnection(port=12345, blocking=TRUE, timeout=2)
writeLines("text", con) # server does not receive a thing
flush(con) # has no effect
readLines(con) # still, nothing happens and gets nothing back
close(con) # server confirms receipt, but I no longer can get the result...
The server only receives the data after closing the connection, so nothing can be read
con <- pipe("nc localhost 12345")
writeLines("text", con)
Now, "result" is printed to STDOUT, so I cannot capture it...
If using a temporary file that contains "text":
res <- readLines(pipe("nc localhost 12345 < tempfile"))
That works, but requires an intermediate, temporary file.
How do I get server communication to work in R so that I can write and then read from the same connection?
I compiled and ran this simple server, leading to
Socket created
bind done
Waiting for incoming connections...
Then in R I created a connection
con <- socketConnection("127.0.0.1", port = 8888)
the server responded
Connection accepted
and back in R...
writeLines("all the world's a stage", con)
x = readLines(con)
x
## [1] "all the world's a stage"
close(con)
to which the server responded
Client disconnected
and then quit, as expected. Not sure how this differs from what you've tried.
Related
I am running SQL-code on a oracle database. Some commands require to run them via sqlplus. Is there a way to avoid my commandline solution but directly running sqlplus via, e.g. dbSendStatement().
Pseudo code to not share any sensible information
# Via dbSendStatement ------------------------------------------------------------------------------
con <- odbc::dbConnect(odbc::odbc(),
Driver = "oracle",
Host = "HOST",
Port = "PORT",
SVC = "SVC",
UID = Sys.getenv("USRDWH"),
PWD = Sys.getenv("PWDDWH"),
ssl = "true",
timeout = 10)
# Error
odbc::dbSendStatement(con, "EXEC SQL CODE")
# actual error message:
#> Error in new_result(connection#ptr, statement, immediate) :
#> nanodbc/nanodbc.cpp:1594: 00000: [RStudio][OracleOCI] (3000) Oracle Caller Interface: ORA-00900: invalid SQL statement
# Via system command -------------------------------------------------------------------------------
cmd <- paste0("sqlplus ",
Sys.getenv("USRDWH"), "/", Sys.getenv("PWDDWH"),
"#", "HOST", ":", "PORT", "/", "SVC", " ",
"#", "EXEC script.sql")
cmd
#> [1] "sqlplus USR/PWD#HOST:PORT/SVC #EXEC script.sql"
# Works
system(cmd,
intern = TRUE)
Code like that always connects directly to the database. sqlplus is a specific client tool; it doesn't have its own API for those kind of interactions. In other words, you always connect to the database; you can't connect to sqlplus as it is not a service.
Your best option would be to convert your SQL in such a way that you can run it natively in your code using a direct database connection (i.e. don't use sqlplus). If your SQL commands cannot be adapted, then you will need to write a shell interaction to manipulate sqlplus as you did with cmd in your example.
That said, this implementation in your example is very insecure, as it will allow anyone with access to your host to see the database username, password, and connection info associated with the process while it is running. There are much more secure ways of scripting this, including the use of an auto-open Oracle Wallet to hold the credentials so you don't have to embed them in your code (which is always a bad idea, too).
Using Oracle Wallet, your cmd call would then look more like this:
sqlplus /#TNS_ALIAS #EXEC script.sql
This is still not perfect, but is a step or two in the right direction.
I'm trying to query a MySQL server through an R/Tidyverse/dbplyr workflow. My MySQL access requires configuring SSH and port forwarding.
I have this code working using python (below), but I'm struggling to get started with the SSH/port forwarding equivalent in R. Any pointers to solutions or equivalent R packages appreciated. thanks.
import pymysql
import paramiko
import pandas as pd
from paramiko import SSHClient
from sshtunnel import SSHTunnelForwarder
from os.path import expanduser
pkeyfilepath = '/.ssh/id_ed25519'
home = expanduser('~')
mypkey = paramiko.Ed25519Key.from_private_key_file(home + pkeyfilepath)
sql_hostname = 'mycompany.com'
sql_username = 'me'
sql_password = '***'
sql_main_database = 'my_db'
sql_port = 3306
ssh_host = 'jumphost.mycompany.com'
ssh_user = 'me'
ssh_port = 22
with SSHTunnelForwarder(
(ssh_host, ssh_port),
ssh_username=ssh_user,
ssh_pkey=mypkey,
remote_bind_address=(sql_hostname, sql_port)) as tunnel:
conn = pymysql.connect(host='127.0.0.1', user=sql_username,
passwd=sql_password, db=sql_main_database,
port=tunnel.local_bind_port)
query = '''SELECT VERSION();'''
data = pd.read_sql_query(query, conn)
print(data)
conn.close()
There are several ways to do ssh port forwarding for R. In no particular order:
I forward it externally to R. All of my work is remote, and for one particular client I need access to various instances of SQL Server, Redis, MongoDB, remote filesystems, and a tunnel-hop to another network only accessible from the ssh bastion host. I tend to do work in more than R, so it's important to me that I generalize this. It is not for everybody or every task.
For this, I used a mismash of autossh and my ssh-agent (in KeePass/KeeAgent).
The ssh package does have a function to Create a Tunnel. The premise is that you have already created a "session" to which you can add a forwarding rule(s). When using ssh::ssh_tunnel, it is blocking, meaning you cannot use it in the same R process and continue to work. Demo:
# R session 1
sess <- ssh::ssh_connect("user#remote")
# insert passphrase
ssh::ssh_tunnel(sess, 21433, "otherremote:1433")
# / Waiting for connection on port 21433...
# R session 2
con <- DBI::dbConnect(..., port=21433)
DBI::dbGetQuery(con, "select 1 as n")
# n
# 1 1
This connection will stay alive so long as con is not closed and the remote end does not close it (e.g., activity timeout).
Note: I cannot get the ssh package to use my ssh-agent, so all passwords must be typed in or otherwise passed in not-ideal ways. There are many ways to not have to type it, such as using the keyring package (secure) or envvars, both of which would pass the password to ssh_connect(..., passwd=<>).
The above, but using callr so that you don't need to explicit sessions active (though you will still have another R session.
bgr <- callr::r_bg(function() {
ssh <- ssh::ssh_connect("r2#remote", passwd=keyring::key_get("r2", "remote"))
ssh::ssh_tunnel(ssh, port=21433, "otherremote:1433")
}, supervise = TRUE)
DBI::dbGetQuery(con, "select 1 as n")
# n
# 1 1
### when your work is done
bgr$kill()
If you do this, I strongly recommend the use of supervise=TRUE, which ensures the background R process is killed when this (primary) R session exits. This will reduce the risk of having phantom unused R sessions hanging around; in addition to just clogging up the process tree, if one of these phantom R processes is actively forwarding a port, that means nothing else can forward that port. This allows you to continue working, but you are not longer in control of the process doing the forwarding ... and subsequent attempts to tunnel will fail.
FYI, I generally prefer using keyring::key_get("r2", "remote") for password management in situations like this: (1) it prevents me from having to set that envvar each time I start R ... which will inadvertently store the plain-string password in ~/.Rhistory, if saved; (2) it prevents me from having to set that envvar in the global environment permanently, which is prone to other stupid mistakes I make; and (3) is much better protected since it is using the native credentials of your base OS. Having said that, you can replace the above use of keyring::key_get(..) with Sys.getenv("mypass") in a pinch, or in a case where the code is running on a headless system where a credentials manager is unavailable.
And if you want this to be a little more resilient to timeout disconnects, you can instead use
bgr <- callr::r_bg(function() {
ssh <- ssh::ssh_connect("r2#remote", passwd=keyring::key_get("r2", "remote"))
while (!inherits(try(ssh::ssh_tunnel(ssh, port=21433, "otherremote:1433"), silent=TRUE), "try-error")) Sys.sleep(1)
}, supervise = TRUE)
which will repeatedly make the tunnel so long as the attempt does not error. You may need to experiment with this to get it "perfect".
callr is really just using processx under the hood to start a background R process and allow you to continue working. If you don't want the "weight" of another R process solely to forward ports, you can use processx to start an explicit call to ssh that does everything you need it to do.
proc <- processx::process$new("ssh", c("-L", "21433:otherremote:1433", "r2#remote", "-N"))
### prompts for password
DBI::dbGetQuery(con, "select 1 as n")
# n
# 1 1
### when done
proc$kill()
# [1] TRUE
I'm using the AzureR packages to access Azure Data Lake Storage from RStudio.
I setup the connections with the script below:
library(AzureRMR)
library(AzureStor)
# setup connections
az <- az_rm$new(tenant="my_tenant_id",
app="my_app_id",
password="my_password")
sub <- az$get_subscription("my_subscription_id")
rg <- sub$get_resource_group("my_resource_group_name")
stor <- rg$get_resource(type="Microsoft.Storage/storageAccounts",
name="my_datalake_account_name")
stor$do_operation("listKeys", http_verb="POST")
The connection works well and I got the following result:
attr(,"status")
[1] 200
Then I upload file to and download file from the ADLS file system with the script below:
fs <- adls_filesystem(
"https://my_datalake_account_name.dfs.core.windows.net/my_file_system_name",
key="my_key"
)
# create new directory
create_adls_dir(fs, "/newdir")
upload_adls_file(
fs, src = "I:/lookup.csv",
dest = "/newdir/lookup.csv"
)
download_adls_file(
fs, src = "/newdir/lookup.csv",
dest = "J:/lookup.csv"
)
The upload works good, while the download show the following errors:
Connection error, retrying (1 of 10)
Connection error, retrying (2 of 10)
Connection error, retrying (3 of 10)
Connection error, retrying (4 of 10)
Connection error, retrying (5 of 10)
Connection error, retrying (6 of 10)
Connection error, retrying (7 of 10)
Connection error, retrying (8 of 10)
Connection error, retrying (9 of 10)
Connection error, retrying (10 of 10)
Error in curl::curl_fetch_memory(url, handle = handle) :
Send failure: Connection was reset
Right now I have two servers that I can work on, however the goal is to switch to the new server and eliminate the old server. The script works good on the old server, the CSV file is very small so the upload and download finish within a sec. However, on the new server, the upload works good while the download failed. Any ideas about what might cause this issue? I'm wondering if there is any system setting that are different between the two servers but I'm really new to the Data Lake. Any help would be very appreciate!
Since the same script is working on the old server , I think if I were you i could have focused the path "J:/lookup.csv" . Is J: a physical drive or FIle share .
Anyways I could have tested with something more obvious like c or d drive and test that part .
I have an R Code from server side:
socket.reader <- function(host="localhost", port=6011, time_freq){
shell(paste("C:/Users/Stefan/source/repos/Quick-Check/Debug/Quick-Check.exe","COM3","127.0.0.1",port,time_freq))
con <- socketConnection("localhost", port,server=TRUE, open="r+")
txt <- readLines(con)
close(con)
return(txt)
}
observeEvent(input$socket_reader, {
input<-socket.reader(time_freq=input$Samp_Freq*60)
#tosave<-ts(as.numeric(input), frequency=200)
write.csv(input,"C:/Users/Stefan/Documents/app/data/samplerun")
})
It is supposed to start socket server, then execute the .exe and read in all that the .exe sents to the server(exe is from cpp measuring some device).
Problem is, as soon as the socket connection opens on server side, the r-process seems to be blocked and does not evaluate the .exe until the socket is closed by timeout again. I tried to put the .exe before the server is opened and put a 3 second delay within the .exe, but it seems, that the .exe starts, gets a delay, proceeds to evaluate the r-server socket connection, which again produces a hold until timeout.
Does anyone have an idea, how to start another process from within R so that both will be evaluated at the same time or to unhold the R instance while the socket connection is opened?
Best Wishes
If wait = FALSE works on your system, you can specify that option whan starting the socket server:
system2("nc", "-l -p 6011 -c /bin/cat", wait = FALSE)
con <- socketConnection(host="localhost", port = 6011, blocking=TRUE,
server=FALSE, open="r+")
write_resp <- writeLines("Hello, world!", con)
server_resp <- readLines(con, 1)
print(paste("Server response: ", server_resp))
close(con)
Alternatively you can use the future package:
library(future)
plan(multiprocess)
server %<-% {
system2("nc", "-l -p 6011 -c /bin/cat")
}
con <- socketConnection(host="localhost", port = 6011, blocking=TRUE,
server=FALSE, open="r+")
write_resp <- writeLines("Hello, world!", con)
server_resp <- readLines(con, 1)
print(paste("Server response: ", server_resp))
close(con)
Here only the server is declared as a future, whereas the client code is executed within the parent session.
Note: the nc command just opens a simple echo server on port 6011.
Does anyone know how to use python to ping a local host to see if it is active or not? We (my team and I) have already tried using
os.system("ping 192.168.1.*")
But the response for destination unreachable is the same as the response for the host is up.
Thanks for your help.
Use this ...
import os
hostname = "localhost" #example
response = os.system("ping -n 1 " + hostname)
#and then check the response...
if response == 0:
print(hostname, 'is up!')
else:
print(hostname, 'is down!')
If using this script on unix/Linux replace -n switch with -c !
Thats all :)
I've found that using os.system(...) leads to false positives (as the OP said, 'destination host unreachable' == 0).
As stated before, using subprocess.Popen works. For simplicity I recommend doing that followed by parsing the results. You can easily do this like:
if ('unreachable' in output):
print("Offline")
Just check the various outputs you want to check from ping results. Make a 'this' in 'that' check for it.
Example:
import subprocess
hostname = "10.20.16.30"
output = subprocess.Popen(["ping.exe",hostname],stdout = subprocess.PIPE).communicate()[0]
print(output)
if ('unreachable' in output):
print("Offline")
The best way I could find to do this on Windows, if you don't want to be parsing the output is to use Popen like this:
num = 1
host = "192.168.0.2"
wait = 1000
ping = Popen("ping -n {} -w {} {}".format(num, wait, host),
stdout=PIPE, stderr=PIPE) ## if you don't want it to print it out
exit_code = ping.wait()
if exit_code != 0:
print("Host offline.")
else:
print("Host online.")
This works as expected. The exit code gives no false positives. I've tested it in Python 2.7 and 3.4 on Windows 7 and Windows 10.
I've coded a little program a while back. It might not be the exact thing you are looking for, but you can always run a program on the host OS that opens up a socket on startup. Here is the ping program itself:
# Run this on the PC that want to check if other PC is online.
from socket import *
def pingit(): # defining function for later use
s = socket(AF_INET, SOCK_STREAM) # Creates socket
host = 'localhost' # Enter the IP of the workstation here
port = 80 # Select port which should be pinged
try:
s.connect((host, port)) # tries to connect to the host
except ConnectionRefusedError: # if failed to connect
print("Server offline") # it prints that server is offline
s.close() #closes socket, so it can be re-used
pingit() # restarts whole process
while True: #If connected to host
print("Connected!") # prints message
s.close() # closes socket just in case
exit() # exits program
pingit() #Starts off whole process
And here you have the program that can recieve the ping request:
# this runs on remote pc that is going to be checked
from socket import *
HOST = 'localhost'
PORT = 80
BUFSIZ = 1024
ADDR = (HOST, PORT)
serversock = socket(AF_INET, SOCK_STREAM)
serversock.bind(ADDR)
serversock.listen(2)
while 1:
clientsock, addr = serversock.accept()
serversock.close()
exit()
To run a program without actually showing it, just save the file as .pyw instead of .py.
It makes it invisible until user checks running processes.
Hope it helped you
For simplicity, I use self-made functions based on socket.
def checkHostPort(HOSTNAME, PORT):
"""
check if host is reachable
"""
result = False
try:
destIp = socket.gethostbyname(HOSTNAME)
except:
return result
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
s.settimeout(15)
try:
conn = s.connect((destIp, PORT))
result = True
conn.close()
except:
pass
return result
if Ip:Port is reachable, return True
If you wanna to simulate Ping, may refer to ping.py
Try this:
ret = os.system("ping -o -c 3 -W 3000 192.168.1.10")
if ret != 0:
print "Host is not up"
-o waits for only one packet
-W 3000 gives it only 3000 ms to reply to the packet.
-c 3 lets it try a few times so that your ping doesnt run forever
Use this and parse the string output
import subprocess
output = subprocess.Popen(["ping.exe","192.168.1.1"],stdout = subprocess.PIPE).communicate()[0]
How about the request module?
import requests
def ping_server(address):
try:
requests.get(address, timeout=1)
except requests.exceptions.ConnectTimeout:
return False
return True
No need to split urls to remove ports, or test ports, and no localhost false-positive.
Timeout amount doesn't really matter since it only hits the timeout when there is no server, which in my case meant performance no longer mattered. Otherwise, this returns at the speed of a request, which is plenty fast for me.
Timeout waits for the first bit, not total time, in case that matters.