I am getting broken pipe error for the following code using paramiko
import paramiko
proxy = paramiko.ProxyCommand('/usr/bin/nc -vvv -xhost:1080 %h %p')
client = paramiko.SSHClient()
client.set_missing_host_key_policy(paramiko.AutoAddPolicy())
client.connect('hostname', 22,username='user', password='pwd', sock=proxy)
stdin, stdout, stderr = client.exec_command('ls')
print "output", stdout.read()
error = stderr.read()
print "err", error, len(error)
ftp = client.open_sftp()
ftp.get('file')
ftp.close()
client.close()
Error I am getting is
paramiko.ssh_exception.ProxyCommandFailure: ('/usr/bin/nc -vvv -w30 -xhost:1080 %h %p', 'Broken pipe')
Related
My issue is: when I run the following code from one laptop in RScript.exe via Task Scheduler, I get the desired output; that is the email is sent. But when I run the same code on another machine in RScript.exe via Task Scheduler, it doesn't run. Another machine (machine 2) is able to send emails (when only the code for email is run), so I think the issue is with the following part.
results <- get_everything(query = q, page = 1, page_size = 2, language = "en", sort_by = "popularity", from = Yest, to = Today)
I am unable to find what is the issue here. Can someone please help me with this?
My code is:
library(readxl)
library(float)
library(tibble)
library(string)
library(data.table)
library(gt)
library(tidyquant)
library(condformat)
library(xtable)
library(plyr)
library(dplyr)
library(newsanchor)
library(blastula)
Today <- Sys.Date()
Yest <- Sys.Date()-1
results <- get_everything(query = "Inflation", page = 1, page_size = 2, language =
"en", sort_by = "popularity", from = Yest, to = Today, api_key =
Sys.getenv("NEWS_API_KEY"))
OP <- results$results_df
OP <- OP[-c(1, 5:9)]
colnames(OP) <- c("News Title", "Description", "URL")
W <- print(xtable(OP), type="html", print.results=FALSE, align = "l")
email1 <-
compose_email(
body = md(
c("<tr>", "<td>", "<table>", "<tr>", "<td>", "<b>", "Losers News", "</b>", W,
"</td>", "</tr>", "</table>","</td>", "<td>")
)
)
email1 %>%
smtp_send(
from = "abc#domain.com",
to = "pqr#domain.com",
subject = "Hello",
credentials = creds_key(
"XYZ"
)
)
Whenever you schedule jobs, consider using a command line shell such as PowerShell or Bash to handle the automation steps, capture, and log errors and messages. Rscript fails on the second machine for some unknown reason which you cannot determine since you do not receive any error messages from console using TaskScheduler.
Therefore, consider PowerShell to run all needed Rscript.exe calls and other commands and capture all errors to date-stamped log file. Below script redirects all console output to a .log file with messages. When Rscript command fails, the log will dump error or any console output (i.e., head, tail) below it. Regularly check logs after scheduled jobs.
PowerShell script (save as .ps1 file)
cd "C:\path\to\scripts"
& {
echo "`nAutomation Start: $(Get-Date -format 'u')"
echo "`nSTEP 1: myscript.R - $(Get-Date -format 'u')"
Rscript myscript.R
# ... ADD ANY OTHER COMMANDS ...
echo "`nCAutomation End: $(Get-Date -format 'u')"
} 3>&1 2>&1 > "C:\path\to\logs\automation_run_$(Get-Date -format 'yyyyMMdd').log"
Command Line (to be used in Task Scheduler)
Powershell.exe -executionpolicy remotesigned -File myscheduler.ps1
Note: Either change directory in TaskScheduler job settings where myscheduler.ps1 resides or run absolute path in -File argument.
So, I created the following HTTP server tunneled via ngrok, and I am trying to send a file to the server, to then read it and display it on the web page of the server.
Here's the code for the server:
import os
from http.server import HTTPServer, BaseHTTPRequestHandler
from pyngrok import ngrok
import time
port = os.environ.get("PORT", 80)
server_address = ("127.0.0.1", port)
class MyServer(BaseHTTPRequestHandler):
def _set_headers(self):
self.send_response(200)
self.send_header('Content-type', 'text/html')
self.end_headers()
def do_GET(self):
self._set_headers()
self.wfile.write(bytes("<html><head><title>https://pythonbasics.org</title></head>", "utf-8"))
self.wfile.write(bytes("<p>Request: %s</p>" % self.path, "utf-8"))
self.wfile.write(bytes("<body>", "utf-8"))
self.wfile.write(bytes("<p>This is an example web server.</p>", "utf-8"))
self.wfile.write(bytes("</body></html>", "utf-8"))
def do_POST(self):
'''Reads post request body'''
self._set_headers()
content_len = int(self.headers.getheader('content-length', 0))
post_body = self.rfile.read(content_len)
self.wfile.write("received post request:<br>{}".format(post_body))
def do_PUT(self):
self.do_POST()
httpd = HTTPServer(server_address, MyServer)
public_url = ngrok.connect(port).public_url
print("ngrok tunnel \"{}\" -> \"http://127.0.0.1:{}\"".format(public_url, port))
try:
# Block until CTRL-C or some other terminating event
httpd.serve_forever()
except KeyboardInterrupt:
print(" Shutting down server.")
httpd.socket.close()
And I have been trying to send a file using POST as follow
>>> url = 'https://httpbin.org/post'
>>> files = {'file': open('report.xls', 'rb')}
>>> r = requests.post(url, files=files)
>>> r.text
I imported requests of course, and here's what I get
Exception occurred during processing of request from ('127.0.0.1', 60603)
Traceback (most recent call last):
File "C:\Program Files\Python39\lib\socketserver.py", line 316, in _handle_request_noblock
self.process_request(request, client_address)
File "C:\Program Files\Python39\lib\socketserver.py", line 347, in process_request
self.finish_request(request, client_address)
File "C:\Program Files\Python39\lib\socketserver.py", line 360, in finish_request
self.RequestHandlerClass(request, client_address, self)
File "C:\Program Files\Python39\lib\socketserver.py", line 720, in __init__
self.handle()
File "C:\Program Files\Python39\lib\http\server.py", line 427, in handle
self.handle_one_request()
File "C:\Program Files\Python39\lib\http\server.py", line 415, in handle_one_request
method()
File "C:\Users\pierr\OneDrive\Desktop\SpyWare-20210104T124335Z-001\SpyWare\Ngrok_Test.py", line 28, in do_POST
content_len = int(self.headers.getheader('content-length', 0))
AttributeError: 'HTTPMessage' object has no attribute 'getheader'
Could someone please help me fix this error ? I don't get where it comes from.
Syntax has changed. You need to use
content_len = int(self.headers.get('Content-Length'))
Instead of
content_len = int(self.headers.getheader('content-length', 0))
The rest should be the same
I have a server with installed Postfix SMTP service, and I can send messages using bash like this:
echo "This is the body of the email" | mail -s "This is the subject line" user#example.com
But when I'm trying to do the same with Python it hangs forever:
import smtplib
s = smtplib.SMTP('localhost')
s.send_message('')
The script hangs on the second line, and it is not clear why.
I've checked the configuration of iptables (it is empty) and I still able to send messages with bash command.
"telnet localhost 25" also works fine, port is open.
Postfix config file:
mailbox_size_limit = 0
recipient_delimiter = +
inet_interfaces = localhost
mynetworks = 127.0.0.0/8
myorigin = /etc/mailname
send_message() expects an EmailMessage not a string. Currently it appears to hang because postfix is waiting for you to tell it to do something, you can enable debugging output using set_debuglevel().
Here's a simple example that should work:
from email.message import EmailMessage
import smtplib
msg = EmailMessage()
msg["From"] = "user#example.com"
msg["To"] = "another-user#example.com"
msg["Subject"] = "Test message subject."
msg.set_content("Test message.")
s = smtplib.SMTP("localhost")
s.set_debuglevel(1)
s.send_message(msg)
s.quit()
I am trying to run some autoit.au3 script from command line and see results there. I have put some ConsoleWrite inside script and also Exit(1) but after I run script nothing is shown in console. It just stop script on Exit and ConsoleWrite is not displayed.
I have use command:
"C:...(path to my AutoIt3.exe)" /ErrorStdOut "path_to_my_script.au3"'
Also I have tried to run script.exe with this same command but with similar (no) result. I would like to see output in console and/or custom error messages when script fail (I don't know if that is possible).
AutoIt3.exe is a GUI program. So the STD streams of a GUI program are not printed at a console by default.
The /ErrorStdOut argument redirects errors messages to Console instead of a Msgbox.
This argument does not enable print at the Console.
Command Prompt:
To print at a Command Prompt, you could pipe to more, i.e.
"C:...(path to my AutoIt3.exe)" /ErrorStdOut "path_to_my_script.au3" 2>&1|more
more reads from the Stdin stream and prints to Console.
I intentionly added 2>&1 so the Stderr stream is merged with
Stdout so you get the merged streams printed.
If you do not want the errors, then you can redirect the Stderr stream to nul i.e.
replace 2>&1 with 2>nul.
If you used a for loop at a Command Prompt, it would be i.e.
for /f "delims=" %A in ('"C:...(path to my AutoIt3.exe)" /ErrorStdOut test1.au3') do echo %A
If you use the for loop in batch-file, use %%A instead of %A. To also capture the Stderr, insert 2^>&1 into the for command or to ignore, insert 2^>nulinto the for command i.e.
for /f "delims=" %A in ('2^>nul "C:...(path to my AutoIt3.exe)" /ErrorStdOut test1.au3') do echo %A
The previous methods will not get the Exitcode.
AutoIt code:
An AutoIt script can get the Stdout and the Exitcode.
#pragma compile(Out, 'consoleau3.exe')
#pragma compile(Console, True)
$sAutoit = 'C:...(path to my AutoIt3.exe)'
$iPid = Run('"' & $sAutoit & '" /ErrorStdout ' & $CMDLINERAW, '', #SW_SHOW, 2) ; 2 = Get Stdout stream.
If #error Then Exit
; Open process handle.
$hPid = _ProcessOpenHandle($iPid)
; Get Stdout stream and then print to Console.
$sStdout = ''
Do
Sleep(10)
If $sStdout Then ConsoleWrite($sStdout & #CRLF)
$sStdout = StdoutRead($iPid)
Until #error
; Require process to be closed before calling _ProcessGetExitCode()
ProcessWaitClose($iPid)
; Get exitcode of process.
$iExitcode = _ProcessGetExitCode($hPid)
; Close process handle.
_ProcessCloseHandle($hPid)
Exit $iExitcode
Func _ProcessOpenHandle($iPID)
; Get the process handle of the process to query\n Return: Success Handle as array. Failure 0
Local Const $PROCESS_QUERY_INFORMATION = 0x400
Local $hPID = DllCall('kernel32.dll', 'ptr', 'OpenProcess', 'int', $PROCESS_QUERY_INFORMATION, 'int', 0, 'int', $iPID)
If #error Then Return SetError(#error, #extended, 0)
Return $hPID[0]
EndFunc
Func _ProcessGetExitcode($hPID)
; Get exitcode of the closed process\n Return: Success Exitcode as integer. Failure 0
Local $vPlaceholder
$hPID = DllCall('kernel32.dll', 'ptr', 'GetExitCodeProcess', 'ptr', $hPID, 'int*', $vPlaceholder)
If #error Then Return SetError(#error, #extended, 0)
Return $hPID[2]
EndFunc
Func _ProcessCloseHandle($hPID)
; Close the handle of a process\n Return: Success 1. Failure 0
DllCall('kernel32.dll', 'ptr', 'CloseHandle', 'ptr', $hPID)
If #error Then Return SetError(#error, #extended, 0)
Return 1
EndFunc
Correct the path to AutoIt.exe in the code.
Compile to AutoIt code to executable. It will be a Console program
and will be named consoleau3.exe as to the #pragma compile directives.
Usage:
consoleau3 "path_to_my_script.au3"
Script arguments can be added i.e.
consoleau3 "path_to_my_script.au3" arg1 arg2 arg3 ...
Looking for an options which let me to redirect R diagnostic messages (produces by message()) to stdout, not stderr as it is by default.
message manual states:
The default handler sends the message to the stderr() connection.
So the question is how can I change this default behavior? still leaving redirection of warning() and stop() intact.
Already tried sink type='message' but it redirects all (messages, warnings, errors).
If anyone is willing to test, this is sample script exec_test.R:
print("using print")
cat("using cat\n")
message("using message")
warning("using warning")
stop("using stop")
q("no")
which then will be executed by:
Rscript exec_test.R 1>> exec_test.Rout 2>> exec_test_error.Rout
I don't what to use 2>&1 because my script produce tons of messages and very rarely the real errors so I need to store those logs in separate files.
Using sink. Here's a modification of your code:
sink(stdout(), type = "message") # sink messages to stdout
print("using print")
cat("using cat\n")
message("using message")
warning("using warning")
sink(NULL, type="message") # close the sink
warning("after ending sink") # this will be the only thing in your err file
q("no")
The OP showed the execution happening via the Rscript command and using some I/O redirection. If you want to use redirection to log everything and only show to console on error, the best method I've found is to use || to check if the script had non-zero exit status before printing to screen:
Rscript myrscript.R > temp.log 2>&1 || cat temp.log
This method relies strictly on the exit code for printing, which only partly gets around message() going to stderr, but I thought this example helpful to mention since messages won't necessarily trigger a non-zero exit status and you can continue to log quietly with this method.
If you'd like to go one step further and keep appending to a single log file, then this will work:
Rscript myrscript.R > temp.log 2>&1 || cat temp.log && cat temp.log >> persistent.log && rm temp.log
The pseudocode for this command is:
Redirect stderr and stdout into temp.log
If command had non-zero exit status then write to screen
Then redirect the contents of temp.log into your persistent.log
Then remove temp.log
While this is very likely not a best practice, you could override message with a version that writes to stdout() by default, right?
message <- function (..., domain = NULL, appendLF = TRUE)
{
args <- list(...)
cond <- if (length(args) == 1L && inherits(args[[1L]], "condition")) {
if (nargs() > 1L)
warning("additional arguments ignored in message()")
args[[1L]]
}
else {
msg <- .makeMessage(..., domain = domain, appendLF = appendLF)
call <- sys.call()
simpleMessage(msg, call)
}
defaultHandler <- function(c) {
cat(conditionMessage(c), file = stdout(), sep = "")
}
withRestarts({
signalCondition(cond)
defaultHandler(cond)
}, muffleMessage = function() NULL)
invisible()
}
Combining #Thomas and #branch14's answers into one, this is what I came up with:
.stderr_message <- message
message <- function(...) {
sink(stdout(), type = "message")
.stderr_message(...)
sink(NULL, type = "message")
}
I verified this on the shell using this code
.stderr_message <- message
message <- function(...) {
sink(stdout(), type = "message")
.stderr_message(...)
sink(NULL, type = "message")
}
print(1)
message(1)
Without my message():
> Rscript bug.R 2> >(sed 's/^/stderr: /') > >(sed 's/^/stdout: /')
stdout: [1] 1
stderr: 1
> Rscript bug.R 2> >(sed 's/^/stderr: /') > >(sed 's/^/stdout: /')
stdout: [1] 1
stdout: 1