I'm working on an assignment where I need to remote connect into my company's UNIX boxes and parse out a particular set of log entires. I've figured out a method for doing so with Grep and the -C flag, but the version of UNIX installed on these machines doesn't support that functionality. One alternative I've considered is doing the work on my local machine through Cygwin and using the local version of Grep to handle this task. However, these logs are especially larges, upwards of 50 megabytes and the connection to the boxes are very slow so it would take several hours to complete the downloads.
My main question, is it possible to remote connect through SSH to a remote server, but be able to invoke the locally installed versions of certain programs? For example, if I SSH into the server, can I make use of the local version of Grep instead of the remote system's version of Grep?
I've attempted to do something similar using Awk and Sed but I haven't had much success. At this point, aside from a long period of downloading, I'm not sure what other options I have. Any advice? Thanks in advance. :)
Even if you could use a remote file with a local application, you'd still be downloading the entirety of the log files - ssh allows output/input to pass between boxes, but you're not actually running your local grep on the local machine - it'd be the remote machine sending its file to your local grep.
One alternative is to gzip the logfiles before sending them through ssh,e.g.
ssh user#remotebox 'gzip -9 - logfile'|gzcat -|grep whatever
You'd still be sending the entirety of the log files, but log files tend to compress very well, so you'd only be sending a small fraction of the original data (e.g. a couple megs v.s. 50 uncompressed).
Or, in the alternative, you could try compiling gnu grep from source on the remote machines, assuming there's an appropriate compiler toolchain on those machines.
Related
I am used to using R in RStudio. For a new project, I have to use R on the command line, because the data storage and analysis are only allowed to be on a specific server that I connect to using ssh. This server doesn't have rstudio-server to support remote RStudio sessions.
The project involves an extremely large dataset, and some pre-written code to load/format the data that I have been told to run using "source()" before I do anything else. This takes several minutes to run and load the data each time.
What would a good workflow be for something like this? Editing my code in a .r file, saving, then running it would require taking several minutes to load the data each time. But just running R in an interactive session would make it hard to keep track of what I am doing and repeat things if necessary.
Is there some command-line equivalent to RStudio where you can have an interactive session but be editing/saving a file of your code as you go?
Sounds like JuPyteR might be your friend here.
The R kernel works great.
You can use it on a remote server either with exposing an open port (and setting up JuPyteR login credentials)
Or via port forwarding over SSH.
It is a lot like an interactive reply, except it holds state.
And you can go back and rerun cells.
(Of course state can be dangerous for reproduceability)
For RStudio you can launch console and ssh to your remote servers even if your servers don't use expensive RStudio for servers platform. You can then execute all commands from R Studio directly into the ssh with the default shortcut key. This might allow to continue using R studio, track what you're doing in the R script, execute interactively.
I have one .bat script on my windows share that is mounted to my UNIX machine. Bat script is set to make file transfer between 2 windows shares, but I would like to trigger this script from a unix machine if that is possible. I was reading that you can do it with wine or dosbox, but I don't have that installed on my unix. Is it possible to resolve this problem with some additional .sh script that will trigger my .bat script correctly?
Thank you in advance.
Best regards.
You cannot run a .bat script on a Unix machine for several reasons :
Unix has not the same commands (on the command line) as Windows. The POSIX standard defined a set of commands, if you use them you'll be portable on various POSIX systems (but not on Windows); for example to list a directory, you'll use DIR on MSDOS and Windows but ls on Unix and POSIX; to copy a file it is COPY on MSDOS and Windows but cp on Unix and POSIX; etc....
Unix has not the same command interpreter as Windows. The POSIX standard and the Unix tradition provides a Unix shell and POSIX has standardized /bin/sh (a.k.a. POSIX shell). Windows has CMD (inherited from MSDOS) and PowerShell.
The way of interpreting commands is different (on Windows look also into PowerShell, which I don't know). On Unix it is the shell (not the invoked programs) that is expanding your command and globbing. See this answer for more. The notion of current working directory is different.
the operating system concepts are (slightly or significantly) different on Windows and on Unix or POSIX. For example, files, directories, processes, libraries are different (for example, a file can be written by a process and removed by another one on Unix and it can have several names on Linux thru hard links), .... etc.... You could read Operating Systems: Three Easy Pieces for an overview.
the Unix philosophy is not (always) applicable to Windows.
So you need to study Unix (or POSIX) and write your own shell script from scratch. Don't try to "translate" a bat script to a Unix shell script, but redesign it entirely (starting from the problem you want it to solve).
(and Wine or DosBox is not helpful in your case)
Read also about SCP and perhaps FTP. Perhaps using some distributed version control system like git could be relevant for you (e.g. to share scripts, source code, etc...).
If you need to run remotely some Windows .bat script on a distant Windows machine (e.g. from a Unix machine), you should use some remote command running service (that is, find and use some equivalent of SSH service on Windows, and use the corresponding client on Unix). See this.
So if you need to remotely run on a Windows server something (e.g. some program, some script, some command) from a Unix machine you should ask a different question (or at least improve a lot the current one).
Read about the client-server model and about application layer to use the correct terminology. You should name what protocol, server, client, service you want to involve. Nothing is magically "triggered" without using them.
PS. I'm using Unix since 1987, Linux since 1993. I never used Windows.
I'm looking for a way to develop some Unix scripts that will connect to a DOS box (Windows server 2012) and interactively execute DOS commands.
I'm comfortable with the Unix side (I'll almost certainly use Expect), but I'm "Windows illiterate" and am unable to find anything about connecting to Windows's DOS command line in this fashion. Is this even possible to do?
(FWIW, this is to enable us to control Tableau Server using its 'tabcmd' DOS command suite from our existing Linux environment.)
UPDATE 1:
I think another way of asking the question is: does Windows provide anything that is the equivalent of the Unix "remote shell", accessible from Unix?
There are no built-in tools to do this, although PsExec is a utility that can almost do what you want. PsTools are not a built-in, but are hosted on Technet. Some things to keep in mind:
PsExec works by actually remotely copying a file over to the Windows System32 folder (copying is one thing that is builtin ;)
Windows uses Kerberos for authentication. This depends on the computer you are running the command from being on the same Active Directory as the computer you want to control, with access set up from that side. Linux can use kerberos through third-party AD integration tools (like Quest Authentication Services, a commercial product), or also Centrify but there are no built-in tools that do it.
Psexec is not encrypted, meaning if you send commands containing sensitive data, they can be seen (though not the authentication part of it).
PsExec is obviously still a Windows utility. I have been able to get it to work using Wine, but only for a local account and after some tweaking with matching hostnames and stuff. It's possible that if you have authenticated using QAS or Centrify that your wine command will somehow pick it up, but I haven't tested it; I don't work where we use AD anymore.
Maybe the biggest problem is the difference of philosophy between the two communities. Windows doesn't use command line execution very often for remote administration. There is more focus on using your local utilities on a remote system (i.e., you can load a remote registry hive from RegEdit or browse the file system of a remote system using your local Windows Explorer program).
Overall, I think Keith's solution is actually the best, and the most straightforward.
I'm using PuTTY to connect to a zOS mainframe (running USS, IBM's Unix-compatible software) and I'd like to download a hex dump (using od) of a file without making a copy of it on the filesystem. Is there a way that I can save or pipe stdout (through PuTTY) directly to a file on my (Windows XP) client?
You could configure session logging for your PuTTY session, as per the documentation. Then just run od on the server, wait until it finishes, and then close your log file. You'll need to trim the cruft at the beginning and end (because it has your whole session), but you should end up with what you want.
Note that upon inspection that documentation link may be for an older version of PuTTY, so YMMV but I'm sure that more recent versions also support session logging.
If you were to install a command-line ssh tool (e.g., running OpenSSH under Cygwin), you could then do the standard "ssh hostname command > file" sort of redirection.
i am working on unix.
i want to write a shell script which will check for a file called "temp_file.txt" on windows
and then execute some commands.
is this possible?
how could we connect to the windows and go to a specific directory and check for a file?
Share the directory on the Windows machine using the "regular" Windows file sharing facilities. On the Linux side, you have two options:
Use smbclient to connect to the Windows machine and check if the file exists or
Use smbmount to mount the shared directory into your Linux file system and check file existence using "standard" Linux commands (e.g. test).
The exact implementation details will depend on the scripting language that you use, but your pseudo-code will look something like this:
loop:
check if file exists
if yes: do something useful
sleep for some reasonable time
(I am assuming that you want to execute the commands on the Linux machine.)
If you're using Linux (you specify that you're using Unix, but not what variant), check out the inotify API --- this will allow you to set up event responders for filesystem events (much more efficient than polling).
From a shell script, you can use the inofitywait command --- see http://linux.die.net/man/1/inotifywait for more information.
you could set up SSH on Windows and then write a script on Unix using the SSH client to connect to Windows and execute the command. The alternative, if you can afford to, it to write a windows batch, and execute your command on Windows itself. Or you can turn on Windows terminal services, and use telnet protocol from Unix to issue the command. Programming languages that support telnet includes Perl (Net::Telnet) and Python(telnetlib)
As ghostdog74 suggested, ssh is your best bet. You can run something like (I assume you have Cygwin or SFU installed)
ssh "[ -e $file ] && do_something.sh" > do_something.log
If your command logs to stdout, you get the log on your Linux box as well.
If you set up private key authentication, it gets even better.