I am trying to use R to unzip password protected files from a drive without using 7-zip. My organisation doesn't have access to it, we use WinZip for everything.
I have searched far and wide here but cannot find a post that satisfies the question.
I have a file that is zipped and contains a single XML file. I need to automate the collation of this data, my thinking is unzip then read. I have found these that I can't see what I need to do:
Using unzip does not support passwords - unzip a .zip file
e.g. unzip(file.xml.zip) produces
Warning message: In unzip(zipfile = "file.xml.zip") : zip file is corrupt
And the file is not corrupt as I can manually unzip it fine afterwards.
Using 7-Zip (I can't access this) - Unzip a password protected file with Powershell
Reading without unzipping (get "error reading from the connection) - Extract files from password protected zip folder in R
read_xml(unz("file.xml", "file.xml.zip"))
produces
Error in open.connection(x, "rb") : cannot open the connection In addition: Warning message: In open.connection(x, "rb") : cannot open zip file 'file.xml'
I have tried looking at Expand-Archive in PowerShell and trying to call that through R but am not having much luck, please someone help me!
With PowerShell I use
Expand-Archive -Path 'file'
which produces:
Exception calling "ExtractToFile" with "3" argument(s): "The archive entry was compressed using an unsupported compression method."
I don't have WinZip, but since both it and unzip.exe (within Rtools-4.2) support password-encoding, then we should be able to use similar methods. (Or perhaps you can use unzip included with Rtools.)
Setup:
$ echo 'hello world' > file1.txt
$ echo -e 'a,b\n11,22' > file2.csv
$ c:/rtools42/usr/bin/zip.exe -P secretpassword files.zip file1.txt file2.txt
adding: file1.txt (stored 0%)
adding: file2.txt (stored 0%)
$ unzip -v files.zip
Archive: files.zip
Length Method Size Cmpr Date Time CRC-32 Name
-------- ------ ------- ---- ---------- ----- -------- ----
12 Stored 12 0% 2023-02-09 10:03 af083b2d file1.txt
10 Stored 10 0% 2023-02-09 10:03 1c1d572e file2.csv
-------- ------- --- -------
22 22 0% 2 files
$ unzip -c files.zip file1.txt
Archive: files.zip
[files.zip] file1.txt password:
Okay, now we have a password-protected zip file.
In R,
readLines(pipe("unzip -q -P secretpassword -c files.zip file1.txt"))
# [1] "hello world"
read.csv(pipe("unzip -q -P secretpassword -c files.zip file2.csv"))
# a b
# 1 11 22
WinZip does support a command-line interface, so we should be able to use it within pipe (or system or similar). It does support passwords, I believe it uses the -s argument instead of -P. I don't know if it supports extracting a file to stdout, so you might need to explore its command-line options for that, and if not then work out storing the document to a temporary directory.
Or, assuming you have Rtools installed, you can use its unzip as above without relying on WinZip.
Note:
Including the password as a command-line argument is relatively unsafe: users on the same host (if a multi-user system) can see the password in clear text by looking at the process list. I'm not certain if there's an easy way around this.
Related
We have a main Linux server, say M, where we have files like below (for 2 months, and new files arriving daily)
Folder1
PROCESS1_20211117.txt.gz
PROCESS1_20211118.txt.gz
..
..
PROCESS1_20220114.txt.gz
PROCESS1_20220115.txt.gz
We want to copy only the latest file on our processing server, say P.
So as of now, we were using the below command, on our processing server.
rsync --ignore-existing -azvh -rpgoDe ssh user#M:${TargetServerPath}/${PROCSS_NAME}_*txt.gz ${SourceServerPath}
This process worked fine until now, but from now, in the processing server, we can keep files only up to 3 days. However, in our main server, we can keep files for 2 months.
So when we remove older files from the processing server, the rsync command copies all files from main server to the processing server.
How can I change rsync command to copy only latest file from Main server?
*Note: the example above is only for one file. We have multiple files on which we have to use the same command. Hence we cannot hardcode any filename.
What I tried:
There are multiple solutions, but all seems to be when I want to copy latest file from the server I am running rsync on, not on the remote server.
Also I tried running below to get the latest file from main server, but I cannot pass variable to SSH in my company, as it is not allowed. So below command works if I pass individual path/file name, but cannot work as with variables.
ssh M 'ls -1 ${TargetServerPath}/${PROCSS_NAME}_*txt.gz|tail -1'
Would really appreciate any suggestions on how to implement this solution.
OS: Linux 3.10.0-1160.31.1.el7.x86_64
ssh quoting is confusing - to properly quote it, you have to double-quote it locally.
Handy printf %q trick is helpful - quote the relevant parts.
file=$(
ssh M "ls -1 $(printf "%q" "${getServerPath}/${PROCSS_NAME}")_*.txt.gz" |
tail -1
)
rsync --ignore-existing -azvh -rpgoDe ssh user#M:"$file" "${SourceServerPath}"
or maybe nicer to run tail -n1 on the remote, so that minimum amount of data are transferred (we only need one filename, not them all), invoke explicit shell and pass the variables as shell arguments:
file=$(ssh M "$(printf "%q " bash -c \
'ls -1 "$1"_*.txt.gz | tail -n1'
'_' "${TargetServerPath}/${PROCSS_NAME}"
)")
Overall, I recommend doing a function and using declare -f :
sshqfunc() { echo "bash -c $(printf "%q" "$(declare -f "$1"); $1 \"\$#\"")"; };
work() {
ls -1 "$1"_*txt.gz | tail -1
}
tmp=$(ssh M "$(sshqfunc work)" _ "${TargetServerPath}/${PROCSS_NAME}")
or you can also use the mighty declare to transfer variables to remote - then run your command inside single quotes:
ssh M "
$(declare -p TargetServerPath PROCSS_NAME);
"'
ls -1 ${TargetServerPath}/${PROCSS_NAME}_*txt.gz | tail -1
'
I was solving a unix problem on Katacoda. I am solving it as instructed but it gives me an error.
The problem is to display number of lines and words in a given input file
Write a command/logic, which will read the content from a given input file and display the number of lines and number of words in the file.
Your default logged in directory: /home/scrapbook/tutorial/
Input file location: /home/scrapbook/tutorial/unix_countproject/
Input file Name: input.txt
Script file location: /home/scrapbook/tutorial/
Script file Name: script.sh
Instructions:
You can view content of input file (input.txt) in the folder at /home/scrapbook/tutorial/unix_countproject/.
Note that this folder and file already exists and are located in the path mentioned above.
You can open the script.sh file using vi editor, Write the logic inside the file to display the number of lines and number of words in the input file (input.txt), After writing the logic , save the file and quit from vi editor .
To test your command/logic, Run the shell script file (script.sh) at the terminal using the below command
sh /home/scrapbook/tutorial/script.sh
If any issues while running the above command, Modify the script.sh file and repeat the point#:3
Don't use any echo statement in the script.sh file, even in the commented line/code
On completion of the task, click on SUMMARY or CONTINUE to proceed to next assignment.
Example : Contents of sample Input file named as input.txt - starting from following line
Hello all
Welcome to all of you
Expected Output:
2 7
As instructed, I store my logic in the script.sh file and it executes successfully and gives me the expected output but it's not accepting my answer.
$ cd unix_countproject
$ cat input.txt
Hi all
Welcome to Unix module
$ vi script.sh
Inside script.sh : wc -l -w input.txt|cut -c-6
(exit vi)
$ chmod +x script.sh
$ ./script.sh
2 6
Then I click on complete but it's not accepting it.
I'm trying to use the sph2pipe tool to convert the SPH files into wav or mp3 files. Although I have downloaded and installed the tool downloaded from here: https://www.ldc.upenn.edu/language-resources/tools/sphere-conversion-tools
still don't see any program that I can use..
On windows 10, after downloading sph2pipe and click the .exe file, a window just quickly popped up and never showed up again. And then I can't find any program called sph2pipe from the system and no command named sph2pipe either.
On Mac, I downloaded the program from where I forgot, but after clicked the executable file on mac, I got this document saying
Last login: Tue May 8 18:57:21 on ttys001
Pennys-MBP:~ me$ /Users/me/Downloads/SPH/sph2pipe_v2.5/sph2pipe ; exit;
Usage: sph2pipe [-h hdr] [-t|-s b:e] [-c 1|2] [-p|-u|-a] [-f typ] infile [outfile]
default conditions (for 'sph2pipe infile'):
input file contains sphere header
output full duration of input file
output all channels from input file
output same sample coding as input file
output format is WAV on Wintel machines, SPH elsewhere
output is written to stdout
optional controls (items bracketed separately above can be combined):
-h hdr -- treat infile as headerless, get sphere info from file 'hdr'
-t b:e -- output portion between b and e sec (floating point)
-s b:e -- output portion between b and e samples (integer)
-c 1 -- only output first channel
-c 2 -- only output second channel
-p -- force conversion to 16-bit linear pcm
-u -- force conversion to 8-bit ulaw
-a -- force conversion to 8-bit alaw
-f typ -- select alternate output header format 'typ'
five types: sph, raw, au, rif(wav), aif(mac)
logout
Saving session...
...copying shared history...
...saving history...truncating history files...
...completed.
[Process completed]
But still when try to type sph2pipe on my terminal, I got the response:
-bash: sph2pipe: command not found
Can somebody help me? I need to do the conversion very soon.
Thank you!
I figured it out:
sph2pipe.exe file file.wav
Hi everyone I'm not to ksh. What i'm trying to do is I'm writing a script to scp a(or many) zip file from a local directory to a remote host. Then get the script to ssh into the remote host to gunzip the files I just scp over. Is there any simple way to do this. I keep trying but once I ssh over to the remote host the rest of my commands no longer run like the cd /file/directory and then gzip -d /files etc.....
NB: don't confuse "zip" and "gzip", two different animals
This should work:
cd <local_directory>
# collect files names as $1 $2 ... $N
set -- *.gz # or use your own filter like "dumps*.gz"
# put source file a tar archive and send it as input to ssh
# then, on the other side, untar the file then decompress
tar cf - $* | ssh <user>#<remote_host> "cd <remote dir> && tar xf - && gunzip $*
Note: using "&&" instead of ";" to prevent "tar" command to be executed if "cd" fails for any reason
I'm trying to get terminal to upload a file for me, in this case: file.txt
Unfortunately, it won't work, no matter what I try.
#!/bin/bash
HOST=*
USER=*
PASS=*
# I'm 100% sure the host/user/pass are correct.
#Terminal also connects with the host provided
ftp -inv $HOST << EOF
user $USER $PASS
cd /Users/myname/Desktop
get file.txt #which is located on my desktop
bye
EOF
I've tried 100 different scripts but it just won't upload :(
This is the output after saving to an .sh file, chmod +x and sudo the .sh file:
Connected to *hostname*.
220 ProFTPD 1.3.4b Server ready.
331 Password required for *username*
230 User *username* logged in
Remote system type is UNIX.
Using binary mode to transfer files.
550 /Users/myname/Desktop: No such file or directory
local: file.txt remote: file.txt
229 Entering Extended Passive Mode (|||35098|)
550 file.txt: No such file or directory
221 Goodbye.
myname:Desktop Myname$
I've browsed through many other topics about the same issue here, but I just can't figure it out. I've started playing with UNIX since this morning, so excuse me for this (probably) foolish question.
Try:
#!/bin/bash
HOST=*
USER=*
PASS=*
# I'm 100% sure the host/user/pass are correct.
#Terminal also connects with the host provided
cd /Users/myname/Desktop # Go the the local folder where the file is located in
ftp -inv $HOST << EOF
user $USER $PASS
cd /User/$USER/Desktop # Go to the folder in which you want to upload the file
put file.txt #which is located on my desktop
bye
EOF
So use put and make sure your file is the current working directory and the remote directory exists.
You are using get but talk about an upload. Probably you just want to use put?
Anyway, I'm not sure this can be done using a basic ftp client. I'm always using ncftp for things like this. This comes with command line utilities like ncftpput which accept command line arguments and options to perform the task.
Alfe is right, you need to use put <filename> to upload a file to FTP. You can find a quick guide here. It should be possible using the basic FTP tool but I would also recommend ncftp :-)
You need to use put to upload a file.