I have several files in a unix directory that I have to move to Hadoop. I know the copyFromLocal command:
Usage: hadoop fs -copyFromLocal URI but that allows me to
move one by one.
Is there any way to move all those files to the HDFS in one command?
I want to know if there is a way to transfer several files at once
put command will work
if you want to copy whole directory from local to hdfs
hadoop fs -put /path1/file1 /pathx/target/
if you want to copy all files from directory to hdfs in one go
hadoop fs -put /path1/file1/* /pathx/target/
The put command supports multiple sources
Copy single src, or multiple srcs from local file system to the destination file system
Related
I'm trying to read a csv file but i get : No such file or directory.
the file is on tmp folder.
This is the commands:
Your file is not at hdfs:///user/hdfs/titles.csv, and this is what the error is saying.
You are only showing ls, not hdfs dfs -ls, so you should be using just cat titles.csv
If you want to read a file from HDFS, you need to hdfs dfs -put titles.csv /user/hdfs/ first. (And create the user directory using hdfs dfs -mkdir -p /user/hdfs if it doesn't already exist)
Trying to copy files from hdfs to local machine using copyToLocal with the following command:
Hadoop fs -copyToLocal remote path(hdfs file) destinationPath (my local path)
But I am getting the following error:
No such file or directory: error
Please help me with this.
You can copy the data from hdfs to the local filesystem by following two ways:
bin/hadoop fs -get /hdfs/source/path /localfs/destination/path
bin/hadoop fs -copyToLocal /hdfs/source/path /localfs/destination/path
Another alternative way would be:
Download the file from hdfs to the local filesystem. Just, point your web browser to HDFS WEBUI(namenode_machine:50070) and select the file and download it.
I want to download only new files from one SFTP server using WinSCP.
Suppose, I have 10 files in source and destination today.
Tomorrow one new file may be added to the source. In this scenario, I want to copy the new file only into destination.
I am using below script:
open sftp_connection
cd /path
option transfer binary
get "*.txt" localpath
close
exit
By using above, I am able to copy all files, but I want only new files which are not available in destination.
Thanks,
Srikrishna.
The easiest solution is to add -neweronly switch to your get command:
get -neweronly *.txt C:\local\path\*
For a very similar results, you can also use synchronize command:
synchronize local . C:\local\path -filemask=*.txt
See also WinSCP article on Downloading the most recent file.
I am trying to zip file which is in the format of Amazon*.xls in unix and also remove the source file after compression.Below is the used command
zip -m Amazon`date +%Y-%m-%d:%H:%M:%S`.zip Amazon*.xls
For the above command i am getting below error
zip I/O error: No such file or directory
zip error: Could not create output file Amazon.zip
PS: GZIP is working fine. I need zip format files.
It is not the zip, it is how your shell deals with expanding/substituting variables. Two lines solution for bash
export mydate=`date +%Y-%m-%d:%H:%M:%S`
zip -m Amazon_$mydate.zip *matrix*
Execute by hand (few secs difference) or better put in a shell script myzipper.sh and just source it.
Use '-p' instead of '-m', if zip files are to be extracted on Windows OS.
export mydate=date +%Y-%m-%d:%H:%M:%S
zip -p Amazon_$mydate.zip matrix
I have a single directory containing multiple zip files that contain .jpg files.
I need to unzip all files and save all contents (.jpgs files) into a single folder.
Any suggestions on a unix command that does that?
Please note that some of the contents (jpgs) might exist with same name in multiple zipped files, I need to keep all jpgs.
thanks
unzip '*.zip' -o -B
as by default these utilities are not installed
see http://www.cyberciti.biz/tips/how-can-i-zipping-and-unzipping-files-under-linux.html regarding installation
read about -B flag to realize its limitations.