FileStatus use to recurse directory - recursion

Question

I have following directory structure,
Dir1
|___Dir2
|___Dir3
|___Dir4
|___File1.gz
|___File2.gz
|___File3.gz
The subdirectories are just nested and donot contain any files
I am trying to use the following for recursing through a directory on HDFS.If its a directory I append /* to the path and addInputPath
arg[0] = "path/to/Dir1"; // given at command line
FileStatus fs = new FileStatus();
Path q = new Path(args[0]);
FileInputFormat.addInputPath(job,q);
Path p = new Path(q.toString()+"/*");
fs.setPath(p);
while(fs.isDirectory())
{
fs.setPath(new Path(p.toString()+"/*"));
FileInputFormat.addInputPath(job,fs.getPath());
}
But the code doesnt seem to go in the while loop and I get not a File Exception

Answer 1

Where is the if statement you are referring to?
Anyway, you may have a look at these utility methods which add all files within a directory to a job's input:
Utils:
public static Path[] getRecursivePaths(FileSystem fs, String basePath)
throws IOException, URISyntaxException {
List<Path> result = new ArrayList<Path>();
basePath = fs.getUri() + basePath;
FileStatus[] listStatus = fs.globStatus(new Path(basePath+"/*"));
for (FileStatus fstat : listStatus) {
readSubDirectory(fstat, basePath, fs, result);
}
return (Path[]) result.toArray(new Path[result.size()]);
}
private static void readSubDirectory(FileStatus fileStatus, String basePath,
FileSystem fs, List<Path> paths) throws IOException, URISyntaxException {
if (!fileStatus.isDir()) {
paths.add(fileStatus.getPath());
}
else {
String subPath = fileStatus.getPath().toString();
FileStatus[] listStatus = fs.globStatus(new Path(subPath + "/*"));
if (listStatus.length == 0) {
paths.add(fileStatus.getPath());
}
for (FileStatus fst : listStatus) {
readSubDirectory(fst, subPath, fs, paths);
}
}
}
Use it in your job runner class:
...
Path[] inputPaths = Utils.getRecursivePaths(fs, inputPath);
FileInputFormat.setInputPaths(job, inputPaths);
...