#Bean
public IntegrationFlow acknowledgeFileFlows() {
return f -> f
.enrichHeaders(h -> h
.headerExpression("originalPayload", "payload.toString()")
.headerExpression(FileHeaders.REMOTE_DIRECTORY, "payload.toString()"))
.log(LoggingHandler.Level.INFO, "eu.xxx", "'acknowledgeFileFlows Header originalPayload=' + headers[originalPayload]")
.handle(Sftp.outboundGateway(sessionFactory, Command.LS.getCommand(), "payload")
.autoCreateDirectory(false)
.autoCreateLocalDirectory(false)
.charset("UTF-8")
.regexFileNameFilter("(?i)(FileAcK_CITI_PMT[a-zA-Z0-9_.-]+|TxnAck_CITI_PMT[a-zA-Z0-9_.-]+|TxnNak_CITI_PMT[a-zA-Z0-9_.-]+)")
.options(Option.NAME_ONLY, Option.RECURSIVE))
.split()
.log(LoggingHandler.Level.INFO, "eu.xxx", "'acknowledgeFileFlows LS Payload= ' + payload.toString()")
.enrichHeaders(h -> h
.headerExpression("startTime", "new java.util.Date()")
.headerExpression("originalRemoteFile", "payload.toString()")
.headerExpression(FileHeaders.REMOTE_FILE, "payload.toString()"))
.handle(Sftp.outboundGateway(sessionFactory, Command.GET.getCommand(), "headers['originalPayload'] + headers['file_remoteFile']")
.autoCreateLocalDirectory(false)
.charset("UTF-8")
.fileExistsMode(FileExistsMode.REPLACE)
.fileNameExpression("headers['file_remoteFile']")
.localDirectoryExpression(new FunctionExpression<Message<?>>(m -> {
IntegrationMessageHeaderAccessor accessor = new IntegrationMessageHeaderAccessor(m);
final String remoteFileName = (String) accessor.getHeader("file_remoteFile");
final String ackFileRootPath = applicationConfiguration.getAcknowledgeFileTargetUri();
if (remoteFileName.toUpperCase().contains("XXX")) {
return Paths.get(ackFileRootPath, "XXX", RCIV, ACK).toString();
} else if (remoteFileName.toUpperCase().contains("YYY")) {
return Paths.get(ackFileRootPath, "YYY", RCIV, ACK).toString();
} else {
String[] parts = remoteFileName.toUpperCase().split("ZZZ");
if (parts != null && parts.length >= 2) {
return Paths.get(ackFileRootPath, "ZZZ", RCIV, ACK).toString();
}
}
return ackFileRootPath;
}))
.localFilenameExpression("headers['file_remoteFile']")
.options(Option.PRESERVE_TIMESTAMP)
.remoteFileSeparator("/"))
.handle((payload, headers) -> {
File file = (File) payload;
file.setWritable(true, false);
file.setReadable(true, false);
return file;
})
.enrichHeaders(h -> h
.headerExpression("fileLength", "payload.length")
.headerExpression("localDirectory", "payload.toString()"))
.wireTap(w -> w
.handle((payload, headers) -> {
return transactionLog("XXX_TR_ACK_FILE",
(Date) headers.get("startTime"),
(String) headers.get("localDirectory"),
(String) headers.get("file_remoteFile"),
headers.get("fileLength"));
})
.handle(Jpa.outboundAdapter(sourceIntegrationEntityManagerFactory)
.entityClass(TranComLog.class)
.persistMode(PersistMode.PERSIST), e -> e.transactional()))
.aggregate()
.channel(new NullChannel());
}
Above is the file download flows configuration.
This flow takes the remote SFTP path as the initial parameter.
LS -> File name process to get the correct target folder -> GET -> Write some log in to the DB.
The remote SFTP path can contain multiple files. I have to process those files one by one due to the requirement. (That's why I added the split)
My issue is when any exception occurs this flow stopped. I have to restart the server.
I think the reason is there is no correct exception handling in my flow, especially between the split and aggregate.
Does the Sftp.outboundGateway process the files one by one? If yes, the split and aggregate can be removed. Am I right?
If not, should I keep the split and aggregate? how can I add the proper exception handling?
Using the below code snippet, we created a file in Android 10, in a sub-folder under getExternalFilesDir(). However, immediately after creation, if we try to take persistableUriPermission, it throws an exception "No such permission exists....".
We need that check to know if that file will be available for read later in a common utility, else we have to make a copy. Please let us know what we might be doing wrong and how to fix this. Appreciate your help.
ParcelFileDescriptor filePFD =
cxt.getContentResolver().openFileDescriptor(Uri.parse(pathFileToSend), "r");
FileDescriptor fd = filePFD.getFileDescriptor();
FileInputStream fIn = new FileInputStream(fd);
File fileBaseFolder = new File(Utils.GetRootDirectory().getAbsolutePath(), Utils.DESTINATION);
if (!fileBaseFolder.exists())
fileBaseFolder.mkdirs();
if (fileBaseFolder.exists()) {
File copyFile = new File(fileBaseFolder.getAbsolutePath(), nameOfFile);
FileOutputStream fOut = new FileOutputStream(copyFile);
byte data[] = new byte[4096];
long total = 0;
int count;
while ((count = fIn.read(data)) != -1) {
total += count;
fOut.write(data, 0, count);
}
fOut.close();
Uri copiedFileUri =
FileProvider.getUriForFile(cxt,
cxt.getString(R.string.file_provider_authority),
copyFile);
if (null != copiedFileUri)
{
try {
/*At this line, an exception is thrown - No persistable permissions exist.. */
cxt.getContentResolver().takePersistableUriPermission(copiedFileUri, Intent.FLAG_GRANT_READ_URI_PERMISSION);
} catch (Exception e) {
e.printStackTrace();
}
}
takePersistableUriPermission() is for Uri values that you get from the Storage Access Framework (e.g., ACTION_OPEN_DOCUMENT). It will not work for FileProvider. And, you do not need permissions to work with getExternalFilesDir() on Android 4.4 and higher.
I am calling ant.copy in a groovy script:
ant.copy(file:jdbcDriverPath, toFile:destJDBCJarFile,overwrite:true)
The call is failing with the exception below, although the file exists under the path.
The same code launched on Windows works. When launched on Unix with java jdk1.7.0_51, it is failing.
Machine details on Unix:
$ uname -a
SunOS 5.10 Generic_142910-17 i86pc i386 i86pc
$ isainfo -kv
64-bit amd64 kernel modules
Any ideas?
Exception in thread "main" : Warning: Could not find file /data/apps/packages/temp/jconn3-6.05_26312.jar to copy.
at org.apache.tools.ant.taskdefs.Copy.copySingleFile(Copy.java:639)
at org.apache.tools.ant.taskdefs.Copy.execute(Copy.java:455)
at org.apache.tools.ant.UnknownElement.execute(UnknownElement.java:292)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.apache.tools.ant.dispatch.DispatchUtils.execute(DispatchUtils.java:106)
at groovy.util.AntBuilder.performTask(AntBuilder.java:250)
at groovy.util.AntBuilder.nodeCompleted(AntBuilder.java:212)
at groovy.util.BuilderSupport.doInvokeMethod(BuilderSupport.java:147)
at groovy.util.AntBuilder.doInvokeMethod(AntBuilder.java:166)
at groovy.util.BuilderSupport.invokeMethod(BuilderSupport.java:64)
at org.codehaus.groovy.runtime.callsite.PogoMetaClassSite.call(PogoMetaClassSite.java:45)
at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:40)
at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:116)
at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:124)
Thanks
The copy task is failing while running the below snippet (from the source of Ant 1.9.4):
private void copySingleFile() {
// deal with the single file
if (file != null) {
if (file.exists()) {
if (destFile == null) {
destFile = new File(destDir, file.getName());
}
if (forceOverwrite || !destFile.exists()
|| (file.lastModified() - granularity
> destFile.lastModified())) {
fileCopyMap.put(file.getAbsolutePath(),
new String[] {destFile.getAbsolutePath()});
} else {
log(file + " omitted as " + destFile
+ " is up to date.", Project.MSG_VERBOSE);
}
} else {
String message = "Warning: Could not find file "
+ file.getAbsolutePath() + " to copy.";
if (!failonerror) {
if (!quiet) {
log(message, Project.MSG_ERR);
}
} else {
throw new BuildException(message);
}
}
}
}
From the message string, there is an additional space in the filename /data/apps/packages/temp/jconn3-6.05_26312.jar.
This can also be reproduced using the following:
java.io.File file = new java.io.File("some_file_that_exists ");
System.out.println(file.exists()); // true on Windows, false on SunOS
The file.exists() returns true on Windows (automatically trimmed), but false on SunOS.
I have following directory structure,
Dir1
|___Dir2
|___Dir3
|___Dir4
|___File1.gz
|___File2.gz
|___File3.gz
The subdirectories are just nested and donot contain any files
I am trying to use the following for recursing through a directory on HDFS.If its a directory I append /* to the path and addInputPath
arg[0] = "path/to/Dir1"; // given at command line
FileStatus fs = new FileStatus();
Path q = new Path(args[0]);
FileInputFormat.addInputPath(job,q);
Path p = new Path(q.toString()+"/*");
fs.setPath(p);
while(fs.isDirectory())
{
fs.setPath(new Path(p.toString()+"/*"));
FileInputFormat.addInputPath(job,fs.getPath());
}
But the code doesnt seem to go in the while loop and I get not a File Exception
Where is the if statement you are referring to?
Anyway, you may have a look at these utility methods which add all files within a directory to a job's input:
Utils:
public static Path[] getRecursivePaths(FileSystem fs, String basePath)
throws IOException, URISyntaxException {
List<Path> result = new ArrayList<Path>();
basePath = fs.getUri() + basePath;
FileStatus[] listStatus = fs.globStatus(new Path(basePath+"/*"));
for (FileStatus fstat : listStatus) {
readSubDirectory(fstat, basePath, fs, result);
}
return (Path[]) result.toArray(new Path[result.size()]);
}
private static void readSubDirectory(FileStatus fileStatus, String basePath,
FileSystem fs, List<Path> paths) throws IOException, URISyntaxException {
if (!fileStatus.isDir()) {
paths.add(fileStatus.getPath());
}
else {
String subPath = fileStatus.getPath().toString();
FileStatus[] listStatus = fs.globStatus(new Path(subPath + "/*"));
if (listStatus.length == 0) {
paths.add(fileStatus.getPath());
}
for (FileStatus fst : listStatus) {
readSubDirectory(fst, subPath, fs, paths);
}
}
}
Use it in your job runner class:
...
Path[] inputPaths = Utils.getRecursivePaths(fs, inputPath);
FileInputFormat.setInputPaths(job, inputPaths);
...
I want to copy a directory from one drive to another drive. My selected directory contains many sub directories and files.
How can I implement the same using Qt?
void copyPath(QString src, QString dst)
{
QDir dir(src);
if (! dir.exists())
return;
foreach (QString d, dir.entryList(QDir::Dirs | QDir::NoDotAndDotDot)) {
QString dst_path = dst + QDir::separator() + d;
dir.mkpath(dst_path);
copyPath(src+ QDir::separator() + d, dst_path);
}
foreach (QString f, dir.entryList(QDir::Files)) {
QFile::copy(src + QDir::separator() + f, dst + QDir::separator() + f);
}
}
Manually, you can do the next things:
1). with func below you generate folders/files list (recursively) - the destination files.
static void recurseAddDir(QDir d, QStringList & list) {
QStringList qsl = d.entryList(QDir::NoDotAndDotDot | QDir::Dirs | QDir::Files);
foreach (QString file, qsl) {
QFileInfo finfo(QString("%1/%2").arg(d.path()).arg(file));
if (finfo.isSymLink())
return;
if (finfo.isDir()) {
QString dirname = finfo.fileName();
QDir sd(finfo.filePath());
recurseAddDir(sd, list);
} else
list << QDir::toNativeSeparators(finfo.filePath());
}
}
2). then you may to start copying files from destination list to the new source directory like that:
for (int i = 0; i < gtdStringList.count(); i++) {
progressDialog.setValue(i);
progressDialog.setLabelText(tr("%1 Coping file number %2 of %3 ")
.arg((conf->isConsole) ? tr("Making copy of the Alta-GTD\n") : "")
.arg(i + 1)
.arg(gtdStringList.count()));
qApp->processEvents(QEventLoop::ExcludeUserInputEvents);
if (progressDialog.wasCanceled()) {
// removing tmp files/folders
rmDirectoryRecursive(tmpFolder);
rmDirectoryRecursive(tmpFolderPlus);
setEnableGUI(true);
return;
}
// coping
if (!QFile::copy(gtdStringList.at(i), tmpStringList.at(i))) {
if (warningFlag) {
QMessageBox box(this);
QString name = tr("Question");
QString file1 = getShortName(gtdStringList.at(i), QString("\\...\\"));
QString file2 = getShortName(tmpStringList.at(i), QString("\\...\\"));
QString text = tr("Cannot copy <b>%1</b> <p>to <b>%2</b>" \
"<p>This file will be ignored, just press <b>Yes</b> button" \
"<p>Press <b>YesToAll</b> button to ignore other warnings automatically..." \
"<p>Or press <b>Abort</b> to cancel operation").arg(file1).arg(file2);
box.setModal(true);
box.setWindowTitle(name);
box.setText(QString::fromLatin1("%1").arg(text));
box.setIcon(QMessageBox::Question);
box.setStandardButtons(QMessageBox::YesToAll | QMessageBox::Yes | QMessageBox::Abort);
switch (box.exec()) {
case (QMessageBox::YesToAll):
warningFlag = false;
break;
case (QMessageBox::Yes):
break;
case (QMessageBox::Abort):
rmDirectoryRecursive(tmpFolder);
rmDirectoryRecursive(tmpFolderPlus);
setEnableGUI(true);
return;
}
}
}
}
And that's all. Good luck!
I wanted something similar, and was googling (in vain), so this is where I've got to:
static bool cpDir(const QString &srcPath, const QString &dstPath)
{
rmDir(dstPath);
QDir parentDstDir(QFileInfo(dstPath).path());
if (!parentDstDir.mkdir(QFileInfo(dstPath).fileName()))
return false;
QDir srcDir(srcPath);
foreach(const QFileInfo &info, srcDir.entryInfoList(QDir::Dirs | QDir::Files | QDir::NoDotAndDotDot)) {
QString srcItemPath = srcPath + "/" + info.fileName();
QString dstItemPath = dstPath + "/" + info.fileName();
if (info.isDir()) {
if (!cpDir(srcItemPath, dstItemPath)) {
return false;
}
} else if (info.isFile()) {
if (!QFile::copy(srcItemPath, dstItemPath)) {
return false;
}
} else {
qDebug() << "Unhandled item" << info.filePath() << "in cpDir";
}
}
return true;
}
It uses an rmDir function that looks pretty similar:
static bool rmDir(const QString &dirPath)
{
QDir dir(dirPath);
if (!dir.exists())
return true;
foreach(const QFileInfo &info, dir.entryInfoList(QDir::Dirs | QDir::Files | QDir::NoDotAndDotDot)) {
if (info.isDir()) {
if (!rmDir(info.filePath()))
return false;
} else {
if (!dir.remove(info.fileName()))
return false;
}
}
QDir parentDir(QFileInfo(dirPath).path());
return parentDir.rmdir(QFileInfo(dirPath).fileName());
}
This doesn't handle links and special files, btw.
The hard way. Copy every file individually.
Use QDir::entryList() to iterate over the content of a directory
Use QDir::cd() and QDir::cdUp() to go in and out of directories
Use QDir::mkdir() and QDir::mkpath() to create the new folders tree
and finally, use QFile::copy() to copy the actual files
This is basically petch's answer with a slight change due to it breaking for me in Qt 5.6 (this is the top question hit), so all credit goes to petch.
function
bool copyPath(QString sourceDir, QString destinationDir, bool overWriteDirectory)
{
QDir originDirectory(sourceDir);
if (! originDirectory.exists())
{
return false;
}
QDir destinationDirectory(destinationDir);
if(destinationDirectory.exists() && !overWriteDirectory)
{
return false;
}
else if(destinationDirectory.exists() && overWriteDirectory)
{
destinationDirectory.removeRecursively();
}
originDirectory.mkpath(destinationDir);
foreach (QString directoryName, originDirectory.entryList(QDir::Dirs | \
QDir::NoDotAndDotDot))
{
QString destinationPath = destinationDir + "/" + directoryName;
originDirectory.mkpath(destinationPath);
copyPath(sourceDir + "/" + directoryName, destinationPath, overWriteDirectory);
}
foreach (QString fileName, originDirectory.entryList(QDir::Files))
{
QFile::copy(sourceDir + "/" + fileName, destinationDir + "/" + fileName);
}
/*! Possible race-condition mitigation? */
QDir finalDestination(destinationDir);
finalDestination.refresh();
if(finalDestination.exists())
{
return true;
}
return false;
}
Use:
/*! Overwrite existing directories. */
bool directoryCopied = copyPath(sourceDirectory, destinationDirectory, true);
/*! Do not overwrite existing directories. */
bool directoryCopied = copyPath(sourceDirectory, destinationDirectory, false);
Try this:
bool copyDirectoryFiles(const QString &fromDir, const QString &toDir, bool coverFileIfExist)
{
QDir sourceDir(fromDir);
QDir targetDir(toDir);
if(!targetDir.exists()){ /* if directory don't exists, build it */
if(!targetDir.mkdir(targetDir.absolutePath()))
return false;
}
QFileInfoList fileInfoList = sourceDir.entryInfoList();
foreach(QFileInfo fileInfo, fileInfoList){
if(fileInfo.fileName() == "." || fileInfo.fileName() == "..")
continue;
if(fileInfo.isDir()){ /* if it is directory, copy recursively*/
if(!copyDirectoryFiles(fileInfo.filePath(),
targetDir.filePath(fileInfo.fileName()),
coverFileIfExist))
return false;
}
else{ /* if coverFileIfExist == true, remove old file first */
if(coverFileIfExist && targetDir.exists(fileInfo.fileName())){
targetDir.remove(fileInfo.fileName());
}
// files copy
if(!QFile::copy(fileInfo.filePath(),
targetDir.filePath(fileInfo.fileName()))){
return false;
}
}
}
return true;
}
I have made a library to manipulate files by a shell command style API. It supports a recursively copy of files and handled several more conditions.
https://github.com/benlau/qtshell#cp
Example
cp("-a", ":/*", "/target"); // copy all files from qrc resource to target path recursively
cp("tmp.txt", "/tmp");
cp("*.txt", "/tmp");
cp("/tmp/123.txt", "456.txt");
cp("-va","src/*", "/tmp");
cp("-a", ":resource","/target");
Since I had some trouble with App-Bundles on macOS, here's a solution with QDirIterator
void copyAndReplaceFolderContents(const QString &fromDir, const QString &toDir, bool copyAndRemove = false) {
QDirIterator it(fromDir, QDirIterator::Subdirectories);
QDir dir(fromDir);
const int absSourcePathLength = dir.absoluteFilePath(fromDir).length();
while (it.hasNext()){
it.next();
const auto fileInfo = it.fileInfo();
if(!fileInfo.isHidden()) { //filters dot and dotdot
const QString subPathStructure = fileInfo.absoluteFilePath().mid(absSourcePathLength);
const QString constructedAbsolutePath = toDir + subPathStructure;
if(fileInfo.isDir()){
//Create directory in target folder
dir.mkpath(constructedAbsolutePath);
} else if(fileInfo.isFile()) {
//Copy File to target directory
//Remove file at target location, if it exists. Otherwise QFile::copy will fail
QFile::remove(constructedAbsolutePath);
QFile::copy(fileInfo.absoluteFilePath(), constructedAbsolutePath);
}
}
}
if(copyAndRemove)
dir.removeRecursively();
}
If you are on a linux based system and the cp command exists and can be run, then you can use a QProcess to launch a bash:
auto copy = new QProcess(this);
copy->start(QStringLiteral("cp -rv %1 %2").arg(sourceFolder, destinationFolder));
copy->waitForFinished();
copy->close();
cp details:
-r means recursively
-v means it prints the successfully copied file
Note: if the copy operation is long, then you need to managed the UI freezing, as noted here
Assuming the target location is empty "no existing files or folders
with the same names" and you have no problem to use a Recursive
function!! to copy a directory recursively then the code will be something like this
void copy_all(QString dst_loc, QString src_loc)
{
QDir src(src_loc);
for(QFileInfo file_info : src.entryInfoList(QDir::AllEntries | QDir::NoDotAndDotDot)){
if(file_info.isDir()){
src.mkpath(dst_loc+'/'+file_info.fileName());
copy_all(dst_loc+'/'+file_info.fileName(),file_info.absoluteFilePath());
}
QFile::copy(file_info.absoluteFilePath(), dst_loc+'/'+file_info.fileName());
}
}
if you ever dealt with tree data structures and tried to create a Recursive function to do a "depth-first search" Alogorithm you will get a 85% similar algorithm, which Actually I got this idea from.
The second way, is by using a map data structure to hold the current
fileInfoList in a directory, and the corresponding state to show if
you have used this fileInfo or not yet. and you gather all
information firstly about sub directories and files from the source location.
And this is how most OS and other file managers do to copy data, by showing you the size of files to be copied, how many files and folders are going to be copied, and finally, if there is any conflict of files or folders with the same name before you even initiate copying "if you will do the same algorithm with the destination so that you can match filenames".
Good luck!.