Using JSch ChannelSftp: How to read multiple files with dynamic names? - sftp

I have to read a bunch of .CSV files with dynamic file names from a SFTP server. These files get generated every 15 minutes.
I am using JSch's ChannelSftp, but there is no method which would give the exact filenames. I only see an .ls() method. This gives a Vector e.g.
[drwxr-xr-x 2 2019 2019 144 Aug 9 22:29 .,
drwx------ 6 2019 2019 176 Aug 27 2009 ..,
-rw-r--r-- 1 2019 2019 121 Aug 9 21:03 data_task1_2011_TEST.csv,
-rw-r--r-- 1 2019 2019 121 Aug 9 20:57 data_task1_20110809210007.csv]
Is there a simple way to read all the files in a directory and copy them to another location?
This code works for copying a single file:
JSch jsch = new JSch();
session = jsch.getSession(SFTPUSER,SFTPHOST,SFTPPORT);
session.setPassword(SFTPPASS);
java.util.Properties config = new java.util.Properties();
config.put("StrictHostKeyChecking", "no");
session.setConfig(config);
session.connect();
channel = session.openChannel("sftp");
channel.connect();
channelSftp = (ChannelSftp)channel;
channelSftp.cd(SFTPWORKINGDIR);
channelSftp.get("data_task1_20110809210007.csv","data_task1_20110809210007.csv");

The ls method is the one you need. It returns a vector of LsEntry objects, each of which you can ask about its name.
So, after your channelSftp.cd(SFTPWORKINGDIR);, you could do the following:
Vector<ChannelSftp.LsEntry> list = channelSftp.ls("*.cvs");
for(ChannelSftp.LsEntry entry : list) {
channelSftp.get(entry.getFilename(), destinationPath + entry.getFilename());
}
(This assumes destinationPath is a local directory name ending with / (or \ in Windows).)
Of course, if you don't want to download the same files again after 15 minutes, you might want to have a list of the local files, to compare them (use a HashSet or similar), or delete them from the server.

Note that ls is case sensitive. This method retrieves all csv files, regardless of the extension case
ArrayList<String> list = new ArrayList<String>();
Vector<LsEntry> entries = sftpChannel.ls("*.*");
for (LsEntry entry : entries) {
if(entry.getFilename().toLowerCase().endsWith(".csv")) {
list.add(entry.getFilename());
}
}

Related

How to visualize a clog2 file from MPI/MPE using jumpshot

I am following the examples given in the “Using MPI” book and am working on the example that turns on MPE logging (pmatmatlog.c - ported it from the Fortran example). It runs and produces a log file called “pmatmat.log.clog2”. I would like to visualize this log file.
I started by trying to use “jumpshot-4”, because that is what is installed and I cannot find a way to download jumpshot-2 (which appears to favor log files in the clog2 format?). Jumpshot-4, however, wants files in slog2 format and gives an error about not finding “clog2TOslog2” in the TAU directory tree.
Looking at the head of the clog2 file, it appears to be correct as far as I can tell:
$ head pmatmat.log.clog2
CLOG-02.44is_big_endian=TRUE is_finalzed=TRUE block_size=65536num_buffered_blocks=128max_comm_world_size=4max_thread_count=1known_eventID_start=0user_eventID_start=600known_solo_eventID_start=-10user_solo_eventID_start=5000known_stateID_count=300user_stateID_count=4known_solo_eventID_count=0user_solo_eventID_count=0commtable_fptr=0107374182466560>? … <and on into a lot of unreadable binary>
When I try to convert my file clog2 file using the command line calling “clog2TOslog2”, I get the following:
$ Clog2ToSlog2 ./pmatmat.log.clog2
GUI_LIBDIR is set. GUI_LIBDIR = /Users/markrbower/mpi/lib
**** Error! State!=State
**** Error! State!=State
**** Error! State!=State
**** Error! State!=State
java.lang.reflect.InvocationTargetException
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:564)
at logformat.clog2TOdrawable.InputLog$ContentIterator.hasNext(InputLog.java:481)
at logformat.clog2TOdrawable.InputLog.peekNextKind(InputLog.java:58)
at logformat.slog2.output.Clog2ToSlog2.main(Clog2ToSlog2.java:77)
Caused by: logformat.clog2TOdrawable.NoMatchingEventException: No matching State end-event for Record RecHeader[ time=1.9073486328125E-5, icomm=0, rank=1, thread=0, rectype=6 ], RecCargo[ etype=1, bytes=bstartbendcomputecomputedpma ]
at logformat.clog2TOdrawable.Topo_State.matchFinalEvent(Topo_State.java:95)
... 7 more
java.lang.reflect.InvocationTargetException
After a lot of errors that all look to be “InvocationTargetException” as above, the tail says:
…
SLOG-2 Header:
version = SLOG 2.0.6
NumOfChildrenPerNode = 2
TreeLeafByteSize = 65536
MaxTreeDepth = 0
MaxBufferByteSize = 1346
Categories is FBinfo(157 # 1454)
MethodDefs is FBinfo(0 # 0)
LineIDMaps is FBinfo(232 # 1611)
TreeRoot is FBinfo(1346 # 108)
TreeDir is FBinfo(38 # 1843)
Annotations is FBinfo(0 # 0)
Postamble is FBinfo(0 # 0)
Number of Drawables = 20
Number of Unmatched Events = 0
Total ByteSize of the logfile = 8320
timeElapsed between 1 & 2 = 13 msec
timeElapsed between 2 & 3 = 79 msec
$
There are several routes I could see to finally get to visualizing the log file:
1. re-install TAU and hope that allows jumpshot-4 to find the converter
2. install MPI2 with MPE2 and try the newer version of everything
3. find a way to download and install Jumpshot-2 and hope that reads clog2 files
4. find some other way to convert clog2 to slog2
Why isn’t the conversion working and which is the best option to pursue?

Preserving recursive directory structure in zips with Java 8

I have the following directory on my laptop:
/tmp/
myapp/
assets/
config.yml
models/
troll.ply
tree.ply
textures/
troll-skin.png
tree-skin.png
I would like to zip /tmp/myapp/assets (and all its recursive contents) up into a ZIP named assets.zip, such that, when I unzip it (via unzip assets.zip), it preserves the directory structure under the assets folder. Hence, when unzipped, it would show config.yml in the "root" of the ZIP, and 2 directories inside the ZIP (models and textures). The rest of the files would be inside these respective subdirectories, etc.
When I run this code:
File sourceDir = new File("/tmp/myapp/assets");
ZipOutputStream zip = new ZipOutputStream(new FileOutputStream("/Users/myuser/archives/assets.zip"));
File[] contents = sourceDir.listFiles();
for(File file : contents) {
zip.putNextEntry(new ZipEntry(file.name));
InputStream isteam = new FileInputStream(file);
Files.copy(isteam, zip);
zip.closeEntry();
isteam.close();
}
zip.close();
The code correctly creates a ZIP at /Users/myuser/archives/assets.zip.
However, when I unzip it (unzip /Users/myuser/archives/assets.zip) and then run ls -al /Users/myuser/archives, my output is:
-rw-r--r-- 1 myuser 1754083733 492 Dec 30 14:14 assets.zip
-rw-r--r-- 1 myuser 1754083733 10 Dec 30 14:14 config.yml
-rw-r--r-- 1 myuser 1754083733 7 Dec 30 14:14 models
-rw-r--r-- 1 myuser 1754083733 9 Dec 30 14:14 textures
So both models and textures are being treated like files (not as directories). Furthermore, when I take a peek at the contents of the "models file", it appears that the contents of troll.ply and tree.ply have been concatenated inside of it, and ditto for the "tree file" with the 2 PNGs.
How can I tweak this so that directory structure (no matter how deep/nested) is always preserved in the resultant ZIP?
you can probably use the recursive method call to preserve the sub directories structure:
private static void addDir(File sourceDir, ZipOutputStream zip) throws IOException {
File[] contents = sourceDir.listFiles();
for(File file : contents) {
if(file.isDirectory()){
addDir(file, zip);
} else {
zip.putNextEntry(new ZipEntry(file.getAbsolutePath().replace("/tmp/myapp/","")));
System.out.println("file name " + file.getAbsolutePath().replace("/tmp/myapp/",""));
Path rn_demo = Paths.get(String.valueOf(file));
Files.copy(rn_demo, zip);
}
}
zip.closeEntry();
}
and you call in main method as below:
public static void main(String[] args) throws IOException {
File sourceDir = new File("/tmp/myapp/assets");
ZipOutputStream zip = new ZipOutputStream(new FileOutputStream("/Users/myuser/archives/assets.zip"));
addDir(sourceDir, zip);
zip.close();
}
Zipping through Java Zip seems to work differently on different OS's. I had the issue that it was working fine on Windows 7. But on Linux (RHEL6) the files were before the folders. This caused tests to fail.
A way to solve it is to sort the files and folders. folders first and then files. So the..
File[] contents = sourceDir.listFiles();
...File array should be sorted via path. Create a List<> from the Files and sort.
Collections.sort(newFiles, (a, b) ->
b.getAbsolutePath().compareTo(a.getAbsolutePath())
);
Note, I created an InputFile object to store the absolute path of the file.

Show changes/diffs without a parent commit (first commit) with JGit

when I do a git show commit of the first commit in a repository, I see all the files along with the diffs of the files (i.e all the lines being added)
$ git show cb5d132
commit cb5d13286cf9d14782f0e10445456dfe41072f55
Author: tw2 tw2LastName <tw2>
Date: Thu Oct 23 05:15:09 2014 -0400
Initial Commit
diff --git a/README.txt b/README.txt
new file mode 100644
index 0000000..96d156e
--- /dev/null
+++ b/README.txt
## -0,0 +1 ##
+First Line in README file!
\ No newline at end of file
The jgit show doesn't seem to get similar info, How can I get similar output using JGit API? I could use DiffFormatter, but it seems to require the baseTree and commitTree, wondering how I can get the DiffFormatter to work on the first commit in the repository.
ByteArrayOutputStream os = new ByteArrayOutputStream();
DiffFormatter df = new DiffFormatter( os )
RawTextComparator cmp = RawTextComparator.DEFAULT;
df.setRepository(repository);
df.setDiffComparator(cmp);
df.setDetectRenames(true);
// wondering how to use the API if we do not have baseCommit.
List<DiffEntry> diffEntries = df.scan(??baseCommitTree??, firstCommit.getTree());
for (DiffEntry diffEntry : diffEntries) {
df.format(diffEntry);
}
System.out.println(df.toString());
Use an o.e.jgit.treewalk.EmptyTreeIterator to compare the first commit against:
AbstractTreeIterator oldTreeIter = new EmptyTreeIterator();
ObjectReader reader = repository.newObjectReader();
AbstractTreeIterator newTreeIter = new CanonicalTreeParser(null, reader, firstCommit.getTree());
List<DiffEntry> diffEntries = df.scan(oldTreeIter, newTreeIter);
...

How to load CSV data into Excel 2013 using https with basic authentication?

To do so, I wrote the following VBA script and bound it a button which sets some parameters and loads the data. Afterwards it removes the connection to avoid a pile of stale connections.
Private Sub LoadData_Click()
Dim ConnString As String
Dim URL As String
URL = "https://some-server/some/path/some-file.csv"
ConnString = "TEXT;" + URL + "?partner_code=" + CStr(Range("B2")) + "&from=" + CStr(Range("B3")) + "&to=" + CStr(Range("B4"))
Sheets("Import").Select
Sheets("Import").UsedRange.Clear
ActiveSheet.Range("A1").Value = "Data is being loaded. Hold on..."
With ActiveSheet.QueryTables.Add(Connection:=ConnString, Destination:=ActiveSheet.Range("A1"))
.Name = _
" "
.FieldNames = True
.RowNumbers = False
.FillAdjacentFormulas = False
.PreserveFormatting = True
.RefreshOnFileOpen = False
.RefreshStyle = xlOverwriteCells
.SavePassword = True
.SaveData = True
.AdjustColumnWidth = True
.RefreshPeriod = 0
.TextFilePromptOnRefresh = False
.TextFilePlatform = xlWindows
.TextFileStartRow = 1
.TextFileParseType = xlDelimited
.TextFileTextQualifier = xlTextQualifierDoubleQuote
.TextFileConsecutiveDelimiter = False
.TextFileTabDelimiter = False
.TextFileSemicolonDelimiter = True
.TextFileCommaDelimiter = False
.TextFileSpaceDelimiter = False
.TextFileColumnDataTypes = Array(1)
.TextFileDecimalSeparator = ","
.TextFileTrailingMinusNumbers = True
.Refresh BackgroundQuery:=False
End With
Do While ActiveWorkbook.Connections.Count > 0
ActiveWorkbook.Connections.Item(ActiveWorkbook.Connections.Count).Delete
Loop
Sheets("Parameter").Select
End Sub
It works fine with Excel 2010, but not with 2013. When clicking my button I'm prompted for credentials and enter them. With Excel 2010 I see on server side in apache logs some superfluous OPTIONS requests and a HEAD, but finally just one GET which pulls the data.
With Excel 2013 I see three OPTIONS, then a HEAD which is successful (200) and finally a GET which is not successful (401). To further investigate it I did activate mod_dumpio module to see what's happening. And Apache is of course right to turn down excel 2013:
HEAD /some/path/some-file.csv?partner_code=25010&from=01.01.2014&to=01.06.2014 HTTP/1.1\r\n
...
Authorization: Basic Y29udHJvbGxpbmc6Rm9vNWFoUGg=\r\n
which is responded with 200 and directly afterwards a GET appears
GET /some/path/some-file.csv?partner_code=25010&from=01.01.2014&to=01.06.2014 HTTP/1.1\r\n
which lacks the basic auth header and therefore is answered by
HTTP/1.1 401 Unauthorized\r\nDate: Thu, 03 Jul 2014 18:18:59 GMT\r\nServer: Apache\r\nPragma: No-cache\r\nCache-Control: no-cache\r\nExpires: Thu, 01 Jan 1970 01:00:00 CET\r\nWWW-Authenticate: Basic realm="Import/Export"\r\nVary: Accept-Encoding\r\nContent-Encoding: gzip\r\nContent-Length: 378\r\nConnection: close\r\nContent-Type: text/html;charset=utf-8\r\n\r\n
I was following instructions in http://support.microsoft.com/kb/2123563/en-us step by step, but it did not help. Also the HEAD is actually working and using Basic auth.
How to load CSV data into Excel 2013 using https with basic authentication?

Oracle dataguard is not working properly because a FAL[client]

I had two databases, a primary and secondary and a configured dataguard between both, i restarted the suse linux but when starting up databases, the replication is not being done, i think i took the wrong way to startup the standby database....no it is just mounted, i have a gap not posibble to be detected at standby gap table, and a 'no FAL server specified' problem at standby database, what could be wrong?
From Primary:
System parameters with non-default values:
processes = 1200
nls_date_format = "MM/DD/YYYY HH24:MI:SS"
memory_target = 8000M
memory_max_target = 8G
control_files = "/oracle/app/oradata/ora11g/control01.ctl"
control_files = "/oracle/app/oradata/ora11g/control02.ctl"
control_files = "/oracle/app/oradata/ora11g/control03.ctl"
db_block_size = 8192
compatible = "11.1.0.0.0"
log_archive_start = TRUE
log_archive_dest_1 = "LOCATION=/home/oracle/archive"
log_archive_format = "%t_%s_%r.dbf"
db_recovery_file_dest = "/oracle/app/flash_recovery_area"
db_recovery_file_dest_size= 2G
undo_tablespace = "UNDOTBS1"
sec_case_sensitive_logon = FALSE
remote_login_passwordfile= "EXCLUSIVE"
db_domain = ""
dispatchers = "(PROTOCOL=TCP) (SERVICE=ora11gXDB)"
local_listener = ""
remote_listener = ""
session_cached_cursors = 450
cursor_sharing = "FORCE"
audit_file_dest = "/oracle/app/admin/ora11g/adump"
audit_trail = "NONE"
db_name = "ora11g"
open_cursors = 300
diagnostic_dest = "/oracle/app"
From standby database alert log:
Thu Feb 13 17:16:02 2014
Starting ORACLE instance (normal)
LICENSE_MAX_SESSION = 0
LICENSE_SESSIONS_WARNING = 0
Picked latch-free SCN scheme 3
Autotune of undo retention is turned on.
IMODE=BR
ILAT =145
LICENSE_MAX_USERS = 0
SYS auditing is disabled
Starting up ORACLE RDBMS Version: 11.1.0.7.0.
Using parameter settings in server-side spfile
/oracle/app/product/11g/db/dbs/spfileora11g.ora
System parameters with non-default values:
processes = 1200
nls_date_format = "MM/DD/YYYY HH24:MI:SS"
memory_target = 8000M
memory_max_target = 8G
control_files = "/oracle/app/oradata/ora11g/control01.ctl"
control_files = "/oracle/app/oradata/ora11g/control02.ctl"
control_files = "/oracle/app/oradata/ora11g/control03.ctl"
db_block_size = 8192
compatible = "11.1.0.0.0"
log_archive_start = TRUE
log_archive_dest_1 = "LOCATION=/home/oracle/archive"
log_archive_format = "%t_%s_%r.dbf"
db_recovery_file_dest = "/oracle/app/flash_recovery_area"
db_recovery_file_dest_size= 2G
undo_tablespace = "UNDOTBS1"
sec_case_sensitive_logon = FALSE
remote_login_passwordfile= "EXCLUSIVE"
db_domain = ""
dispatchers = "(PROTOCOL=TCP) (SERVICE=ora11gXDB)"
local_listener = ""
remote_listener = ""
session_cached_cursors = 450
cursor_sharing = "FORCE"
audit_file_dest = "/oracle/app/admin/ora11g/adump"
audit_trail = "NONE"
db_name = "ora11g"
open_cursors = 300
diagnostic_dest = "/oracle/app"
Deprecated system parameters with specified values:
log_archive_start
End of deprecated system parameter listing
Thu Feb 13 17:16:04 2014
.
.
.
starting up 1 dispatcher(s) for network address '(ADDRESS=(PARTIAL=YES) (PROTOCOL=TCP))'...
Thu Feb 13 17:16:04 2014
MMNL started with pid=15, OS id=10039
starting up 1 shared server(s) ...
ORACLE_BASE from environment = /oracle/app
Thu Feb 13 17:16:04 2014
ALTER DATABASE MOUNT
Setting recovery target incarnation to 2
ARCH: STARTING ARCH PROCESSES
Thu Feb 13 17:16:09 2014
ARC0 started with pid=19, OS id=10272
Thu Feb 13 17:16:09 2014
ARC1 started with pid=20, OS id=10274
Thu Feb 13 17:16:09 2014
ARC2 started with pid=21, OS id=10276
ARC0: Archival started
ARC1: Archival started
ARC2: Archival started
Thu Feb 13 17:16:09 2014
ARC3 started with pid=22, OS id=10278
ARC3: Archival started
ARCH: STARTING ARCH PROCESSES COMPLETE
ARC0: Becoming the 'no FAL' ARCH
ARC0: Becoming the 'no SRL' ARCH
ARC0: Thread not mounted
ARC1: Becoming the heartbeat ARCH
ARC2: Thread not mounted
ARC1: Thread not mounted
ARC3: Thread not mounted
Successful mount of redo thread 1, with mount id 4235628820
Physical Standby Database mounted.
Lost write protection disabled
Completed: ALTER DATABASE MOUNT
FAL[client]: Error fetching gap sequence, no FAL server specified
Primary
SQL> select max(sequence#) from v$log_history;
MAX(SEQUENCE#)
--------------
1606
SQL> SELECT name FROM v$archived_log WHERE thread# = 1 AND dest_id = 1 AND sequence# BETWEEN 1591 and 1606;
/home/oracle/archive/1_1606_792822090.dbf
16 rows selected.
SQL> SELECT GROUP#, BYTES FROM V$LOG;
GROUP# BYTES
---------- ----------
1 52428800
2 52428800
3 52428800
Secondary
SQL> select max(sequence#) from v$log_history;
MAX(SEQUENCE#)
--------------
1591
SQL>select process, thread#, sequence#, status from v$managed_standby where process='MRP0';
no rows selected
SQL> SELECT GROUP#, BYTES FROM V$STANDBY_LOG;
no rows selected
You need to set to parameters in the init file or spfile(sqlplus)
In the Primary database :
FAL_SERVER='standby_database'
FAL_CLIENT='primary_database'
In the standby database :
FAL_SERVER='primary_database'
FAL_CLIENT='standby_database'
These two parameters are needed for fetching archived log files (FAL mean Fetch ArchiveLog).
Hope that i help you.
Kind
This question is 5 years old but I feel it hasn't been completely answered yet.
First how does oracle resolves the gap:
The MRP process is the one triggering the request for a GAP.
That process is turned on like so:
Without standby redo logs:
alter database recover managed standby database disconnect;
With standby redo:
alter database recover managed standby database using current logfile disconnect;
If you have only one standby database for your primary database, then fal_server and fal_client parameters actually don't need to be configured.
If the fal_server is missing, oracle will grab that information from log_archive_dest_n.
This means that log_archive_dest_2 needs to be configured also on the standby database.
So how to solve a GAP resolution issue:
Make sure log_archive_dest_n is setup in both primary and standby
Make sure that there's no typo in the "service" value of
log_archive_dest_n.
Make sure that the service value references a valid tns entry in tnsnames.ora
Make sure that the same password file is used on all nodes of your primary and standby cluster.
Make sure you can connect with sqlplus "sys/syspassword#primary as sysdba" and sqlplus "sys/syspassword#standby as sysdba" from both primary and standby.List item
The MRP process sends a GAP resolution requests every so often. If you want to get it immediately in order to make sure it works:
SQL>alter database recover managed standby database cancel;
SQL>alter database recover managed standby database using current logfile;
(use standby redo logs, it applies logs faster on standby)
fal_server and fal_client parameters actually exists if you want to setup a cascaded standby setup.
Primary DB A sends archived logs to standby B
If standby B becomes primary then send archived logs to standby C.
FAL behavior on 11.2 (Doc ID 1394472.1)
From 11.2 no need to mention FAL_CLIENT primary will take it from
log_archive_dest_n (remote destination standby from where it received
the FAL request) service.
FAL_SERVER And FAL_CLIENT Settings For Cascaded Standby (Doc ID 358767.1)

Resources