Related
I want to provide support to convert single-page and multi-page tiff files into PDFs. There is an executable in Bit Miracle's LibTiff.NET called Tiff2Pdf.
How do I use Tiff2Pdf in my application to convert tiff data stream (not a file) into a pdf data stream (not a file)?
I do not know if there is an API exposed because the documentation only lists Tiff2Pdf as a tool. I also do not see any examples in the examples folder using it in a programmatic way to determine if it can handle data streams or how to use it in my own program.
libtiff tools expect a filename so the background run shown below is simply from upper right X.tif to various destinations, first is default
tiff2pdf x.tif
and we can see it writes a tiff2pdf file stream to console (Standard Output) however it failed in memory without a directory to write to. However on second run we can redirect
tiff2pdf x.tif > a.pdf
or alternately specify a destination
tiff2pdf -o b.pdf x.tif
So in order to use those tools we need a File System to receive the file objects, The destination folder/file directory can be a Memory File System drive or folder.
Thus you need to initiate that first.
NuGet is a package manager simply bundling the lib and as I don't use .net your a bit out on a limb as BitMiricle are not offering free support (hence point you at Stack Overflow, a very common tech support PLOY, Pass Liability Over Yonder) however looking at https://github.com/BitMiracle/libtiff.net/tree/master/Samples
they suggest memory in some file names such as https://github.com/BitMiracle/libtiff.net/tree/master/Samples/ConvertToSingleStripInMemory , perhaps get more ideas there?
I'm trying to write a TIFF file in MPI code. Different processors have different parts of the image, and I want to write the image to the file in parallel.
The write fails, only the 1st processor can write to it.
How do I do this?
There is no error in my implementation, just it does not work.
I used h=TIFFOpen(file, "a+") on each processor to open the same file (I am not sure whether this is a right way or not), then each processor who is responsible for a directory will write the header at its own place using TIFFSetDirectory(h, directorynumber), then the content of each directory will be written. I will finalize with TIFFWriteDirectory(h). The result would be the first directory which is written on the file.
I thought that I need to open the file using MPI_IO but doing this way it is not TIFFOpen?
Different MPI tasks are independent programs, running on independent hosts from the OS point of view. In your case the TIFF library is not designed to handle parallel operations, so opening the file will lead the first process to succeed, all the rest to fail because they found the file already opened (on a shared filesystem).
Except in case you are dealing with huge images (eg: astronomical images) where it's important for performance to perform parallel I/O (you need a filesystem supporting it however... I am aware of IBM GPFS), I would avoid to write a custom TIFF driver with MPI_IO.
Instead the typical solution is to gather (MPI_Gather()) the image parts on the process with rank==0 and let it only save the tiff file.
I would like to read and write from a single netCDF-4 file from R. The file will be accessed by many nodes processes at the same time, (~100's for development, ~1000's for production).
What is the best way to access parallel I/O features in netCDF from within R?
What I have found:
It appears from the Unidata page that all I need to do is compile with parallel features enabled (--enable-parallel). Is that really all I need to do?
I can not find any mention of parallel io in the ncdf4 package description.
Given that I/O is the bottleneck in my computation, any hints on how to optimize my computation - are there circumstances when it would be better to write to multiple files during computation (e.g., locally), and combine the files later (e.g. using nco)?
Information related to using parallel I/O with Unidata NetCDF may be found here:
https://www.unidata.ucar.edu/software/netcdf/docs/parallel_io.html
The --enable-parallel flag is no longer necessary when configuring netCDF; It will check the documentation and update it if need be. The flag is necessary when building the hdf5 library, however.
In order to use parallel I/O with netCDF-4, you need to make sure that it was built against an hdf5 library with parallel I/O enabled. At configure time, netCDF will query the hdf5 library to see whether or not the parallel I/O symbols are present.
If they are, parallel I/O for netCDF-4 is assumed.
If they are not, parallel I/O for netCDF-4 files is turned off.
If you are installing the netCDF library yourself, you can specify the --enable-parallel-tests flag when configuring; when you run make check, parallel tests will be run. You can also scan the output in config.log to see if parallel I/O functionality was found in the hdf5 library; there should be a message notifying you whether or not it was enabled.
Note that there are some limitations to Parallel I/O with netCDF-4, specifically:
NetCDF-4 provides access to HDF5 parallel I/O features for netCDF-4/HDF5 files. NetCDF classic and 64-bit offset format may not be opened or created for use with parallel I/O. (They may be opened and created, but parallel I/O is not available.)
Assuming that the underlying netCDF library has parallel I/O enabled, and you are operating on the correct type of file, the standard API call invoked by ncdf4 should leverage parallel I/O automatically.
There is one more R package dedicated for parallel handling of NetCDF files which is called pbdNCDF4.
This solution is based on standard ncdf4 package, so the syntax is very similar to the "traditional" approach. Further information available on CRAN: https://cran.r-project.org/web/packages/pbdNCDF4/vignettes/pbdNCDF4-guide.pdf
Ward gave a fine answer. I wanted to add that there is another way to get parallel I/O features out of Unidata NetCDF-4.
NetCDF-4 has an architecture that separates the API from the back end storage implementation. Commonly, that's the NetCDF API on an HDF5 back end. But, here's the neat thing: you can also have the NetCDF API on the Northwestern/Argonne "Parallel-NetCDF" (http://cucis.ece.northwestern.edu/projects/PnetCDF/ and http://www.mcs.anl.gov/parallel-netcdf) back end.
This approach will give you a parallel I/O method to classic and 64-bit offset formatted datasets.
Both Ward and Rob gave fine answers! ;-)
But there is yet another way to get parallel I/O on classic and 64-bit offset files, through the standard netCDF API.
When netCDF is built with --enable-pnetcdf, then the parallel-netcdf library is used, behind the scenes, to perform parallel I/O on classic, 64-bit offset, and CDF5 (though I have not tested that last format with parallel I/O).
When opening the file, use the NC_PNETCDF flag to the mode to indicate that you want to use parallel I/O for that file.
I'm trying to get familiar with a large project, possibly, initially written in Allegro Common Lisp. I have come across this piece of code:
(load "epilog:lib;compile.lisp")
Could please anyone explain what does it mean? Perhaps, if that helps, "epolig" is the name of a package "lib;compile.lisp" is a file "lib/compile.lisp", or so I could understand.
Is this a standard way to do something? And if so, what was the intention of this code? SBCL doesn't recognize colon as a special character in file name, i.e. it reports Couldn't load "epilog:lib;compile.lisp": file does not exist.
Logical Pathnames are a standard Common Lisp feature
It's not a symbol, it is a logical pathname.
Common Lisp has a portable logical pathname facility. The purpose is to abstract from physical pathnames like /usr/local/lisp/src/epilog/lib/compile.lisp or lispm:>sources>epilog>lib>compile.lisp.432 or any other type of pathname (just think of the differences between Unix, Mac OS X, Windows, ...).
The purpose is to use one single pathname scheme and one single logical file organization for your software. Regardless on what machine you are and where your files are, all you need is a mapping from the real file organization into the logical Lisp organization.
History
This facility came from a time when there were lots of different operating system and many different files (DEC VMS, IBM MVS, Multics, Unix, Lisp Machines, MS DOS, Macs, ...). The Lisp Machines were networked and could talk to all kinds of computers - so they learned the native file syntax for all those. In different laboratories (MIT, Xerox, SRI, ...) there were different machines on the network and different file servers. But the Lisp users wanted to load epilog:src;load.lisp and not remember where the stuff really is: on the local machine? but where? On a file server? But where? So on each network there was a registry for the translations from real file locations to logical pathnames.
So this is like an early 'URIs' facility for files - Uniform Resource Identifiers'.
The example explained
"epilog:lib;compile.lisp" is the name of a logical pathname.
epilog is the name of the logical host
lib; is the directory path
compile is the file name
lisp is the file type
Logical Pathname Translations
What you need is a translation between logical pathnames and physical pathnames:
Let's say we have a logical host EPILOG with just one translation rule. All files are on this machine for this Lisp under /usr/local/sources/epilog/. So we use some Unix conventions.
CL-USER 40 > (setf (logical-pathname-translations "EPILOG")
`(("**;*.*" "/usr/local/sources/epilog/**/*.*")))
(("**;*.*" "/usr/local/sources/epilog/**/*.*"))
Above only has one translation rule:
From EPILOG:**;*.* to /usr/local/sources/epilog/**/*.*.
It maps the logical hosts and all its subdirectories to a directory in a UNIX file system.
One could have more rules:
the documentation might be in a different place
there might be data files on a larger file system
compiled fasl files might be stored somewhere else
it might use logical subdirectories from other physical directories
But, again, here we use only one translation rule.
the example explained - part 2
Now we can parse a logical pathname:
CL-USER 41 > (pathname "epilog:lib;compile.lisp")
#P"EPILOG:LIB;COMPILE.LISP"
Let's describe it:
CL-USER 42 > (describe *)
#P"EPILOG:LIB;COMPILE.LISP" is a LOGICAL-PATHNAME
HOST "EPILOG"
DEVICE :UNSPECIFIC
DIRECTORY (:ABSOLUTE "LIB")
NAME "COMPILE"
TYPE "LISP"
VERSION NIL
As you see above, the parts have been parsed from our string.
Now we can also see how a logical pathname translates into a real pathname:
Translate a Logical Pathname to a physical pathname
CL-USER 43 > (translate-logical-pathname "epilog:code;ui;demo.lisp")
#P"/usr/local/sources/epilog/code/ui/demo.lisp"
So, now when you call (load "epilog:lib;compile.lisp"), then Lisp will translate the logical pathname and then really load the file from the translated physical pathname. What we also really want is that the Lisp for all purposes remembers the logical pathname - not the physical one. For example, when the file has a function named FOO, we want that Lisp records the location of the source of the function - but using the logical pathname. This way you can move a compiled file, a compiled application or a Lisp image to a different computer, update the translations and immediately it will be able to locate the source of FOO - if it is available on that machine or somewhere on a network accessible to that machine.
Logical Pathnames need to have a translation
To work with a logical pathname one needs to have a logical pathname translation like above. Often they are stored in a translations file by themselves. Define the translation, load it and then you can use corresponding logical pathnames to compile and load files. A typical software system using them, thus needs a corresponding translation. Sometimes it needs to be edited according to your file path, but sometimes they can be computed while loading the translations file. You'd to look where and how the logical host and the translations are defined.
History part 2
On a Symbolics Lisp Machine there is a site-wide directory, where systems and logical pathnames can be registered. Loading a system can then look up the system definition using this central directory and it also usually load a translations file. Thus the mechanism tells you what the structure of the system is (files, versions, patches, system versions, ...) and it tells you where it is located (which can be scattered around over several hosts or file systems).
Logical pathnames are not much used in newer software - you will encounter them sometimes in certain older software and especially those which were running on Lisp Machines - where this feature was extensively used throughout the system.
What are the differences between a Program, an Executable, and a Process?
In simple words -
Program: Program is a set of instructions which is in human readable format.(HelloWorld.c)
Executable: Executable is a compiled form of a Program (HelloWorld.exe file)
Process: Process is the executable being run by OS. The one you see in Task Manager or Task List (HelloWord.exe Process when we double click it.)
A Program or Computer Program essentially provides a sequence instructions (or algorithms if you rather) to the operating system or computer. These computer programs come in an executable form that the Operating System recognizes and can use to directly execute the instructions.
Essentially, an Executable is a file in a format that the computer can directly execute as opposed to source files which cannot be directly executed and must first be compiled. An executable is the result of a compilation. I mentioned that the operating system recognizes executable, it does so via the extension. A common extension used for windows executable files is .exe.
Once an executable has been executed a process begins. A process is simply an instance of a computer program. You can think of a process as the execution of the instructions contained in a computer program. When you view the Task Manager on a Windows computer you can see all of the current processes. Processes own resources such as virtual memory, operating system descriptions (handles, data sources, sinks etc), security attributes and various other elements required to process effectively.
A process is basically a program in execution. Associated with each process is its address space, a list of memory locations from 0 to some maximum, which the process can read and write. The address space contains the executable program, the program’s data, and its stack. Also associated with each process is a set of resources, commonly including registers (including the program counter and stack pointer), a list of open files, out- standing alarms, lists of related processes, and all the other information needed to run the program. A process is fundamentally a container that holds all the information needed to run a program, which is a set of instructions defined by a user/developer.
A program is a set of instruction and a passive entity.Program is a part of process while a process is running state of the program and it is a unit of work in a system.
Program: It is a passive entity, like the contents of a file stored on the Hard disk. In other words, It is just like another text file on your disk. Mostly, it will be in human readable format (ex: .java file).
Executable: It is again a passive entity. It is just another file on the disk which is derived by compiling the Program. So, it is a machine readable version of the Program file (ex: .class file.). Please note that it is still sitting out there on disk with not being executed currently.
Process: It is the active part of the Program/Executable. A Program/Executable loaded into memory(RAM) and executing is called as Process. A Process consists of set of instructions. CPU executes these instructions one by one.(ex: JVM loads your .class file and gives instructions to the CPU).
Also you can have two processes executing the same Program/Executable.
A program is a collection of source files in some high level language that you write to do some
function, for example, C++ files that implement sorting lists. An executable is the file that the compiler
creates from these source files containing machine instructs that can execute on the CPU. A process is the
active execution of the executable on the CPU and in the memory. It includes the memory management
information, the current PC, SP, HP, registers, etc.
Process is a part of a program. Process is the part where logic of that particular program exsists.
Program is given as a set of process. In some cases we may divide a problem into number of parts. At these times we write a seperate logic for each part known as process.
Consider it like this.
A program is a blueprint. Like a blueprint for a building. There is no building, but an abstraction of how a building would look like.
Process is the actual construction of Building which is built according to the blueprint.
While constructing a Building, there are many things happening at the same time. You are preparing the concrete, building multiple rooms at the same time, Laying the electrical cables etc. These would be Threads.
No difference. Remember, there is no spoon.
Program is a static entity but process is a dinamic entity.
Program is nothing but the contained in a file.Where process is a program in execution.
3.Program does not use the CPU resister set but process use the CPU resister set to store the intermediate and final result.