Resize HDF5 dataset in Julia - julia

Is there a way to resize a chunked dataset in HDF5 using Julia's HDF5.jl? I didn't see anything in the documentation. Looking through the source, all I found was set_dims!(), but that cannot extend a dataset (only shrink it). Does HDF5.jl have the ability to enlarge an existing (chunked) dataset? This is a very important feature for me, and I would rather not have to call into another language.

The docs have a brief mention of extendible dimensions in hdf5.md excerpted below.
You can use extendible dimensions,
d = d_create(parent, name, dtype, (dims, max_dims), "chunk", (chunk_dims), [lcpl, dcpl, dapl])
set_dims!(d, new_dims)
where dims is a tuple of integers. For example
b = d_create(fid, "b", Int, ((1000,),(-1,)), "chunk", (100,)) #-1 is equivalent to typemax(Hsize)
set_dims!(b, (10000,))
b[1:10000] = [1:10000]

I believe I've got it figured out. The issue is that I forgot to give the dataspace a large enough max_dims. Doing that required digging into the lower-level API. The solution I found was:
dspace = HDF5.dataspace((6,20)::Dims, max_dims=(6,typemax(Int64)))
dtype = HDF5.datatype(Float64)
dset = HDF5.d_create(prt, "trajectory", dtype, dspace, "chunk", (6,10))
Once I created a dataset that can be resized appropriately, the set_dims! function resizes the dataset correctly.
I think I located a few minor issues with the API, which I had to work around or change in my local version. I will get in touch with the HDF5.jl owner regarding those. For those interested:
The constant H5S_UNLIMITED is of type Uint64, but the dataspace function will only accept tuples of Int64, hence why I used typemax(Int64) for my max_dims to imitate how H5S_UNLIMITED is derived.
The form of d_create which I used calls h5d_create incorrectly; it passes parent instead of checkvalid(parent).id (can be seen by comparison with other forms of d_create).

Related

How to append / add layers to geopackages in PyQGIS

For a project I am creating different layers which should all be written into one geopackage.
I am using QGIS 3.16.1 and the Python console inside QGIS which runs on Python 3.7
I tried many things but cannot figure out how to do this. This is what I used so far.
vl = QgsVectorLayer("Point", "points1", "memory")
vl2 = QgsVectorLayer("Point", "points2", "memory")
pr = vl.dataProvider()
pr.addAttributes([QgsField("DayID", QVariant.Int), QgsField("distance", QVariant.Double)])
vl.updateFields()
f = QgsFeature()
for x in range(len(tag_temp)):
f.setGeometry(QgsGeometry.fromPointXY(QgsPointXY(lon[x],lat[x])))
f.setAttributes([dayID[x], distance[x]])
pr.addFeature(f)
vl.updateExtents()
# I'll do the same for vl2 but with other data
uri ="D:/Documents/QGIS/test.gpkg"
options = QgsVectorFileWriter.SaveVectorOptions()
context = QgsProject.instance().transformContext()
QgsVectorFileWriter.writeAsVectorFormatV2(vl1,uri,context,options)
QgsVectorFileWriter.writeAsVectorFormatV2(vl2,uri,context,options)
Problem is that the in the 'test.gpkg' a layer is created called 'test' and not 'points1' or 'points2'.
And the second QgsVectorFileWriter.writeAsVectorFormatV2() also overwrites the output of the first one instead of appending the layer into the existing geopackage.
I also tried to create single .geopackages and then use 'Package Layers' processing tool (processing.run("native:package") to merge all layers into one geopackage, but then the attributes types are all converted into strings unfortunately.
Any help is much appreciated. Many thanks in advance.
You need to change the SaveVectorOptions, in particular the mode of actionOnExistingFile after creating the gpkg file :
options = QgsVectorFileWriter.SaveVectorOptions()
#options.driverName = "GPKG"
options.layerName = v1.name()
QgsVectorFileWriter.writeAsVectorFormatV2(v1,uri,context,options)
#switch mode to append layer instead of overwriting the file
options.actionOnExistingFile = QgsVectorFileWriter.CreateOrOverwriteLayer
options.layerName = v2.name()
QgsVectorFileWriter.writeAsVectorFormatV2(v2,uri,context,options)
The documentation is here : SaveVectorOptions
I also tried to create single .geopackages and then use 'Package Layers' processing tool (processing.run("native:package") to merge all layers into one geopackage, but then the attributes types are all converted into strings unfortunately.
This is definitively the recommended way, please consider reporting the bug

Sage TypeError positive characteristics not allowed in symbolic computations

I am new to sage and have got a code (link to code) which should run.
I am still getting an error message in the decoding part. The error trace looks like this:
in decode(y)
--> sigma[i+1+1] = sigma[i+1]*(z)\
-(delta[i+1]/delta[mu+1])*z^(i-mu)*sigma[mu+1]*(z);
in sage.structure.element.Element.__mul__
if BOTH_ARE_ELEMNT(cl):
--> return coercion_model.bin_op(left, right, mul)
in sage.structure.coerce.CoercionModel_cache_maps.bin_op
--> action = self.get_action(xp,yp,op,x,y)
...... some more traces (don't actually know if they are important)
TypeError: positive characteristics not allowed in symbolic computations
Does anybody know if there is something wrong in this code snipped? Due to previous errors, I changed the following to get to where I am at the moment:
.coeffs() changed to .coefficients(sparse=False) due to a warning message.
in the code line sigma[i+1+1] = sigma[i+1](z)\
-(delta[i+1]/delta[mu+1])*z^(i-mu)*sigma[mu+1](z); where the error occurs, i needed to insert * eg. sigma[i+1]*(z)
I would be grateful for any guess what could be wrong!
Your issue is that you are multiplying things not of characteristic zero (like elements related to Phi.<x> = GF(2^m)) with elements of symbolic computation like z which you have explicitly defined as a symbolic variable
Phi.<x> = GF(2^m)
PR = PolynomialRing(Phi,'z')
z = var('z')
Basically, the z you get from PR is not the same one as from var('z'). I recommend naming it something else. You should be able to access this with PR.gen() or maybe PR(z).
I'd be able to be more detailed, but I encourage you next time to paste a fully (non-)working example; trying to slog through a big worksheet is not the easiest thing to track all this down. Finally, good luck, hope Sage ends up being useful for you!

Reading structured data from file

How can I read a structured data file in Julia?
Speaking in python, if I have defined some c-styled format:
df = [
['Label', '4s'],
['Version', 'i'],
['Revision', 'i'],
['Date', '28s'],
['FileFormat', '2i'],
['FileType', '4s'],
['OriginalFileName', '68s'], ...etc]
df is parsed out into var_names and structure and then read:
fh = open(fn,'rb')
data = fh.read()
header = struct.unpack_from(struct_format, data[0:structsize])
I can't figure out how to do something equivalent in Julia.
In a somewhat similar question, I see how I could build a type to handle this but have yet to figure out how a variable can have a specific amount of base types; 4 ints for example.
I've read all the documentation I can find on sprint and its cousins, but haven't found any examples to help make sense of the them.
It's possible that StrPack.jl will do some or all of what you want. I think the documentation emphasizes its use for in-memory objects, but its original purpose was to handle this problem. Might be worth browsing the source and perhaps contributing patches.

Unix write() function (libc)

I am making a C application in Unix that uses raw tty input.
I am calling write() to characters on the display, but I want to manipulate the cursor:
ssize_t
write(int d, const void *buf, size_t nbytes);
I've noticed that if buf has the value 8 (I mean char tmp = 8, then passing &tmp), it will move the cursor/pointer backward on the screen.
I was wondering where I could find all the codes, for example, I wish to move the cursor forward but I cannot seem to find it via Google.
Is there a page that lists all the code for the write() function please?
Thank you very much,
Jary
8 is just the ascii code for backspace. You can type man ascii and look at all the values (the man page on my Ubuntu box has friendlier names for the values). If you want to do more complicated things you may want to look at a library like ncurses.
You have just discovered that character code 8 is backspace (control-H).
You would probably be best off using the curses library to manage the screen. However, you can find out what control sequences curses knows about by using infocmp to decompile the terminfo entry for your terminal. The format isn't particularly easy to understand, but it is relatively comprehensive. The alternative is to find a manual for the terminal, which tends to be rather hard.
For instance, I'm using a color Xterm window; infocmp says:
# Reconstructed via infocmp from file: /usr/share/terminfo/78/xterm-color
xterm-color|nxterm|generic color xterm,
am, km, mir, msgr, xenl,
colors#8, cols#80, it#8, lines#24, ncv#, pairs#64,
acsc=``aaffggiijjkkllmmnnooppqqrrssttuuvvwwxxyyzz{{||}}~~,
bel=^G, bold=\E[1m, clear=\E[H\E[2J, cr=^M,
csr=\E[%i%p1%d;%p2%dr, cub=\E[%p1%dD, cub1=^H,
cud=\E[%p1%dB, cud1=^J, cuf=\E[%p1%dC, cuf1=\E[C,
cup=\E[%i%p1%d;%p2%dH, cuu=\E[%p1%dA, cuu1=\E[A,
dch=\E[%p1%dP, dch1=\E[P, dl=\E[%p1%dM, dl1=\E[M, ed=\E[J,
el=\E[K, enacs=\E)0, home=\E[H, ht=^I, hts=\EH, il=\E[%p1%dL,
il1=\E[L, ind=^J,
is2=\E[m\E[?7h\E[4l\E>\E7\E[r\E[?1;3;4;6l\E8, kbs=^H,
kcub1=\EOD, kcud1=\EOB, kcuf1=\EOC, kcuu1=\EOA,
kdch1=\E[3~, kf1=\E[11~, kf10=\E[21~, kf11=\E[23~,
kf12=\E[24~, kf13=\E[25~, kf14=\E[26~, kf15=\E[28~,
kf16=\E[29~, kf17=\E[31~, kf18=\E[32~, kf19=\E[33~,
kf2=\E[12~, kf20=\E[34~, kf3=\E[13~, kf4=\E[14~,
kf5=\E[15~, kf6=\E[17~, kf7=\E[18~, kf8=\E[19~, kf9=\E[20~,
kfnd=\E[1~, kich1=\E[2~, kmous=\E[M, knp=\E[6~, kpp=\E[5~,
kslt=\E[4~, meml=\El, memu=\Em, op=\E[m, rc=\E8, rev=\E[7m,
ri=\EM, rmacs=^O, rmcup=\E[2J\E[?47l\E8, rmir=\E[4l,
rmkx=\E[?1l\E>, rmso=\E[m, rmul=\E[m,
rs2=\E[m\E[?7h\E[4l\E>\E7\E[r\E[?1;3;4;6l\E8, sc=\E7,
setab=\E[4%p1%dm, setaf=\E[3%p1%dm, sgr0=\E[m, smacs=^N,
smcup=\E7\E[?47h, smir=\E[4h, smkx=\E[?1h\E=, smso=\E[7m,
smul=\E[4m, tbc=\E[3g, u6=\E[%i%d;%dR, u7=\E[6n,
u8=\E[?1;2c, u9=\E[c,
That contains information about box drawing characters, code sequences generated by function keys, various cursor movement sequences, and so on.
You can find out more about X/Open Curses (v4.2) in HTML. However, that is officially obsolete, superseded by X/Open Curses v7, which you can download for free in PDF.
If you're using write just so you have low-level cursor control, I think you are using the wrong tool for the job. There are command codes for many types of terminal. VT100 codes, for example, are sequences of the form "\x1b[...", but rather than sending raw codes, you'd be much better off using a library like ncurses.

How can you print an Ada.Calendar.Time variable in Ada?

Need to know how to print a Time variable in Ada. I assume there is no portable way because Time is implementation defined. I've already seen the GNAT.Calendar.Formatting package available under GNAT, I'd also be interested in a GHS for VME.
See package "Ada.Calendar.Formatting" function "Image" for Ada2005. If you have an Ada95 compiler you could and this package isn't available, try my implementation from here
This was written using GNAT 3.15p, so pretty old.
Sure, time output can be portable, Ada.Calendar contains standard functions that extract the components of a time value, so it's straightforward to put together your own conversion package.
For example, here's one. One just needs to either create a minor addition to create a "Formatted_Time" record for a given Time value (see the package's Get_Time() function for guidance), or make Main_Formatter() visible in the package spec.
Generally what I do is use Calendar.Split and then do a 'image on the parts I care about.
Here's an example that displays the date and time using the GNAT.Calendar.Time_IO package:
with ada.calendar;
with gnat.calendar.time_io;
procedure display_time is
begin
gnat.calendar.time_io.put_time(ada.calendar.clock, "Date: %Y/%m/%d Time: %H:%M:%S");
end display_time;
Date/time format options are available here:
https://en.wikibooks.org/wiki/Ada_Programming/Libraries/GNAT.Calendar.Time_IO

Resources