I have a netCDF4 datafile where the time variable is stored as a float (netCDF: 'f8', numpy: float64) and I need to change it to a 32bit int (netCDF: 'i4', numpy: int32). I have tried making the change in python
tds.variables['time'][:] = np.int32(tds.variables['time'][:])
but this hasn't worked. What is the best way to make this change?
Since you tagged the question with nco, I assume a solution with nco is also acceptable.. This can be done with ncap2 (example with a NetCDF file that I had lying around):
ncdump -h drycblles.default.0000000.nc`:
gives:
netcdf drycblles.default.0000000 {
dimensions:
z = 128 ;
zh = 129 ;
t = UNLIMITED ; // (37 currently)
variables:
double t(t) ;
t:units = "s" ;
t:long_name = "Time" ;
.....
Same dump (of modified file) after:
ncap2 -s 't=int(t)' drycblles.default.0000000.nc drycblles.default.0000000_2.nc
gives:
int t(t) ;
t:long_name = "Time" ;
t:units = "s" ;
What you are trying in Python won't work since you cast the data of the variable time to int, but still store it as a float (you don't change the variable type in the NetCDF file). I don't see any options to change the data type in place, I guess you could copy the variable time to another name, create a new variable time with type int, copy the data, and remove the old time variable.
Related
In my program I am outputting a .csv file which exceeds 1000000 lines. Currently after the file is exported, I am splitting the file from linux using the below commands. However, I would like to know if we can split the files using a progress code. If so, could someone plese let me know on how to do it.
Below is the linux command I use manually.
ls -l xa*
split -1000000 filename.csv
mv xaa filename-01.csv
mv xab filename-02.csv
Without any code to work with I invented some code outputting to different files. You will have to work with OUTPUT TO and set new filenames.
This example will output 1050 lines split in files of 100 lines each.
DEFINE VARIABLE iLinesToOutput AS INTEGER NO-UNDO INIT 1050.
DEFINE VARIABLE iSplitAt AS INTEGER NO-UNDO INIT 100.
DEFINE VARIABLE iLine AS INTEGER NO-UNDO.
DEFINE VARIABLE cFile AS CHARACTER NO-UNDO.
DEFINE VARIABLE iFile AS INTEGER NO-UNDO.
DEFINE VARIABLE iOpen AS INTEGER NO-UNDO.
DEFINE STREAM str.
DO iLine = 1 TO iLinesToOutput:
// Open a new stream/file
IF (iLine - 1 ) MOD iSplitAt = 0 THEN DO:
iFile = iFile + 1.
cFile = "c:\temp\file-" + STRING(iFile, "999") + ".txt".
OUTPUT STREAM str TO VALUE(cFile).
EXPORT STREAM str DELIMITER "," "Customer ID" "Order Number" "Contact" "Count"
END.
// Output some data
PUT STREAM str UNFORMATTED "Line " iLine SKIP.
// Close the stream/file
IF iLine MOD iSplitAt = 0 THEN DO:
OUTPUT STREAM str CLOSE.
END.
END.
/* Close last file if not exactly right number of lines */
/* This could also be checked but close twice doesn't really matter */
OUTPUT STREAM str CLOSE.
I'm trying to implement a Conditional GAN in NiftyNet.
I have a collection of 3D images in Nifti format, each of a specific class. The class should be the conditioning input for the GAN.
So far I have managed to make it work, but it's kind of a hack: I create a 1x1x1 dummy image corresponding to each 3D image, which contains the class as its single value. Then, in my config file (... stand for omissions):
[images]
csv_file = index.csv ; This contains the subject_id to path correspondence
...
[labels]
csv_file = labels.csv ; This contains the subject_id to path to the dummy image correspondence
interp_order = 0
spatial_window_size = (1)
...
[SYSTEM]
...
dataset_split_file = fold0.csv
...
[GAN]
image=images
conditioning=labels
...
Although it technically works, it's very flimsy. I would like to specify the class with a CSV file with columns subject_id and target.
I've tried the following, using the undocumented csv_data_file parameter in the input description:
[images]
csv_file = index.csv ; This contains the subject_id to path correspondence
...
[labels]
csv_data_file = ./modality_labels.csv
to_ohe = False
...
[SYSTEM]
...
dataset_split_file = fold0.csv
...
[GAN]
image=images
conditioning=labels
...
but I get the following error:
CRITICAL:niftynet: Reader required input section name [labels], but in the filename list the column is empty.
CRITICAL:niftynet: file_list parameter should be a pandas.DataFrame instance and has input section name [labels] as a column name.
CRITICAL:niftynet: Reader requires section(s): ['images', 'labels']
CRITICAL:niftynet: Configuration input sections are: ['subject_id', 'images']
What would be the correct way to specify it?
Using the built-in "Import Data..." functionality we can import a properly formatted text file (like CSV and/or tab-delimited) as an image. It is rather straight forward to write a script to do so. However, my scripting approach is not efficient - which requires me to loop through each raw (use the "StreamReadTextLine" function) so it takes a while to get a 512x512 image imported.
Is there a better way or an "undocumented" script function that I can tap in?
DigitalMicrograph offers an import functionality via the File/Import Data... menu entry, which will give you this dialog:
The functionality evoked by this dialog can also be accessed by script commands, with the command
BasicImage ImageImportTextData( String img_name, ScriptObject stream, Number data_type_enum, ScriptObject img_size, Boolean lines_are_rows, Boolean size_by_counting )
As with the dialog, one has to pre-specify a few things.
The data type of the image.
This is a number. You can find out which number belongs to which image data type by, f.e., creating an image outputting its data type:
image img := Realimage( "", 4, 100 )
Result("\n" + img.ImageGetDataType() )
The file stream object
This object describes where the data is stored. The F1 help-documention explains how one creates a file-stream from an existing file, but essentially you need to specify a path to the file, then open the file for reading (which gives you a handle), and then using the fileHandle to create the stream object.
string path = "C:\\test.txt"
number fRef = OpenFileForReading( path )
object fStream = NewStreamFromFileReference( fRef, 1 )
The image size object
This is a specific script object you need to allocate. It wraps image size information. In case of auto-detecting the size from the text, you don't need to specify the actual size, but you still need the object.
object imgSizeObj = Alloc("ImageData_ImageDataSize")
imgSizeObj.SetNumDimensions(2) // Not needed for counting!
imgSizeObj.SetDimensionSize(0,10) // Not used for counting
imgSizeObj.SetDimensionSize(1,10) // Not used for counting
Boolean checks
Like with the checkboxes in the UI, you spefic two conditions:
Lines are Rows
Get Size By Counting
Note, that the "counting" flag is only used if "Lines are Rows" is also true. Same as with the dialog.
The following script improrts a text file with couting:
image ImportTextByCounting( string path, number DataType )
{
number fRef = OpenFileForReading( path )
object fStream = NewStreamFromFileReference( fRef, 1 )
number bLinesAreRows = 1
number bSizeByCount = 1
bSizeByCount *= bLinesAreRows // Only valid together!
object imgSizeObj = Alloc("ImageData_ImageDataSize")
image img := ImageImportTextData( "Imag Name ", fStream, DataType, imgSizeObj, bLinesAreRows, bSizeByCount )
return img
}
string path = "C:\\test.txt"
number kREAL4_DATA = 2
image img := ImportTextByCounting( path, kREAL4_DATA )
img.ShowImage()
I have an autoexec file that encrypts my password when I'm connecting to different servers....the code looks as follows:
%global wspwd ewspwd hpwd ehpwd ;
/* Enter WORKSTATION Password Below */
filename ewspwdfl "/home/&sysuserid./ewspwd.txt" ;
proc pwencode in=’XXXXXXXX’ out=ewspwdfl ; run ;
data _null_ ;
infile ewspwdfl obs=1 length=l ;
input # ;
input #1 line1 $varying1024. l ;
call symput('ewspwd',cats(substr(line1,1,l))) ;
call symput('wspwd',cats(‘XXXXXXXX’)) ;
run ;
My question is: why is
input # ;
included and why
input #1 line1 $varying1024. l ;
doesn't suffice.
Whenever I have created datasets with SAS I have never had to include "input #;" in my statement. I just simply write something along the lines of:
input #1 firstname $ #15 lastname $ #30 date mmddyy6.;
You don't need it for that data step. You could simplify it to.
data _null_ ;
infile ewspwdfl obs=1 TRUNCOVER ;
input line1 $CHAR1024. ;
call symputX('ewspwd',line1);
call symputX('wspwd',‘XXXXXXXX’) ;
run ;
Using input # is a good way to create a program where you want to read different lines using different input statements. You could test the content of the _infile_ variable and execute different parts of the data step based on what is read.
It is also useful when using the EOV= option on the INFILE statement to detect when you are starting to read from a new file, since it is not set until you begin reading the new file. So the INPUT # gets SAS to begin reading so that the EOV variable is set, but keeps the line waiting for your real INPUT statement to read later.
The #1 is useful if you want to read the same columns over again into different variables. For example you might want to read the first few characters as a string and then test them and based on what you find re-read as a number or a date.
I'm fairly new to tcl/tk and i'm trying to create a stripchart plot that allows me to set the time frame for x-axis example:
Link to pic of a time graph
I'm able to do it with dates, like this:
set s [::Plotchart::createXYPlot .c [list [clock scan 2006-01-01] [clock scan 2007-01-01] ""] {0.0 100.0 20.0} \
-xlabels {2006-01-01 2006-06-01 2007-01-01}]
but when i try to enter a time instead of two dates it's not working.
Thanks
When using clock scan, you can parse dates and times quite easily (and the result has precision to the level of seconds). However, it is best to specify the format that you expect the time to be in, and to pick a fixed instant as baseline if you are just parsing times. When such things become common, it's best to write a helper procedure:
# This is just for parsing time-of-day, *NOT DATES*.
# Handles both HH:MM and HH:MM:SS...
proc timestamp {timeString} {
if {[scan $timeString %d:%d:%d dummy dummy dummy] == 3} {
clock scan $timeString -format %H:%M:%S -base 0 -gmt 1
} else {
clock scan $timeString -format %H:%M -base 0 -gmt 1
}
}
# This should be recognizably similar
set s [::Plotchart::createXYPlot .c \
[list [timestamp 2:40] [timestamp 7:20] [timestamp 12:00:19]] \
{0.0 100.0 20.0} \
-xlabels {2:40 7:20 12:00:19}]
If course, if you've got both times and dates then you can do a better job by picking different format(s).