I need to load two data files each of which have 6 columns and I want to plot the ratio of a column from first data file and another column from the second data file. But I keep getting a memory error. None of the numbers are zeros. I plotted the same in Excel. Worked ok. But I need it in Matlab, What do I do?
My current code is something like this:
load file1.dat;
y=file1(:,2);
time=file1(:,1);
hold on;
load file2.dat;
x=file2(:,5);
figure;
plot (t,y/(3*x),'LineWidth',1);
xlabel('Time (s)');
ylabel('Mitochondrial Calcium (um)');
This is the error I get:
Error using \
Out of memory. Type HELP MEMORY for your
options.
Error in plotCmyo (line 9) --> File name, line 9 is the one with the plot command
plot(t,y/x, 'LineWidth',1);
Use y./(3*x) to do element-wise division.
Note the operator: ./
Related
Summary
I need help getting NCO tools to be helpful. I'm running into the error
"One or more variable sizes violate format constraints"
... when trying to just view the list of variables in the file with:
ncdump -h isrm_v1.2.1.ncf
It seems odd to trip on this when I'm not asking for any large variables to be read ... just metadata. Are there any flags I should or could be passing to avoid this error?
Reprex
isrm_v1.2.1.ncf (165 GB) is available on Zenodo.
Details
I've just installed the NCO suite via brew --install nco --build-from-source on a Mac (I know, I know) running OS X 11.6.5. ncks --version says 5.0.6.
Tips appreciated. I've been trawling through the ncks docs for a couple of hours without much insight. A friend was able to slice the file on a different system running actual Linux, so I'm pretty sure my NCO install is to blame.
How can I dig in deeper to find the root cause? NCO tools don't seem very verbose. I understand there are different sub-formats of NetCDF (3, 4, ...) but I'm not even sure how to verify the version/format of the .nc file that I'm trying to access.
My larger goal is to be able to slice it, like ncks -v pNH4 -d layer,0 isrm_v1.2.1.ncf pNH4L0.nc, but if I can't even view metadata, I'm thinking I need to solve that first.
The more-verbose version of the error message, for the record, is:
HINT: NC_EVARSIZE errors occur when attempting to copy or aggregate input files together into an output file that exceeds the per-file capacity of the output file format, and when trying to copy, aggregate, or define individual variables that exceed the per-variable constraints of the output file format. The per-file limit of all netCDF formats is not less than 8 EiB on modern computers, so any NC_EVARSIZE error is almost certainly due to violating a per-variable limit. Relevant limits: netCDF3 NETCDF_CLASSIC format limits fixed variables to sizes smaller than 2^31 B = 2 GiB ~ 2.1 GB, and record variables to that size per record. A single variable may exceed this limit if and only if it is the last defined variable. netCDF3 NETCDF_64BIT_OFFSET format limits fixed variables to sizes smaller than 2^32 B = 4 GiB ~ 4.2 GB, and record variables to that size per record. Any number of variables may reach, though not exceed, this size for fixed variables, or this size per record for record variables. The netCDF3 NETCDF_64BIT_DATA and netCDF4 NETCDF4 formats have no variable size limitations of real-world import. If any variable in your dataset exceeds these limits, alter the output file to a format capacious enough, either netCDF3 classic with 64-bit offsets (with -6 or --64), to PnetCDF/CDF5 with 64-bit data (with -5), or to netCDF4 (with -4 or -7). For more details, see http://nco.sf.net/nco.html#fl_fmt
Tips appreciated!
ncdump is not an NCO program, so I can't help you there, except to say that printing metadata should not cause an error in this case, so try ncks -m in.nc instead of ncdump -h in.nc.
Nevertheless, the hyperslab problem you have experienced is most likely due to trying to shove too much data into a netCDF format that can't hold it. The generic solution to that is to write the data to a more capacious netCDF format:
Try either one of these commands:
ncks -5 -v pNH4 -d layer,0 isrm_v1.2.1.ncf pNH4L0.nc
ncks -7 -v pNH4 -d layer,0 isrm_v1.2.1.ncf pNH4L0.nc
Formats are documented here
In tensorflow the training from the scratch produced following 6 files:
events.out.tfevents.1503494436.06L7-BRM738
model.ckpt-22480.meta
checkpoint
model.ckpt-22480.data-00000-of-00001
model.ckpt-22480.index
graph.pbtxt
I would like to convert them (or only the needed ones) into one file graph.pb to be able to transfer it to my Android application.
I tried the script freeze_graph.py but it requires as an input already the input.pb file which I do not have. (I have only these 6 files mentioned before). How to proceed to get this one freezed_graph.pb file? I saw several threads but none was working for me.
You can use this simple script to do that. But you must specify the names of the output nodes.
import tensorflow as tf
meta_path = 'model.ckpt-22480.meta' # Your .meta file
output_node_names = ['output:0'] # Output nodes
with tf.Session() as sess:
# Restore the graph
saver = tf.train.import_meta_graph(meta_path)
# Load weights
saver.restore(sess,tf.train.latest_checkpoint('path/of/your/.meta/file'))
# Freeze the graph
frozen_graph_def = tf.graph_util.convert_variables_to_constants(
sess,
sess.graph_def,
output_node_names)
# Save the frozen graph
with open('output_graph.pb', 'wb') as f:
f.write(frozen_graph_def.SerializeToString())
If you don't know the name of the output node or nodes, there are two ways
You can explore the graph and find the name with Netron or with console summarize_graph utility.
You can use all the nodes as output ones as shown below.
output_node_names = [n.name for n in tf.get_default_graph().as_graph_def().node]
(Note that you have to put this line just before convert_variables_to_constants call.)
But I think it's unusual situation, because if you don't know the output node, you cannot use the graph actually.
As it may be helpful for others, I also answer here after the answer on github ;-).
I think you can try something like this (with the freeze_graph script in tensorflow/python/tools) :
python freeze_graph.py --input_graph=/path/to/graph.pbtxt --input_checkpoint=/path/to/model.ckpt-22480 --input_binary=false --output_graph=/path/to/frozen_graph.pb --output_node_names="the nodes that you want to output e.g. InceptionV3/Predictions/Reshape_1 for Inception V3 "
The important flag here is --input_binary=false as the file graph.pbtxt is in text format. I think it corresponds to the required graph.pb which is the equivalent in binary format.
Concerning the output_node_names, that's really confusing for me as I still have some problems on this part but you can use the summarize_graph script in tensorflow which can take the pb or the pbtxt as an input.
Regards,
Steph
I tried the freezed_graph.py script, but the output_node_name parameter is totally confusing. Job failed.
So I tried the other one: export_inference_graph.py.
And it worked as expected!
python -u /tfPath/models/object_detection/export_inference_graph.py \
--input_type=image_tensor \
--pipeline_config_path=/your/config/path/ssd_mobilenet_v1_pets.config \
--trained_checkpoint_prefix=/your/checkpoint/path/model.ckpt-50000 \
--output_directory=/output/path
The tensorflow installation package I used is from here:
https://github.com/tensorflow/models
First, use the following code to generate the graph.pb file.
with tf.Session() as sess:
# Restore the graph
_ = tf.train.import_meta_graph(args.input)
# save graph file
g = sess.graph
gdef = g.as_graph_def()
tf.train.write_graph(gdef, ".", args.output, True)
then, use summarize graph get the output node name.
Finally, use
python freeze_graph.py --input_graph=/path/to/graph.pbtxt --input_checkpoint=/path/to/model.ckpt-22480 --input_binary=false --output_graph=/path/to/frozen_graph.pb --output_node_names="the nodes that you want to output e.g. InceptionV3/Predictions/Reshape_1 for Inception V3 "
to generate the freeze graph.
I have a list of netcdf files that I am trying to concatenate along the time dimension.
I am attempting to use the steps outlined here, which seem simple enough. However, I am running into some errors (likely some small/stupid oversight on my part...)
When I try to first make time a record dimension, I am using the following command:
ncks -O --mk_rec_dmn time TiMREX_20080526_000001.nc test_out.nc
This, however, give me the following error:
ncks: invalid option -- '-'
It seems like this is just some simple syntax/typo error on my part, but try as I might I can' find anything wrong.
Just to be sure, when I run a ncdump -h on the file, it confirms that there is indeed a time dimension
ncdump -h TiMREX_20080526_000001.nc
netcdf TiMREX_20080526_000001 {
dimensions:
time = 1 ;
bounds = 2 ;
x0 = 300 ;
y0 = 300 ;
z0 = 40 ;
Additionally, if I try to skip this step and just go right to the ncrcat part...
ncrcat -O TiMREX_20080526_000001.nc TiMREX_20080526_000733.nc test_out.nc
I get the following error:
ncopen: filename "TiMREX_20080526_000001.nc": Not a netCDF file
Which is especially odd...I'm pretty confident it is indeed at netCDF file (I just ran ncdump on it after all, and have no problem viewing it with ncview...)
Any thoughts? What simple step am I embarrassingly missing?
This is a weird error as your command looks syntactically correct. To be sure, I copied it to my machine where it ran as expected, with no 'invalid option' error. Thus I am unable to reproduce the problem. Based on the error message you report, it seems as though you might (somehow) be using a character that the system does not understand as a dash. In other words, the error you report is what I would expect if ncks received a funky character that looks like a dash but is not really a dash. Maybe when you copy it to stackoverflow it gets converted to a dash, so it works for me (try copying your own command above back into your console). Make sure the dash character you type is the same as the minus sign on a normal keyboard, and something else. Some keyboard/character sets make characters that look similar to dashes but are not ASCII dashes. Good luck.
Before watch the mail list, I'm confused with the lack of "size" of symbol table in the Mach-o file. And I found the solution in source file posted in that E-Mail, which note that:
//Mach-O symbol table does have size in it
//so need to scan ahead to find symbol with next highest address.
But when I parse out the symbol table in a Mach-O file (I got the symbol table from the symtab_command and the following nlists) and trying to calculate the size of one global symbol as the same way, I was confused again when I compared the symbol table from the output of dwarfdump (dwarfdump -ae). The end address of the symbol in the symbol table from the dwarfdump is different from the result my program's output. Is there some problem with the symbol table I parsed out? Or is there some other way to work out it?
Some of the output from my program:
<start address> <section index> <method>
0x0006d030 1 ___arclite_objc_autoreleasePoolPop
0x0006d048 1 _patch_lazy_pointers
0x0006d1f0 1 ___arclite_objc_autoreleasePoolPush
The corresponding part of the output from dwarfdump:
0x0014a37b: [0x0006d030 - 0x0006d046) __arclite_objc_autoreleasePoolPop
0x0014a122: [0x0006d048 - 0x0006d1ee) patch_lazy_pointers
0x0014a3a0: [0x0006d1f0 - 0x0006d212) __arclite_objc_autoreleasePoolPush
So if I use the way in the "MachONormalizedFileToAtoms.cpp" to calculate the end address of the symbol (look ahead to find symbol with next highest address), the result must be different from the output of dwarfdump. And does anyone know how dwarfdump calculate it?
Thank you!
From the answer by Nick Kledzik:
The compiler often aligns functions to start at aligned address (e.g. 8 or 16 bytes). So, there is padding bytes (usually NOPs) after the end of a function and before the start of the next function.
dwarfdump has access to the debug info which does have size info for functions. So dwarfdump can show the size of a function without the alignment padding at the end. Whereas the linker just looks at the next symbol address. There is not much point in the linker digging through the debug info to get a function’s true size, because when writing the output, the linker has to align the next function which would just add back the pad bytes.
I hope that can help others who has the same confusion.
Hi I am new to scilab and don't have much mathematical background.
I have been following code for another example and am being shown error 10000 for the following code:
function [z]=f(x,y)
z=0.026*(1.0-(y/ym))*y;
endfunction;
ym=12000;
x0=1950;y0=2555;xn=5;h=10;
x=[x0:h:xn];
y=ode("rk",y0,x0,x,f);
disp("x y")
disp("--------")
disp([x'y']);
function z=fe(x)
z=ym/(1-(1-ym/y0)*e^(-k*(t-t0)));
endfunction;
xe=(x0:h/10:xn);
n=length(xe)
for i=1:n
ye(i)=fe(xe(i));
end;
plot (x,y,'ro',xe, ye,'-b');legend ('rk4','Exact',3);
xtitle('solving dy/dx=k(1-y/ym)y','x','y');
I have worked through several other error messages. I am lost and don't know if the problem is in the code or the way I set up the problem. The following is the current error message:
!--error 10000
plot: Wrong size for input argument #2: A non empty matrix expected.
at line 57 of function checkXYPair called by :
at line 235 of function plot called by :
plot (x,y,'ro',xe, ye,'-b');legend ('rk4','Exact',3);
at line 25 of exec file called by :
I would appreciate any help.
Thanks
Start by adding clear as first statement, this will erase all variables before running your function. In the above script you don't declare ye.
Also the statement x=[x0:h:xn]; is strange with those values for x0,h and xn. You are now trying to get a list of x-values starting at 1950 and with positive steps of 10 up until 5 is reached.
I would recommend to try each line and see if the outcome is as expected. You do not need to know everything about the code, but probaly x and y should contain at least some values. The error is telling you that it expected a non-empty matrix for argument 2. This is y, so essentially it is telling you y is empty.