How to add all arrays from all .fits files in a directory? - idl

I have a directory on my computer with several .fits files that I am trying to work with in IDL. For some reason I am unable to add all of the arrays together correctly, as I'm getting negative numbers in the totaled array when all of the individual elements in all files are positive.
In IDL, I tried the following code:
flatsfiles = file_search(Vflats, filter)
addedflats = make_array(1530,1020,value=0)
FOREACH element, flatsfiles DO BEGIN
flatsarray = readfits(element)
addedflats = [addedflats + flatsarray]
ENDFOREACH
Pretty simple code, yet the values in addedflats are negative, while all the arrays have ONLY positive elements on the order of 10^4. Can anyone see where I'm going wrong, or have a different way of doing this?
I've also tried adding the arrays one by one to see where it goes wrong:
array = readfits(flatsfiles[0])
addedflats = [addedflats + array]
array2 = readfits(flatsfiles[1])
addedflats = [addedflats + array2]
Here, the first addedflats array shows the same thing as array, which is expected since it's just being added to an array of 0's. The second addedflats, however, gives negative numbers again. For reference, the first element of array is 25189, the first element of array2 is 24030, but the first element of addedflats is -16317 rather than the expected 49219. TIA!!

Be careful of integer types that might not be able to hold the sum. Your example is probably 16-bit signed integers (the default integers in IDL):
IDL> print, 25189 + 24030
-16317
To fix, declare at least one of the arrays that you are adding to be an integer type that is big enough to hold there result. Here, 32-bit longs are sufficient (just doing scalars for simplicity, but the same works for arrays):
IDL> print, 25189L + 24030L
49219

Related

Octave: How to assign the right dimensionality to an existing cell array to avoid "error: =: nonconformant arguments (op1 is 2x2x2, op2 is 2x2x2)"?

I am a beginner at Octave, that is why I will share something too obvious for the most.
cell_arr = cell(2,2,2);
cell_arr(2,2,2) = cell(2,2,2);
error: =: nonconformant arguments (op1 is 2x2x2, op2 is 2x2x2)
I am assigning an array of the same dimensionality as the cell_array, and it is not accepted. What should I change?
This is a spin-off from Please Explain Octave-Error : operator /: nonconformant arguments (op1 is 1x1, op2 is 1x10) and Error: nonconformant arguments (op1 is 1x3, op2 is 1x2) which both do not deal with cell arrays.
It's not clear what you're trying to do, but your understanding of cells seems to be a bit confused.
To make matters worse, I think you are coming across a bug: https://savannah.gnu.org/bugs/?func=detailitem&item_id=59637
I don't want to get too technical and confuse you even more, but what is happening here is this. We usually introduce cell arrays by saying this little story:
"There are two kinds of arrays: normal arrays, and cell arrays. Normal arrays always need to be 'rectangular', and contain elements of the same type. Cell arrays on the other hand, can contain elements of different types."
However, this isn't exactly true. It's a simplification. In reality, a 'cell' is simply a special kind of object; a container if you like. And a cell array then, is simply a normal array, whose elements are all 'cell objects'. In fact, the cell command is simply a shortcut way for creating an array of empty cells, nothing more.
More generally, cell arrays are indexed using {}, which opens up the cell object, and gives you its contents.
However, since they can also be thought of as normal arrays of 'cell objects', you can also index it with () like a normal array, and return the 'cell object' itself (as opposed to its contents).
E.g.
a = cell(1,2) # this is equivalent to a = { [], [] }
a{1} # returns an empty numerical array, which is what the first cell contains.
a(1) # returns a cell object, which happens to contain an empty numerical array.
Regarding the bug you're coming across, octave seems to report the wrong size for the elements you're trying to access when it comes to multidimensional cell arrays. This has been reported. What you should have been getting was something like
Error: op1 is 1x1, op2 is 2x2x2
In other words: "you are trying to cram a 2x2x2 array (whose elements happen to be cell objects) into a space that only fits a single element (i.e. at position 2,2,2)."
The reason for the error is that a new cell array can only be assigned to an existing cell array if the dimensions are exactly the same.
cell_arr = cell(2,2,2);
x = cell(2,2,2);
cell_arr(2,2,2) = x;
The last line causes an error, since cell_arr(2,2,2) chooses just the second item of each dimension, and not the 2 items of each dimension. Instead, only the initialisation of cell(2,2,2) builds a cell array of 2 items for each dimension.
It is thus important on which side you are, and whether you initiate or re-use a cell array.
The following works:
cell_arr(:,:,:) = x;
cell_arr(1:2,1:2,1:2) = x;
cell_arr(1:end,1:end,1:end) = x;
cell_arr(:,:,:) = x(1:2,1:2,1:2);
cell_arr(:,:,:) = x(:,:,:);
cell_arr(:,:,:) = x(1:end,1:end,1:end);
Of course, a directly initiated cell array can also be assigned:
cell_arr(:,:,:) = cell(1:2,1:2,1:2);
and this does not work with "end", since it does not exist till then:
cell_arr(1:end,1:end,1:end) = cell(1:end,1:end,1:end);
error: invalid use of 'end': may only be used to index existing value
Thus the solution is as follows, if you want to assign something to cell_arr(2,2,2):
Right: Two cell arrays of 2x2x2 are created.
cell_arr = cell(2,2,2);
x = cell(2,2,2);
And then:
Left: Only the 2nd item of each dim is chosen = 1x1. Right: Thus, only one item can be chosen per dim, for example the first item as follows:
cell_arr(2,2,2) = x(1,1,1);

Weird set index error

I am stuck on this chunk of code
hdiag = zeros(Float64,2)
hdiag = [0,0]
println(hdiag)
hdiag[1] = randn()
In the last line I obtain an InexactError.
It is strange because randn() it's a Float64, but for some reason I have to do hdiag=randn(2) and then there should not be a problem.
The line:
hdiag = [0,0]
changes hdiag to refer to a completely new and different array than what it was before. In this case, that new array is an integer array, and so any subsequent assignments into it need to be convertible to integers.
Indexed assignment is different; it changes the contents of the existing array. So you can use hdiag[:] = [0,0] and it will change the contents, converting the integers to floats as it does so. This gets even easier in version 0.5, where you can use the new .= dot assignment syntax to assign into an existing array:
hdiag .= [0,0]
will do what you want. For more details on arrays, bindings, and assignment, I recommend reading this blog post: Values vs. Bindings: The Map is Not the Territory.

Boolean (BitArray) multidimensional array indexing or masking in Julia?

As part of a larger algorithm, I need to produce the residuals of an array relative to a specified limit. In other words, I need to produce an array which, given someArray, comprises elements which encode the amount by which the corresponding element of someArray exceeds a limit value. My initial inclination was to use a distributed comparison to determine when a value has exceeded the threshold. As follows:
# Generate some test data.
residualLimit = 1
someArray = 2.1.*(rand(10,10,3).-0.5)
# Determine the residuals.
someArrayResiduals = (residualLimit-someArray)[(residualLimit-someArray.<0)]
The problem is that the someArrayResiduals is a one-dimensional vector containing the residual values, rather than a mask of (residualLimit-someArray). If you check [(residualLimit-someArray.<0)] you'll find that it is behaving as expected; it's producing a BitArray. The question is, why doesn't Julia allow to use this BitArray to mask someArray?
Casting the Bools in the BitArray to Ints using int() and distributing using .*produces the desired result, but is a bit inelegant... See the following:
# Generate some test data.
residualLimit = 1
someArray = 2.1.*(rand(10,10,3).-0.5)
# Determine the residuals.
someArrayResiduals = (residualLimit-someArray).*int(residualLimit-someArray.<0)
# This array should be (and is) limited at residualLimit. This is correct...
someArrayLimited = someArray + someArrayResiduals
Anyone know why a BitArray can't be used to mask an array? Or, any way that this entire process can be simplified?
Thanks, all!
Indexing with a logical array simply selects the elements at indices where the logical array is true. You can think of it as transforming the logical index array with find before doing the indexing expression. Note that this can be used in both array indexing and indexed assignment. These logical arrays are often themselves called masks, but indexing is more like a "selection" operation than a clamping operation.
The suggestions in the comments are good, but you can also solve your problem using logical indexing with indexed assignment:
overLimitMask = someArray .> residualLimit
someArray[overLimitMask] = residualLimit
In this case, though, I think the most readable way to solve this problem is with min or clamp: min(someArray, residualLimit) or clamp(someArray, -residualLimit, residualLimit)

Implicit conversion from system array to ilnumerics array doesnt retain double precision

When I assign a system array of doubles to an ilnumerics double array, the values are rounded off to nearest integer. This happens particularly for only large arrays.
Is there any way in ILnumerics to specify up to how many decimals the rounding should occur?
The following screenshot shows the problem . Sample_pulsedata is double array of length 1860 which I am assigning to sample_ydata.
The elements are not really rounded. The effect rather comes from the way the elements are displayed in Visual Studios data tips. ILNumerics tries to find a common scale factor which allows to display all elements in an array aligned.
In your example - presumably - there exist large values at higher indices, which are not shown currently (scroll down in order to find them). These elements cause the scale factor to be 1/10^4. This is indicated in the first line, index [0]: '(:;:) 1e+004'. The 32.57 therefore must get rounded to 33 in order to fit into the 4 digits after the decimal point. '4' is a fixed value in ILNumerics and cannot easily get changed.
The actual values of the array elements are not affected, of course. You can use the Watch window to show only the interesting part of the array, without the rounding effect:
sample_ydata["0:13"]
Or, even better, use the ILNumerics Array Visualizer in order to visualize your data graphically. This not only gives a nice overview of the whole array but also prevents from such artefacts as you encountered.

Stackoverflow with specialized Hashtbl (via Hashtbl.make)

I am using this piece of code and a stackoverflow will be triggered, if I use Extlib's Hashtbl the error does not occur. Any hints to use specialized Hashtbl without stackoverflow?
module ColorIdxHash = Hashtbl.Make(
struct
type t = Img_types.rgb_t
let equal = (==)
let hash = Hashtbl.hash
end
)
(* .. *)
let (ctable: int ColorIdxHash.t) = ColorIdxHash.create 256 in
for x = 0 to width -1 do
for y = 0 to height -1 do
let c = Img.get img x y in
let rgb = Color.rgb_of_color c in
if not (ColorIdxHash.mem ctable rgb) then ColorIdxHash.add ctable rgb (ColorIdxHash.length ctable)
done
done;
(* .. *)
The backtrace points to hashtbl.ml:
Fatal error: exception Stack_overflow Raised at file "hashtbl.ml",
line 54, characters 16-40 Called from file "img/write_bmp.ml", line
150, characters 52-108 ...
Any hints?
Well, you're using physical equality (==) to compare the colors in your hash table. If the colors are structured values (I can't tell from this code), none of them will be physically equal to each other. If all the colors are distinct objects, they will all go into the table, which could really be quite a large number of objects. On the other hand, the hash function is going to be based on the actual color R,G,B values, so there may well be a large number of duplicates. This will mean that your hash buckets will have very long chains. Perhaps some internal function isn't tail recursive, and so is overflowing the stack.
Normally the length of the longest chain will be 2 or 3, so it wouldn't be surprising that this error doesn't come up often.
Looking at my copy of hashtbl.ml (OCaml 3.12.1), I don't see anything non-tail-recursive on line 54. So my guess might be wrong. On line 54 a new internal array is allocated for the hash table. So another idea is just that your hashtable is just getting too big (perhaps due to the unwanted duplicates).
One thing to try is to use structural equality (=) and see if the problem goes away.
One reason you may have non-termination or stack overflows is if your type contains cyclic values. (==) will terminates on cyclic values (while (=) may not), but Hash.hash is probably not cycle-safe. So if you manipulate cyclic values of type Img_types.rgb_t, you have to devise your one cycle-safe hash function -- typically, calling Hash.hash on only one of the non-cyclic subfields/subcomponents of your values.
I've already been bitten by precisely this issue in the past. Not a fun bug to track down.

Resources