I am trying to store a set of 1 dimensional IDL arrays inside a 2-dimensional array in IDL. I cannot find any documentation on Harris Geospatial and I am not having luck just mucking around with syntax.
A 3-dimensional array seems the obvious solution, but the length of the 1D arrays I need to store varies from 1 - 800 integers in length, so I would utilize very little of a 3D array.
Any pointers much appreciated.
Thanks.
Your statement,
Any pointers much appreciated
is somewhat ironic, given that one possible solution to your problem is to use pointers! To my knowledge, it's the only way you can store variable-length arrays in a single array. Basically, pointers are just variables, but instead of containing data, they contain a pointer to where data is stored in memory.
You can create a pointer by using the PTR_NEW function:
IDL> p = ptr_new(findgen(5))
IDL> help, p
P POINTER = <PtrHeapVar22>
To "dereference" the pointer (i.e., access the data), you need to use an asterisk:
IDL> print, p
<PtrHeapVar22>
IDL> print, *p
0.00000 1.00000 2.00000 3.00000 4.00000
So, what you need is a PTRARR (pointer array):
IDL> test_arr = ptrarr(2,3,/ALLOCATE_HEAP)
IDL> help, test_arr
TEST_ARR POINTER = Array[2, 3]
where each element of the array is one of your 1-D arrays. You can populate the array by storing pointers:
IDL> test_arr[0,0] = p
IDL> print, *test_arr[0,0]
0.00000 1.00000 2.00000 3.00000 4.00000
or by assigning your arrays to the dereferenced elements of the pointer array:
IDL> *test_arr[0,1] = randomu(seed, 4)
IDL> print, *test_arr[0,1]
0.838449 0.967399 0.0669304 0.101592
One downside of using pointers is that you lose a lot (if not all) of the nice vectorization benefits of ordinary IDL arrays. That is, you will generally need to loop over the elements of the array in order to access the values stored in the pointers, which means you will take a performance hit.
One other thing to look out for is how operator precedence affects the use of the dereferencing operator (the asterisk). For example, if you store a structure in a pointer, you need to use parentheses in the following way to access the data in the structure:
IDL> str = {a:1,b:'x'}
IDL> str_ptr = ptr_new(str)
IDL> print, (*str_ptr).a
1
Otherwise you get an error:
IDL> print, *str_ptr.a
% Expression must be a structure in this context: STR_PTR.
% Execution halted at: $MAIN$
Here's the documentation for further reference.
Related
This is a constructor for arrays:
Array{T}(undef, dims)
I am new to Julia, and don't have a good background in programming. In this syntax, why is undef used for creating the array?
What is a constructor in Julia, in what situation do we use a constructor?
If we don't type constructor, Julia will automatically create a constructor. Then, Why we use constructor?
First, you want to understand what is a constructor:
For that, I suggest you the Julia doc: Constructors in Julia
Now that you have the theory, let's break apart this expression:
a = Array{Int}(undef, (2, 2))
What this expression is saying is "I want a to be an Array of dimension (2, 2)". So Julia will ask for some memory space. When I write it on the Julia REPL:
julia> a = Array{Int}(undef, (2, 2))
2×2 Array{Int64,2}:
0 0
0 0
Now Array{T}(undef, dims) is the generalization of that. "Construct an array of a specific type T with a specific number of dimensions dims"
So far, I didn't explain what is undef. undef is a shortcut for UndefInitializer(). In this example, we wanted an uninitialized array. What does it mean? For that, you have to understand that variables are not created ex nihilo on your terminal. They are occupying a specific place in the memory of your computer. And sometimes, the same memory space was occupied by another variable. So the space my new variable can take might not be empty:
julia> a = Array{Float64}(undef, (2, 2))
2×2 Array{Float64,2}:
6.94339e-310 6.94339e-310
6.94339e-310 0.0
Here, I never asked for these values to be there. I could erase it to work with a clean variable. But that would mean to erase the value for each cell, and it's much more expensive for the computer to replace each value rather than declaring "here is the new variable".
So basically, undef and uninitialized arrays are used for performance purposes. If you want an array well initialized, you can use fill.
I have a pointer uvw(:,:) which is two-dimensional, and I got a 1d buffer array x(:).
Now I need to point uvw(1,:)=>x(1:ncell) and uvw(2,:)=>x(ncell+1:ncell*2) etc.
I made a very simple example. I know that array of pointers does not work, but does anybody have an idea how this can be worked around?
PS: For a pragmatic reason I do not want to wrap my uvw with a declared type. ( i am changing some bit of code, and need uvw as 2D pointer. Currently is an array, and my idea is to avoid changing the way uvw is being used as it being used thousands of times)
program test
real, allocatable,target :: x(:)
real, pointer :: ptr(:,:)
allocate(x(100) )
x = 1.
ptr(1,:) => x(1:10)
end program
The error message says:
`error #8524: The syntax of this data pointer assignment is incorrect:
either 'bound spec' or 'bound remapping' is expected in this context.
[1]
ptr(1,:) => x(1:10)
----^`
You are trying to perform pointer bounds remapping, but you have the incorrect syntax and approach.
Pointer bounds remapping is a way to have the shape of the pointer different from that of the target. In particular, the rank of the pointer and target may differ. However, in such an assignment it is necessary to explicitly specify the lower and upper bounds of the remapping; it isn't sufficient to use : by itself.
Also, you'll need to assign the whole pointer in one go. That is, you can't have "the first ten elements point to this slice, the next ten to this slice" and so on in multiple statements.
The assignment statement would be
ptr(1:10,1:10) => x
Note, that this also means that you can't actually have what you want. You are asking for the elements ptr(1,1:10) to correspond to x(1:10) and ptr(2,2:10) to correspond to x(11:20). That isn't possible: the array elements must match in order: ptr(1:10,1) being the first ten elements of ptr must instead be associated with the first ten elements x(1:10). The corrected pointer assignment above has this.
If you prefer avoiding a pointer, then the UNION/MAP is an option depending on compiler. It was added to gfortran a while ago... then you can think of the array as a rank=2 but also use the vector (Rank=1) for SIMD operations.
All this assumes that one wants to avoid pointers...
As part of a larger algorithm, I need to produce the residuals of an array relative to a specified limit. In other words, I need to produce an array which, given someArray, comprises elements which encode the amount by which the corresponding element of someArray exceeds a limit value. My initial inclination was to use a distributed comparison to determine when a value has exceeded the threshold. As follows:
# Generate some test data.
residualLimit = 1
someArray = 2.1.*(rand(10,10,3).-0.5)
# Determine the residuals.
someArrayResiduals = (residualLimit-someArray)[(residualLimit-someArray.<0)]
The problem is that the someArrayResiduals is a one-dimensional vector containing the residual values, rather than a mask of (residualLimit-someArray). If you check [(residualLimit-someArray.<0)] you'll find that it is behaving as expected; it's producing a BitArray. The question is, why doesn't Julia allow to use this BitArray to mask someArray?
Casting the Bools in the BitArray to Ints using int() and distributing using .*produces the desired result, but is a bit inelegant... See the following:
# Generate some test data.
residualLimit = 1
someArray = 2.1.*(rand(10,10,3).-0.5)
# Determine the residuals.
someArrayResiduals = (residualLimit-someArray).*int(residualLimit-someArray.<0)
# This array should be (and is) limited at residualLimit. This is correct...
someArrayLimited = someArray + someArrayResiduals
Anyone know why a BitArray can't be used to mask an array? Or, any way that this entire process can be simplified?
Thanks, all!
Indexing with a logical array simply selects the elements at indices where the logical array is true. You can think of it as transforming the logical index array with find before doing the indexing expression. Note that this can be used in both array indexing and indexed assignment. These logical arrays are often themselves called masks, but indexing is more like a "selection" operation than a clamping operation.
The suggestions in the comments are good, but you can also solve your problem using logical indexing with indexed assignment:
overLimitMask = someArray .> residualLimit
someArray[overLimitMask] = residualLimit
In this case, though, I think the most readable way to solve this problem is with min or clamp: min(someArray, residualLimit) or clamp(someArray, -residualLimit, residualLimit)
Is it possible to convert a pointer to certain value to a slice?
For example, I want to read single byte from io.Reader into uint8 variable. io.Reader.Read accepts a slice as its argument, so I cannot simply provide it a pointer to my variable as I'd do in C.
I think that creating a slice of length 1, capacity 1 from a pointer is safe operation. Obviously, it should be the same as creating a slice from an array of length 1, which is allowed operation. Is there an easy way to do this with plain variable? Or maybe I do not understand something and there are reasons why this is prohibited?
A slice is not only a pointer, like an array in C. It also contains the length and capacity of the data, like this:
struct {
ptr *uint8
len int
cap int
}
So, yes, you will need to create a slice. Simplest way to create a slice of the var a uint8 would be []uint8{a}
a := uint8(42)
fmt.Printf("%#v\n", []uint8{a})
(But after rereading your question, this is not a solution as all)
But if you wish to create the slice from the variable, pointing to the same space of memory, you could use the unsafe package. This is most likely to be discouraged.
fmt.Printf("%#v\n", (*[1]uint8)(unsafe.Pointer(&a))[:] )
Instead of (over)complicating this trivial task, why not to use the simple solution? I.e. pass .Read a length-1 slice and then assign its zeroth element to your variable.
I found a way to overcome my case when I want to supply a variable to io.Reader. Go standard library is wonderful!
import (
"io"
"encoding/binary"
)
...
var x uint8
binary.Read(reader, LittleEndian, &x)
As a side effect this works for any basic type and even for some non-basic.
Does anyone know how to do this and what the pseudo code would look like?
As we all know a hash table stores key,value pairs and when a key is a called, the function will return the value associated with that key. What I want to do is understand the underlying structure in creating that mapping function. For example, if we lived in a world where there were no previously defined functions except for arrays, how could we replicate the Hashmaps that we have today?
Actually, some of todays Hashmap implentations are indeed made out of arrays as you propose. Let me sketch how this works:
Hash Function
A hash function transforms your keys into an index for the first array (array K). A hash function such as MD5 or a simpler one, usually including a modulo operator, can be used for this.
Buckets
A simple array-based Hashmap implementation could use buckets to cope with collissions. Each element ('bucket') in array K contains itself an array (array P) of pairs. When adding or querying for an element, the hash function points you to the correct bucket in K, which contains your desired array P. You then iterate over the elements in P until you find a matching key, or you assign a new element at the end of P.
Mapping keys to buckets using the Hash
You should make sure that the number of buckets (i.e. the size of K) is a power of 2, let's say 2^b. To find the correct bucket index for some key, compute Hash(key) but only keep the first b bits. This is your index when cast to an integer.
Rescaling
Computing the hash of a key and finding the right bucket is very quick. But once a bucket becomes fuller, you will have to iterate more and more items before you get to the right one. So it is important to have enough buckets to properly distribute the objects, or your Hashmap will become slow.
Because you generally don't know how much objects you will want to store in the Hashmap in advance, it is desirable to dynamically grow or shrink the map. You can keep a count of the number of objects stored, and once it goes over a certain threshold you recreate the entire structure, but this time with a larger or smaller size for array K. In this way some of the buckets in K that were very full will now have their elements divided among several buckets, so that performance will be better.
Alternatives
You may also use a two-dimensional array instead of an array-of-arrays, or you may exchange array P for a linked list. Furthermore, instead of keeping a total count of stored objects, you may simply choose to recreate (i.e. rescale) the hashmap once one of the buckets contains more than some configured number of items.
A variation of what you are asking is described as 'array hash table' in the Hash table Wikipedia entry.
Code
For code samples, take a look here.
Hope this helps.
Could you be more precise? Does one array contain the keys, the other one the values?
If so, here is an example in Java (but there are few specificities of this language here):
for (int i = 0; i < keysArray.length; i++) {
map.put(keysArray[i], valuesArray[i]);
}
Of course, you will have to instantiate your map object (if you are using Java, I suggest to use a HashMap<Object, Object> instead of an obsolete HashTable), and also test your arrays in order to avoid null objects and check if they have the same size.
Sample Explanation:
At the below source, basically it does two things:
1. Map Representation
Some (X number of List) of lists
X being 2 power N number of lists is bad. A (2 power N)-1, or (2 power N)+1, or a prime number is good.
Example:
List myhashmap [hash_table_size];
// an array of (short) lists
// if its long lists, then there are more collisions
NOTE: this is array of arrays, not two arrays (I can't see a possible generic hashmap, in a good way with just 2 arrays)
If you know Algorithms > Graph theory > Adjacency list, this looks exactly same.
2. Hash function
And the hash function converts string (input) to a number (hash value), which is index of an array
initialize the hash value to first char (after converted to int)
for each further char, left shift 4 bits, then add char (after converted to int)
Example,
int hash = input[0];
for (int i=1; i<input.length(); i++) {
hash = (hash << 4) + input[i]
}
hash = hash % list.size()
// list.size() here represents 1st dimension of (list of lists)
// that is 1st dimension size of our map representation from point #1
// which is hash_table_size
See at the first link:
int HTable::hash (char const * str) const
Source:
http://www.relisoft.com/book/lang/pointer/8hash.html
How does a hash table work?
Update
This is the Best source: http://algs4.cs.princeton.edu/34hash/
You mean like this?
The following is using Ruby's irb as an illustration:
cities = ["LA", "SF", "NY"]
=> ["LA", "SF", "NY"]
items = ["Big Mac", "Hot Fudge Sundae"]
=> ["Big Mac", "Hot Fudge Sundae"]
price = {}
=> {}
price[[cities[0], items[1]]] = 1.29
=> 1.29
price
=> {["LA", "Hot Fudge Sundae"]=>1.29}
price[[cities[0], items[0]]] = 2.49
=> 2.49
price[[cities[1], items[0]]] = 2.99
=> 2.99
price
=> {["LA", "Hot Fudge Sundae"]=>1.29, ["LA", "Big Mac"]=>2.49, ["SF", "Big Mac"]=>2.99}
price[["LA", "Big Mac"]]
=> 2.49