Is there a way to do weighted randomization in System Verilog based on runtime data. Say, I have a queue of integers and a queue of weights (unsigned integers) and wish to select a random integer from the first queue as per the weights in the second queue.
int data[$] = '{10, 20, 30};
uint_t weights[$] = '{100, 200, 300};
Any random construct expects the weights hardcoded as in
constraint range { Var dist { [0:1] := 50 , [2:7] := 50 }; }
But in my case, I need to pick an element from an unknown number of elements.
PS: Assume the number of elements and weights will be the same always.
Unfortunately, the dist constraint only lets you choose from a fixed number of values.
Two approaches I can think of are
Push each data value into a queue using the weight as a repetition count. In your example, you wind up with a queue of 600 values. Randomly pick an index into the queue. The selected element has the distribution you want. An example is posted here.
Create an array of ranges for each weight. For your example the array would be uint_t ranges[][2]'{{0,99},{100,299},{300,599}}. Then you could do the following in a constraint
index inside {[0:weights.sum()-1]};
foreach (data[ii])
index inside {[ranges[ii][0]:ranges[ii][1]} -> value == date[ii];
Related
I am currently having an issue. Basically, I have 2 similar functions in terms of concept but the results do not align. These are the codes I learned from Bioinformatics I on Coursera.
The first code is simply creating a dictionary of occurrences of each k-mer pattern from a text (which is a long stretch of nucleotides). In this case, k is 5.
def FrequencyMap(text,k):
freq ={}
for i in range (0, len(text)-k+1):
freq[text[i:i+k]]=0
for j in range (0, len(text)-k+1):
if text[j:j+k] == text[i:i+k]:
freq[text[i:i+k]] +=1
return freq, max(freq)
The text and the result dictionary are kinda long, but the main point is when I call max(freq), it returns the key 'TTTTC', which has a value of 1.
Meanwhile, I wrote another code that is simply based on the previous code to generate the 5-mer patterns that have the max values (number of occurrences in the text).
def FrequentWords(text, k):
a = FrequencyMap(text, k)
m = max(a.values())
words = []
for i in a:
if a[i]==m:
words.append(i)
return words,m
And this code returns 'ACCTA', which has the value of 99, meaning it appears 99 times in the text. This makes total sense.
I used the same text and k (k=5) for both codes. I ran the codes on Jupyter Notebook. Why does the first one not return 'ACCTA'?
Thank you so much,
Here is the text, if anyone wants to try:
"ACCATCCCTAGGGCATACCTAAGTCTACCTAAAAGGCTACCTAATACCATACCTAATTACCTAACTACCTAAAATAAGTCTACCTAATACCTAATACCTAAAGTTACCTAACGTACCTAATACCTAATACCTAACCACTACCTAATCCGATTTACCTAACAACCGATCGAGTACCTAATCGATACCTAAATAACGGACAATATACCTAATTACCTAATACCTAATACCTAAGTGTACCTAAGACGTCTACCTAATTGTACCTAACTACCTAATTACCTAAGATTAATACCTAATACCTAATTTACCTAATACCTAACGTGGACTACCTAATACCTAACTTTTCCCCTACCTAATACCTAACTGTACCTAAATACCTAATACCTAAGCTACCTAAAGAACAACATTGTACGTGCGCCGTACCTAAATACCTAACAACTACCTAACTGATACCTAATAGTGATTACCTAACGCTTCTACCTAACTACCTAAGTACCTAACGCTACCTAACTACCTAATGTCCACAAAATACCTAATACCTAATAGCTACCTAATTGTGTACCTAAGTACCTAACCTACCTAATAATACCTAAAAATACCTAAGTACCTAACGTACCTAAATTTTACCTAATCTACCTAACGTACCTAATACCTAATTATACCTAATTACCTAATGGTTACCTAAGTTACCTAATATGCCACTACCTAACCTTACCTAAGACCTACCTAATAGGTACCTAACTGGGTACCTAAGGCAGTTTACCTAATTCAGGGCTACCTAATGTACCTAATACCTAAGTACCTAATACCTAATCCCATACCTAATATTTACCTAAGGGCACCGGTACCTAATACCTAATACCTAATACCTAAACCTTCGTACCTAAATACCTAATCTACCTAATGTACCTAAGGTACCTAATACCTAAGTCACTACCTAATACCTAATACCTAATGGGAGGAGCTTACCTAAGGTTACCTAATTACCTAAATACCTAATCGTTACCTAA"
Why does the first one not return 'ACCTA'?
Because max(freq) returns the maximum key of the dictionary. In this case the keys are strings (the k-mers), and strings are compared alphabetically. Hence the maximum one is the last string when the are sorted alphabetically.
If you want the first function to return the k-mer that occurs most often, you should change max(freq) to max(freq.items(), key=lambda key_value_pair: key_value_pair[1])[0]. Here, you are sorting the (kmer, count) pairs (that's the key_value_pair parameter of the lambda expression) based on the frequency and then selecting the kmer.
I am trying to simulate a step through a Markov chain with the intent to loop through the procedure multiple times until a condition is met. (i.e., find out how many steps it takes, on average, to reach a specific state).
In this case, a state can only go one way. E.g., State 4 can transition forward to State 5, but cannot transition backward to State 3. This means the left-lower half of the transition matrix is set to zero. This is also why the method below puts arbitrarily large values in the 'prior' states. I attempt to find the correct new state by examining which probability in the specified row of the transition matrix is closest to a random number.
get_new_state <- function(current_state, trans_matrix)
{
# generate a random number between 0-1 to compare to the transition matrix probabilities
rand <- runif(1)
# transition to where the
# random number falls within the transition matrix
row <- current_state # initial condition determines the row of the trans_matrix
col = current_state # start in the column
# loop thru all columns and find the correct state
potential_states <- rep(0, each=ncol(trans_matrix)) # holds the value of the potential state it transitions to
# in this case, we can't transition to a previous state so we set the previous state values arbitrarily high
# so they don't get selected in the which.min() function later
potential_states[1:col] <- 999
for(k in row:ncol(trans_matrix)) # loop thru non-zero matrix values
{
if(trans_matrix[row,k] > rand)
{
potential_states[k] <- trans_matrix[row,k] / rand
potential_states[k] <- 1 - potential_states[k]
}
}
# new state is equal to the index of the lowest value
# lowest value = closest to random number
new_state = which.min(potential_states)
return(as.numeric(new_state))
}
I'm not sure if this approach is reasonable. I'm assuming there is a better way to simulate without the kluge that puts arbitrarily large values in potential_states[].
Would something like this work better (it is a one-line Markov transition):
> new_state <- sample(1:number_states, size=1,
prob=transition_matrix[old_state,])
and just put this in a (for instance) while() loop with a counter.
I'm trying to create constraint using a 2d array filled with 1's and 0's.
I want every row to have a total sum of either 0,1,2,4 and every column sums to 1. Moreover, I want the entire sum of the matrix to be equal to 8 (which worked using sum function).
I tried using the function .sum() to constraint of every row and column to the value I demanded, but I keep getting syntax error. What am I doing wrong?
rand bit MAT[8][8];
constraint range {
MAT.sum() with (32' (item)) == 8;
foreach (MAT[j]){
foreach(MAT[j][i]){
MAT[j][].sum() with (32' (item)) == 1;
MAT[][i].sum() with (32' (item)) inside {0,1,2,4};
}
}
Section 7.12.3 Array reduction methods of the IEEE 1800-2012 standard states that
[array] reduction methods may be applied to any unpacked array of integral values to reduce the array to a
single value.
While MAT[0].sum() or MAT[1].sum() is allowed (apply sum on line 0 and 1 respectively of MAT), MAT.sum() is not. A line in MAT is an array of bit and bit is an integral type, but MAT is an array of unpacked array of bit which is not an integral type.
Also, it's not possible to select a single column from an array. You can only slice row-wise. This is a bit trickier to implement, but doable.
Let's look at each constraint. First, constraining the sum of each row is easily done with the sum() function:
constraint sum_on_row {
foreach (MAT[i])
MAT[i].sum() with (32'(item)) inside { 0, 1, 2, 4 };
}
To constraint the sum on a column you'll need to transpose the array (rows become columns, columns become rows) and constrain that. First, let's define the transpose of MAT:
rand bit MAT_transp[8][8];
constraint construct_MAT_transp {
foreach (MAT[i,j])
MAT_transp[j][i] == MAT[i][j];
}
We allocate another array and keep its contents in sync with those of MAT. Any constraints on MAT_transp will indirectly affect MAT. As before, we can constrain the rows of MAT_transp, which will effectively constrain the columns of MAT:
constraint sum_on_col {
foreach (MAT_transp[i])
MAT_transp[i].sum() with (32'(item)) == 1;
}
And lastly, you want the sum of all elements in the array to be 8. This is the trickiest to do. While we can't constrain the array sum directly, we can split the problem into two parts. First, we can compute the sum for each row in MAT and store them all in an array:
rand int unsigned row_sums[8];
constraint compute_row_sums {
foreach (row_sums[i])
row_sums[i] == MAT[i].sum() with (32'(item));
}
Now that we have the sum on each row, it's easy to constrain the sum of the entire array by constraining the sum of all row sums:
constraint sum_of_matrix {
row_sums.sum() == 8;
}
The cool thing is that with this one problem we've covered a lot of the usual "tricks" we can apply when constraining arrays. You can find more array constraint idioms in an old post I wrote.
Actually, array2d.sum() works with VCS.
For sum along 2nd index:
Instead of
MAT[j][].sum() with (32'(item))
Use
MAT[j].sum() with (32'(item))
For sum along 1st index:
Instead of
MAT[][i].sum() with (32'(item))
Use
MAT.sum() with (32'(item[i]))
Note: For sum along any index of a multi-dimensional array, specify the higher dimension indices and then reduce the sub array obtained, using with clause for lower dimension indices.
Example:
bit bit_array [n0][n1][n2][n3][n4];
int sum_0 = bit_array.sum() with (int'(item[i1][i2][i3][i4]));
int sum_1 = bit_array[i0].sum() with (int'(item[i2][i3][i4]));
int sum_2 = bit_array[i0][i1].sum() with (int'(item[i3][i4]));
int sum_3 = bit_array[i0][i1][i2].sum() with (int'(item[i4]));
int sum_4 = bit_array[i0][i1][i2][i3].sum() with (int'(item));
I'm interested in distributing an MxN integer array across p workers. Is there a way to specify which dimension gets distributed? In particular, I want to keep the number of rows M fixed and distribute over N columns. In my case M > N (I have a term-document matrix with vocabulary of size M and number of documents N).
By default, Julia appears to distribute over the dimension that has the largest size, which doesn't work for my application (I want to distribute over the documents and not the vocabulary). Is there a way to control which dimension gets distributed?
SharedArray constructor has a pids optional parameter which maps elements to processes (see documentation).
So, an MxN matrix can be initialized with the following code:
# a helper function which might be useful in other contexts
function balancedfill(v,n,b)
d,r = divrem(n,b)
return v[[repeat(1:r,inner=d+1);repeat(r+1:b,inner=d)]]
end
# N,M = size(mat)
pidvec = repeat(balancedfill(1:nprocs(),N,nprocs()),inner=M)
sharedmat = SharedArray{Float64}((N,M); pids=pidvec)
This creates a Float64 shared array, with columns balanced between processes. Float64 can be replaced by the element-type needed. With a little change (switching inner with outer and N with M in pidvec creation) a row-wise distributed array can be created.
I'm writing something that reads bytes (just a List<int>) from a remote random number generation source that is extremely slow. For that and my personal requirements, I want to retrieve as few bytes from the source as possible.
Now I am trying to implement a method which signature looks like:
int getRandomInteger(int min, int max)
I have two theories how I can fetch bytes from my random source, and convert them to an integer.
Approach #1 is naivé . Fetch (max - min) / 256 number of bytes and add them up. It works, but it's going to fetch a lot of bytes from the slow random number generator source I have. For example, if I want to get a random integer between a million and a zero, it's going to fetch almost 4000 bytes... that's unacceptable.
Approach #2 sounds ideal to me, but I'm unable come up with the algorithm. it goes like this:
Lets take min: 0, max: 1000 as an example.
Calculate ceil(rangeSize / 256) which in this case is ceil(1000 / 256) = 4. Now fetch one (1) byte from the source.
Scale this one byte from the 0-255 range to 0-3 range (or 1-4) and let it determine which group we use. E.g. if the byte was 250, we would choose the 4th group (which represents the last 250 numbers, 750-1000 in our range).
Now fetch another byte and scale from 0-255 to 0-250 and let that determine the position within the group we have. So if this second byte is e.g. 120, then our final integer is 750 + 120 = 870.
In that scenario we only needed to fetch 2 bytes in total. However, it's much more complex as if our range is 0-1000000 we need several "groups".
How do I implement something like this? I'm okay with Java/C#/JavaScript code or pseudo code.
I'd also like to keep the result from not losing entropy/randomness. So, I'm slightly worried of scaling integers.
Unfortunately your Approach #1 is broken. For example if min is 0 and max 510, you'd add 2 bytes. There is only one way to get a 0 result: both bytes zero. The chance of this is (1/256)^2. However there are many ways to get other values, say 100 = 100+0, 99+1, 98+2... So the chance of a 100 is much larger: 101(1/256)^2.
The more-or-less standard way to do what you want is to:
Let R = max - min + 1 -- the number of possible random output values
Let N = 2^k >= mR, m>=1 -- a power of 2 at least as big as some multiple of R that you choose.
loop
b = a random integer in 0..N-1 formed from k random bits
while b >= mR -- reject b values that would bias the output
return min + floor(b/m)
This is called the method of rejection. It throws away randomly selected binary numbers that would bias the output. If min-max+1 happens to be a power of 2, then you'll have zero rejections.
If you have m=1 and min-max+1 is just one more than a biggish power of 2, then rejections will be near half. In this case you'd definitely want bigger m.
In general, bigger m values lead to fewer rejections, but of course they require slighly more bits per number. There is a probabilitistically optimal algorithm to pick m.
Some of the other solutions presented here have problems, but I'm sorry right now I don't have time to comment. Maybe in a couple of days if there is interest.
3 bytes (together) give you random integer in range 0..16777215. You can use 20 bits from this value to get range 0..1048575 and throw away values > 1000000
range 1 to r
256^a >= r
first find 'a'
get 'a' number of bytes into array A[]
num=0
for i=0 to len(A)-1
num+=(A[i]^(8*i))
next
random number = num mod range
Your random source gives you 8 random bits per call. For an integer in the range [min,max] you would need ceil(log2(max-min+1)) bits.
Assume that you can get random bytes from the source using some function:
bool RandomBuf(BYTE* pBuf , size_t nLen); // fill buffer with nLen random bytes
Now you can use the following function to generate a random value in a given range:
// --------------------------------------------------------------------------
// produce a uniformly-distributed integral value in range [nMin, nMax]
// T is char/BYTE/short/WORD/int/UINT/LONGLONG/ULONGLONG
template <class T> T RandU(T nMin, T nMax)
{
static_assert(std::numeric_limits<T>::is_integer, "RandU: integral type expected");
if (nMin>nMax)
std::swap(nMin, nMax);
if (0 == (T)(nMax-nMin+1)) // all range of type T
{
T nR;
return RandomBuf((BYTE*)&nR, sizeof(T)) ? *(T*)&nR : nMin;
}
ULONGLONG nRange = (ULONGLONG)nMax-(ULONGLONG)nMin+1 ; // number of discrete values
UINT nRangeBits= (UINT)ceil(log((double)nRange) / log(2.)); // bits for storing nRange discrete values
ULONGLONG nR ;
do
{
if (!RandomBuf((BYTE*)&nR, sizeof(nR)))
return nMin;
nR= nR>>((sizeof(nR)<<3) - nRangeBits); // keep nRangeBits random bits
}
while (nR >= nRange); // ensure value in range [0..nRange-1]
return nMin + (T)nR; // [nMin..nMax]
}
Since you are always getting a multiple of 8 bits, you can save extra bits between calls (for example you may need only 9 bits out of 16 bits). It requires some bit-manipulations, and it is up to you do decide if it is worth the effort.
You can save even more, if you'll use 'half bits': Let's assume that you want to generate numbers in the range [1..5]. You'll need log2(5)=2.32 bits for each random value. Using 32 random bits you can actually generate floor(32/2.32)= 13 random values in this range, though it requires some additional effort.