How to find a smallest value without trying all posibilities - math

I am trying to write a program (in Lua, but I guess this is more of a mathematical problem) that calculates the total distance between a set of numbers and there are two or three different possibilities for some of these numbers. 
For example a set of numbers is: 2, 5, 0, 1. The sum of distances in this case is 9. (5-2 + 5-0 + 1-0)
The first row with alternatives is: 5, -, 3, 2.
The second row with alternatives is: 3, -, -, 3
The combination between the two rows with the smallest sum of distances is: 5, 5, 3, 3 with a sum of distances of 2.
My first attempt was to write a program that tries all possible iterations, but with a set of about 40 numbers, there are so many possibilities that my computer crashed...
Here is the code for that version, it first creates all the different possibilities and then calculates the differences and places them in column 0. Afterwards I can find the smallest value easily and see also the combination of numbers which results in that value.
local table1 = {2, 5, 0 ,1}
local table2 = {5, nil, 3, 2}
local imax = 1
local solution = {}
local answer = table1[1]
for x = 1,#table1 do
solution[x]={}
for i= 1, (2^imax)/2 do
solution[x][i] = table1[x]
end
if table2[x] ~= nil then -- there is an alternative number
for y = 1, x-1 do -- copy all the previous table entries except the last one
for j = ((2^imax)/2)+1, 2^imax do -- the number of new rows increases exponentially
solution[y][j] = solution[y][j-imax]
end
end
for j = ((2^imax)/2)+1, 2^imax do -- create the new table entry with the alternative number
solution[x][j] = table2[x]
end
imax = imax + 1 -- this number is to remind how many alternative numbers where found
end
end
solution[0]={}
for x = 1, #table1 do
for i = 1, (2^imax)/2 do
if x < #table1 then answer = math.sqrt((solution[x+1][i]-solution[x][i])^2) else answer = 0 end
if solution[0][i] == nil then solution[0][i] = answer else solution[0][i] = solution[0][i] + answer end
end
end
After reading about dynamic programming, I wrote a new version of this program. It calculates the smallest sum of differences, but I also want to know the path (the combination of numbers) to that sum... Still work to do...
local table1 = {2, 5, 0 ,1}
local table2 = {5, nil, 3, 2}
local solution = {}
local smallestsolution = {}
solution[1]={}
solution[2]={}
solution[3]={}
solution[4]={}
for i = 1, (#table1-1) do
solution[1][i] = math.sqrt((table1[i+1]-table1[i])^2)
if table2[i] ~= nil then solution[2][i] = math.sqrt((table1[i+1]-table2[i])^2) end
if table2[i+1] ~= nil then solution[3][i] = math.sqrt((table2[i+1]-table1[i])^2) end
if table2[i] ~= nil and table2[i+1] ~= nil then solution[4][i] = math.sqrt((table2[i+1]-table2[i])^2) end
end
for i = 1, (#table1-1) do
smallestsolution[i]=100000
for j = 1, 4 do
if solution[j][i] ~= nil and solution[j][i] < smallestsolution[i] then smallestsolution[i]=solution[j][i] end
end
end
local smallestsum = 0
for i = 1, (#table1-1) do
smallestsum = smallestsum + smallestsolution[i]
end
Thanks,
Emile

I managed to solve it myself! The hint to dynamic programming by #EgorSkriptunoff did the trick!
local table1 = {2, 5, 0 ,1}
local table2 = {5, nil, 3, 2}
local solution = {}
local smallestsolution = {}
solution[1]={}
solution[2]={}
solution[3]={}
solution[4]={}
local path = {}
local temp = {}
for i = 1, (#table1-1) do
solution[1][i] = math.sqrt((table1[i+1]-table1[i])^2)
if table2[i] ~= nil then solution[2][i] = math.sqrt((table1[i+1]-table2[i])^2) end
if table2[i+1] ~= nil then solution[3][i] = math.sqrt((table2[i+1]-table1[i])^2) end
if table2[i] ~= nil and table2[i+1] ~= nil then solution[4][i] = math.sqrt((table2[i+1]-table2[i])^2) end
end
for i = 1, (#table1-1) do
smallestsolution[i]=100000
temp[i] = 0
for j = 1, 4 do
if solution[j][i] ~= nil and solution[j][i] < smallestsolution[i] then smallestsolution[i]=solution[j][i] temp[i] = j end
end
end
local smallestsum = 0
for i = 1, (#table1-1) do
smallestsum = smallestsum + smallestsolution[i]
end
for i = 1, (#table1) do -- find the path belonging to the smallest sum of differences
if temp[i] == 1 then path[i] = table1[i] end
if temp[i] == 2 then path[i] = table2[i] end
if temp[i] == 3 then path[i] = table1[i] end
if temp[i] == 4 then path[i] = table2[i] end
if i == (#table1) then
if temp[i-1] == 1 then path[i] = table1[i] end
if temp[i-1] == 2 then path[i] = table1[i] end
if temp[i-1] == 3 then path[i] = table2[i] end
if temp[i-1] == 4 then path[i] = table2[i] end
end
end

Related

Generating list of integers with given number of bit set and sum of bit indices

I would like to generate in an efficient way a list of integers (preferably ordered)
with the following defining properties:
All integers have the same number of bit set N.
All integers have the same sum of bit indices K.
To be definite, for an integer I
its binary representation is:
$I=\sum_{j=0}^M c_j 2^j$ where $c_j=0$ or $1$
The number of bit sets is:
$N(I)=\sum_{j=0}^M c_j$
The sum of bit indices is:
$K(I)=\sum_{j=0}^M j c_j$
I have an inefficient way to generate the list as follows:
make a do/for loop over integers incrementing by use
of a "snoob" function - smallest next integer with same number of bit set
and at each increment checking if it has the correct value of K
this is grossly inefficient because in general starting from an integer
with the correct N and K value the snoob integer from I does not have the correct K and one has to make many snoob calculations to get the next integer
with both N and K equal to the chosen values.
Using snoob gives an ordered list which is handy for dichotomic search but
not absolutely compulsory.
Counting the number of elements in this list is easily done by recursion
when viewed as a partition numner counting. here is a recursive function in fortran 90 doing that job:
=======================================================================
recursive function BoundedPartitionNumberQ(N, M, D) result (res)
implicit none
! number of partitions of N into M distinct integers, bounded by D
! appropriate for Fermi counting rules
integer(8) :: N, M, D, Nmin
integer(8) :: res
Nmin = M*(M+1)/2 ! the Fermi sea
if(N < Nmin) then
res = 0
else if((N == Nmin) .and. (D >= M)) then
res = 1
else if(D < M) then
res = 0
else if(D == M) then
if(N == Nmin) then
res = 1
else
res = 0
endif
else if(M == 0) then
res = 0
else
res = BoundedPartitionNumberQ(N-M,M-1,D-1)+BoundedPartitionNumberQ(N-M,M,D-1)
endif
end function BoundedPartitionNumberQ
========================================================================================
My present solution is inefficient when I want to generate lists with several $10^7$
elements. Ultimately I want to stay within the realm of C/C++/Fortran and reach lists of lengths
up to a few $10^9$
my present f90 code is the following:
program test
implicit none
integer(8) :: Nparticles
integer(8) :: Nmax, TmpL, CheckL, Nphi
integer(8) :: i, k, counter
integer(8) :: NextOne
Nphi = 31 ! word size is Nphi+1
Nparticles = 16 ! number of bit set
print*,Nparticles,Nphi
Nmax = ishft(1_8, Nphi + 1) - ishft(1_8, Nphi + 1 - Nparticles)
i = ishft(1, Nparticles) - 1
counter = 0
! integer CheckL is the sum of bit indices
CheckL = Nparticles*Nphi/2 ! the value of the sum giving the largest list
do while(i .le. Nmax) ! we increment the integer
TmpL = 0
do k=0,Nphi
if (btest(i,k)) TmpL = TmpL + k
end do
if (TmpL == CheckL) then ! we check whether the sum of bit indices is OK
counter = counter + 1
end if
i = NextOne(i) ! a version of "snoob" described below
end do
print*,counter
end program
!==========================================================================
function NextOne (state)
implicit none
integer(8) :: bit
integer(8) :: counter
integer(8) :: NextOne,state,pstate
bit = 1
counter = -1
! find first one bit
do while (iand(bit,state) == 0)
bit = ishft(bit,1)
end do
! find next zero bit
do while (iand(bit,state) /= 0)
counter = counter + 1
bit = ishft(bit,1)
end do
if (bit == 0) then
print*,'overflow in NextOne'
NextOne = not(0)
else
state = iand(state,not(bit-1)) ! clear lower bits i &= (~(bit-1));
pstate = ishft(1_8,counter)-1 ! needed by IBM/Zahir compiler
! state = ior(state,ior(bit,ishft(1,counter)-1)) ! short version OK with gcc
state = ior(state,ior(bit,pstate))
NextOne = state
end if
end function NextOne
Since you mentioned C/C++/Fortran, I've tried to keep this relatively language agnostic/easily transferable but have also included faster builtins alternatives where applicable.
All integers have the same number of bit set N
Then we can also say, all valid integers will be permutations of N set bits.
First, we must generate the initial/min permutation:
uint32_t firstPermutation(uint32_t n){
// Fill the first n bits (on the right)
return (1 << n) -1;
}
Next, we must set the final/max permutation - indicating the 'stop point':
uint32_t lastPermutation(uint32_t n){
// Fill the last n bits (on the left)
return (0xFFFFFFFF >> n) ^ 0xFFFFFFFF;
}
Finally, we need a way to get the next permutation.
uint32_t nextPermutation(uint32_t n){
uint32_t t = (n | (n - 1)) + 1;
return t | ((((t & -t) / (n & -n)) >> 1) - 1);
}
// or with builtins:
uint32_t nextPermutation(uint32_t &p){
uint32_t t = (p | (p - 1));
return (t + 1) | (((~t & -~t) - 1) >> (__builtin_ctz(p) + 1));
}
All integers have the same sum of bit indices K
Assuming these are integers (32bit), you can use this DeBruijn sequence to quickly identify the index of the first set bit - fsb.
Similar sequences exist for other types/bitcounts, for example this one could be adapted for use.
By stripping the current fsb, we can apply the aforementioned technique to identify index of the next fsb, and so on.
int sumIndices(uint32_t n){
const int MultiplyDeBruijnBitPosition[32] = {
0, 1, 28, 2, 29, 14, 24, 3, 30, 22, 20, 15, 25, 17, 4, 8,
31, 27, 13, 23, 21, 19, 16, 7, 26, 12, 18, 6, 11, 5, 10, 9
};
int sum = 0;
// Get fsb idx
do sum += MultiplyDeBruijnBitPosition[((uint32_t)((n & -n) * 0x077CB531U)) >> 27];
// strip fsb
while (n &= n-1);
return sum;
}
// or with builtin
int sumIndices(uint32_t n){
int sum = 0;
do sum += __builtin_ctz(n);
while (n &= n-1);
return sum;
}
Finally, we can iterate over each permutation, checking if the sum of all indices matches the specified K value.
p = firstPermutation(n);
lp = lastPermutation(n);
do {
p = nextPermutation(p);
if (sumIndices(p) == k){
std::cout << "p:" << p << std::endl;
}
} while(p != lp);
You could easily change the 'handler' code to do something similar starting at a given integer - using it's N & K values.
A basic recursive implementation could be:
void listIntegersWithWeight(int currentBitCount, int currentWeight, uint32_t pattern, int index, int n, int k, std::vector<uint32_t> &res)
{
if (currentBitCount > n ||
currentWeight > k)
return;
if (index < 0)
{
if (currentBitCount == n && currentWeight == k)
res.push_back(pattern);
}
else
{
listIntegersWithWeight(currentBitCount, currentWeight, pattern, index - 1, n, k, res);
listIntegersWithWeight(currentBitCount + 1, currentWeight + index, pattern | (1u << index), index - 1, n, k, res);
}
}
That is not my suggestion, just the starting point. On my PC, for n = 16, k = 248, both this version and the iterative version take almost (but not quite) 9 seconds. Almost exactly the same amount of time, but that's just a coincidence. More pruning can be done:
currentBitCount + index + 1 < n if the number of set bits cannot reach n with the number of unfilled positions that are left, continuing is pointless.
currentWeight + (index * (index + 1) / 2) < k if the sum of positions cannot reach k, continuing is pointless.
Together:
void listIntegersWithWeight(int currentBitCount, int currentWeight, uint32_t pattern, int index, int n, int k, std::vector<uint32_t> &res)
{
if (currentBitCount > n ||
currentWeight > k ||
currentBitCount + index + 1 < n ||
currentWeight + (index * (index + 1) / 2) < k)
return;
if (index < 0)
{
if (currentBitCount == n && currentWeight == k)
res.push_back(pattern);
}
else
{
listIntegersWithWeight(currentBitCount, currentWeight, pattern, index - 1, n, k, res);
listIntegersWithWeight(currentBitCount + 1, currentWeight + index, pattern | (1u << index), index - 1, n, k, res);
}
}
On my PC with the same parameters, this only takes half a second. It can probably be improved further.

How can I write this algorithm using iteration?

How can I write this algorithm using iteration?
function generate(num1:byval)
if num1 > 10 then
return 10
else
return num1 + (generate(num1 + 1) DIV 2)
endif
endfunction
So it's not straight forward so I start by doing some grunt work:
n result
11.. 10
10 10
9 9 + 10/2
8 8 + (9 + 10/2)/2
7 7 + (8 + (9 + 10/2)/2)/2
This looks like a pattern.. While the recursive version started on the input and went upwards it's easy to see that by starting at 10 and going downwards one can simply update the accumulator by halving the value and adding the current value.
This can easily be implemented using a helper:
procedure generate(num : integer) : integer
begin
generate := generateHelper(10, num, 0)
end
procedure generateHelper(cur : integer, num: integer, acc : integer) : integer
begin
if cur = num then
generateHelper := cur + acc/2;
else
generateHelper := generateHelper(cur - 1, num, acc/2 + cur);
end
Or by using a loop:
procedure generate(num : integer) : integer
var cur, acc : integer;
begin
for cur := 10 downto cur do
acc := acc / 2 + cur;
generate := acc;
end
If you work out some values for the function…
f(10) = 10
f(9) = 9+f(10)/2 = 9+10/2 = 14
f(8) = 8+f(9)/2 = 8+14/2 = 15
…
You will get a sense that you could repeatedly apply the same formula to a value in a loop. You see if you start from 10, you divide by 2 and add 9, then divide by 2 and add 8, and keep going until you reach the number given to the function. That would look something like this, e.g. in JavaScript:
function generate(n) {
let x = 10;
for(let i = 10; i > n; i--) {
x = i - 1 + x / 2;
}
return x;
}

error bernstein vandermonde julia.

the following error occurs. I tried to change the n .... but not working
"LoadError: BoundsError: attempt to access 9-element Array{Float64,1}:"
function bernstein_vandermonde( n )
if n == 1
v = ones(1, 1);
return v
end
v = zeros( n, n );
x = linspace( 0, 1, n );
for i = 1:n
println("entra no loop")
v[i,1:n] = bernstein_poly_01(n - 1, x[i])
end
return v
end
function bernstein_poly_01( n, x )
bern = ones(n)
if n == 0
bern[1] = 1
elseif 0 < n
bern[1] = 1 -x
bern[2] = x
for i = 2:n
bern[i+1] = x*bern[i];
for j = i-1:-1: 1
bern[j+1] = x*bern[j] + (1 - x)*bern[j+1]
end
bern[1] = (1 - x)*bern[1]
end
end
return bern
end
I can not solve :(

recover index in triangular for loops

Is there a simple way to recover an index in nested for loops? For example, in for loops which construct Pascals triangle
int index = 0;
for (int i = 0; i < N; ++i)
for (int j = 0; j < N-i; ++j)
index++;
is there a way to recover i and j given only index?
I am adding this as a second answer since it is in a different language (now C) and has a more direct approach. I am keeping the original answer since the following code is almost inexplicable without it. I combined my two functions into a single one to cut down on function call overhead. Also, to be 100% sure that it answers the original question, I used the loops from that question verbatim. In the driver function I show explicitly that the output is correct for N = 4 and then stress-test it for N = 10000 (with a total of 100,000,000 passes through the inner loop). I don't have any formal timing code, but it takes about 1 second on my machine to run through and test those 100 million cases. My code assumes a 32-bit int. Change to long if needed:
#include <stdio.h>
#include <math.h>
void from_index(int n, int index, int *i, int *j);
int main(void){
int N;
int ri,rj; //recovered i,j
N = 4;
int index = 0;
for (int i = 0; i < N; ++i)
for (int j = 0; j < N-i; ++j){
from_index(N,index,&ri,&rj);
printf("i = %d, j = %d, index = %d, ",i,j,index);
printf("recovered i = %d, recovered j = %d\n",ri,rj);
index++;
}
//stress test:
N = 10000;
index = 0;
for (int i = 0; i < N; ++i)
for (int j = 0; j < N-i; ++j){
from_index(N,index,&ri,&rj);
if(i != ri || j != rj){
printf("Don't post buggy code to Stack Overflow!\n");
printf("(i,j) = (%d,%d) but recovered indices are (%d,%d)\n",i,j,ri,rj);
return 0;
}
index++;
}
printf("\nAll %d tests passed!\n",N*N);
return 0;
}
void from_index(int n, int index, int *i, int *j){
double d;
d = 4*n*(n+1) - 7 - 8 * index;
*i = floor((-1 + sqrt(d))/2);
*j = *i * (*i + 1)/2;
*j = n*(n+1)/2 - 1 - index - *j;
*j = *i - *j;
*i = n - *i - 1;
}
Output:
i = 0, j = 0, index = 0, recovered i = 0, recovered j = 0
i = 0, j = 1, index = 1, recovered i = 0, recovered j = 1
i = 0, j = 2, index = 2, recovered i = 0, recovered j = 2
i = 0, j = 3, index = 3, recovered i = 0, recovered j = 3
i = 1, j = 0, index = 4, recovered i = 1, recovered j = 0
i = 1, j = 1, index = 5, recovered i = 1, recovered j = 1
i = 1, j = 2, index = 6, recovered i = 1, recovered j = 2
i = 2, j = 0, index = 7, recovered i = 2, recovered j = 0
i = 2, j = 1, index = 8, recovered i = 2, recovered j = 1
i = 3, j = 0, index = 9, recovered i = 3, recovered j = 0
All 100000000 tests passed!
In this particular case we have
index = N+(N-1)+...+(N-i+1) + (j+1) = i(2N-i+1)/2 + (j+1) = -i^i/2 + (2N-1)i/2 + (j+1)
with j in the interval [1,N-i].
We neglect j and regard this as a quadratic equation in i. Thus we solve
-i^i/2 + (2N-1)i/2 + (1-index) = 0.
We approximate i to be the greatest out of the two resulting solutions (or the ceil of this value, since neglecting j has the effect of lowering the value of i).
We then come back to the complete version of the equation and substitute the approximation of the value of i. If j is outside the interval [1,N-i] we increase/decrease the value of i and re-substitute until we get a value of j in this interval. This loop will probably repeat for a maximum constant number of steps (I suspect a maximum of three steps, but not in the mood to prove it). So this should be doable in a constant number of steps.
As an alternative, we could approximate j to be N/3, instead of zero. This is approximately the expected value of j (over all possible cases), thus the method will probably converge 'faster' at the local search step.
In the general case, you do something very similar, i.e. you solve a fake equation and you perform a local search around the solution.
I found it easier to find i,j from the index in the following number pattern:
0
1 2
3 4 5
6 7 8 9
Since the indices going down the left are the triangular numbers of the form k*(k+1)/2. By solving an appropriate quadratic equation I was able to recover the row and the column from the index. But -- your loops give something like this:
0 1 2 3
4 5 6
7 8
9
which is trickier. It might be possible to solve this problem directly, but note that if you subtract each of these numbers from 9 you get
9 8 7 6
5 4 3
2 1
0
this is the original triangle turned upside down and reflected horizontally. Thus -- I can reduce the problem of your triangle to my triangle. The following Python code shows how it works (the only thing not quite obvious is that in Python 3 // is integer division). The function fromIndexHelper is my solution to my original triangle problem and fromIndex is how I shift it to your triangle. To test it I first printed the index pattern for n = 4 and then the corresponding indices recovered by my function fromIndex:
from math import floor, sqrt
def fromIndexHelper(n,index):
i = floor((-1+sqrt(1+8*index))/2)
j = index - i*(i+1)//2
return i,j
def fromIndex(n,index):
shift = n*(n+1)//2 - 1
i,j = fromIndexHelper(n,shift-index)
return n-i-1,i - j
#test
index = 0
for i in range(4):
for j in range(4-i):
print(index,end = ' ')
index +=1
print('')
print(' ')
index = 0
for i in range(4):
for j in range(4-i):
print(fromIndex(4,index),end = ' ')
index +=1
print('')
Output:
0 1 2 3
4 5 6
7 8
9
(0, 0) (0, 1) (0, 2) (0, 3)
(1, 0) (1, 1) (1, 2)
(2, 0) (2, 1)
(3, 0)

How to modify drawdown functions in PerformanceAnalytics package for value

I am calculating the average drawdown, average length, recovery length, etc. in R for a PnL data series rather than return data. This is data frame like this
PNL
2008-11-03 3941434
2008-11-04 4494446
2008-11-05 2829608
2008-11-06 2272070
2008-11-07 -2734941
2008-11-10 -2513580
I used the maxDrawDown function from fTrading package and it worked. How could I get the other drawdown functions? If I directly run AverageDrawdown(quantbook) function, it will give out error message like this
Error in if (thisSign == priorSign) { : missing value where TRUE/FALSE needed
I checked the documentation for AverageDrawdown and it is as below:
findDrawdowns(R, geometric = TRUE, ...)
R an xts, vector, matrix, data frame, timeSeries or zoo object of asset returns
My quantbook is a data frame but doesn't work for this function.
Or do you have anything other packages to get the same funciton, please advise.
I've modified the package's functions. Here is one solution in PnL case (or any other case you want to get the value rather than the return) and hope you find it useful. The parameter x is a dataframe and the row.names for x are dates so you don't bother to convert amongst different data types (which I actually suffer a lot). With the function findPnLDrawdown, you could perform a lot other functions to calculate averageDrawDown, averageLength, recovery, etc.
PnLDrawdown <- function(x) {
ts = as.vector(x[,1])
cumsum = cumsum(c(0, ts))
cmaxx = cumsum - cummax(cumsum)
cmaxx = cmaxx[-1]
cmaxx = as.matrix(cmaxx)
row.names(cmaxx) = row.names(x)
cmaxx = timeSeries(cmaxx)
cmaxx
}
findPnLDrawdown <- function(R) {
drawdowns = PnLDrawdown(R)
draw = c()
begin = c()
end = c()
length = c(0)
trough = c(0)
index = 1
if (drawdowns[1] >= 0) {
priorSign = 1
} else {
priorSign = 0
}
from = 1
sofar = as.numeric(drawdowns[1])
to = 1
dmin = 1
for (i in 1:length(drawdowns)) {
thisSign =ifelse(drawdowns[i] < 0, 0, 1)
if (thisSign == priorSign) {
if (as.numeric(drawdowns[i]) < as.numeric(sofar)) {
sofar = drawdowns[i]
dmin = i
}
to = i+ 1
}
else {
draw[index] = sofar
begin[index] = from
trough[index] = dmin
end[index] = to
from = i
sofar = drawdowns[i]
to = i + 1
dmin = i
index = index + 1
priorSign = thisSign
}
}
draw[index] = sofar
begin[index] = from
trough[index] = dmin
end[index] = to
list(pnl = draw, from = begin, trough = trough, to = end,
length = (end - begin + 1),
peaktotrough = (trough - begin + 1),
recovery = (end - trough))
}

Resources