Related
New to Spark, and new to Ada, so this question may be overly broad. However, it's asked in good faith, as part of an attempt to understand Spark. Besides direct answers to the questions below, I welcome critique of style, workflow, etc.
As my first foray into Spark, I chose to try to implement (easy) and prove correctness (unsuccessful so far) the function .
Question: What is the proper way of implementing and proving the correctness of this function?
I started with the following util.ads:
package Util is
function Floor_Log2(X : Positive) return Natural with
Post => 2**Floor_Log2'Result <= X and then X < 2**(Floor_Log2'Result + 1);
end Util;
I have no pre-condition because the ranges of the input fully expresses the only interesting pre-condition. The post-condition I wrote based on the mathematical definition; however, I have an immediate concern here. If X is Positive'Last, then 2**(Floor_Log2'Result + 1) exceeds Positive'Last and Natural'Last. Already I'm up against my limited knowledge of Ada here, so: Sub-question 1: What is the type of the sub-expression in the post condition, and is this overflow a problem? Is there a general way to resolve it? To avoid the issue in this particular case, I revised the specification to the less-intuitive but equivalent:
package Util is
function Floor_Log2(X : Positive) return Natural with
Post => 2**Floor_Log2'Result <= X and then X/2 < 2**Floor_Log2'Result;
end Util;
There are many ways to implement this function, and I'm not particularly concerned about performance at this point, so I'd be happy with any of them. I'd consider the "natural" implementation (given my particular C background) to be something like the following util.adb:
package body Util is
function Floor_Log2 (X : Positive) return Natural is
I : Natural := 0;
Remaining : Positive := X;
begin
while Remaining > 1 loop
I := I + 1;
Remaining := Remaining / 2;
end loop;
return I;
end Floor_Log2;
end Util;
Attempting to prove this with no loop invariants fails, as expected. Results (this and all results are GNATprove level 4, invoked from GPS as gnatprove -P%PP -j0 %X --ide-progress-bar -u %fp --level=4 --report=all):
util.adb:6:13: info: initialization of "Remaining" proved[#2]
util.adb:7:15: info: initialization of "I" proved[#0]
util.adb:7:17: medium: overflow check might fail[#5]
util.adb:8:23: info: initialization of "Remaining" proved[#1]
util.adb:8:33: info: range check proved[#4]
util.adb:8:33: info: division check proved[#8]
util.adb:10:14: info: initialization of "I" proved[#3]
util.ads:3:14: medium: postcondition might fail, cannot prove 2**Floor_Log2'Result <= X[#7]
util.ads:3:15: medium: overflow check might fail[#9]
util.ads:3:50: info: division check proved[#6]
util.ads:3:56: info: overflow check proved[#10]
Most of the errors here make basic sense to me. Starting with the first overflow check, GNATprove cannot prove that the loop terminates in less than Natural'Last iterations (or at all?), so it cannot prove that I := I + 1 doesn't overflow. We know that this isn't the case, because Remaining is decreasing. I tried to express this adding the loop variant pragma Loop_Variant (Decreases => Remaining), and GNATprove was able to prove that loop variant, but the potential overflow of I := I + 1 is unchanged, presumedly because proving the loop terminates at all is not equivalent to proving that it terminates in less than Positive'Last iterations. A tighter constraint would show that the loop terminates in at most Positive'Size iterations, but I'm not sure how to prove that. Instead, I "forced it" by adding a pragma Assume (I <= Remaining'Size); I know this is bad practice, the intent here was purely to let me see how far I could get with this first issue "swept under the covers." As expected, this assumption lets the prover prove all range checks in the implementation file. Sub-question 2: What is the correct way to prove that I does not overflow in this loop?
However, we've still made no progress on proving the postcondition. A loop invariant is clearly needed. One loop invariant that holds at the top of the loop is that pragma Loop_Invariant (Remaining * 2**I <= X and then X/2 < Remaining * 2**I); besides being true, this invariant has the nice property that it is clearly equivalent to the post-condition when the loop termination condition is true. However, as expected GNATprove is unable to prove this invariant: medium: loop invariant might fail after first iteration, cannot prove Remaining * 2**I <= X[#20]. This makes sense, because the inductive step here is non-obvious. With division on the real numbers one could imagine a straightforward lemma stating that for all I, X * 2**I = (X/2) * 2**(I+1), but (a) I don't expect GNATprove to know that without a lemma being provided, and (b) it's messier with integer division. So, Sub-Question 3a: Is this the appropriate loop invariant to try to use to prove this implementation? Sub-Question 3b: If so, what's the right way to prove it? Externally prove a lemma and use that? If so, what exactly does that mean?
At this point, I thought I'd explore a completely different implementation, just to see if it led anywhere different:
package body Util is
function Floor_Log2 (X : Positive) return Natural is
begin
for I in 1 .. X'Size - 1 loop
if 2**I > X then
return I - 1;
end if;
end loop;
return X'Size - 1;
end Floor_Log2;
end Util;
This is a less intuitive implementation to me. I didn't explore this second implementation as much, but I leave it here to show what I tried; to give a potential avenue for other solutions to the main question; and to raise additional sub-questions.
The idea here was to bypass some of the proof around overflow of I and termination conditions by making termination and ranges explicit. Somewhat to my surprise, the prover first choked on overflow checking the expression 2**I. I had expected 2**(X'Size - 1) to be provably within the bounds of X -- but again, I'm up against the limits of my Ada knowledge. Sub-Question 4: Is this expression actually overflow-free in Ada, and how can that be proven?
This has turned out to be a long question... but I think the questions I'm raising, in the context of a nearly-trivial example, are relatively general and likely to be useful to others who, like me, are trying to understand if and how Spark is relevant to them.
I can't help with your SPARK questions, but I can answer some of your Sub-Questions.
Sub-Question 1: Since you're using "<" for Integer, the sub-expression will be of type Integer as well. For Positive'Last (2 ** 31 - 1 with GNAT), your function result should be 30, and the sub-expression will overflow. (This is from a SPARK point of view; compilers are allowed to use larger ranged types when evaluating expressions to obtain the mathematically/logically correct result even if a sub-expression would overflow, and GNAT will do this for some values of -gnato.)
Sub-Question 4: 2 ** (X'Size - 1) can overflow. The reason has to do with the 2 meanings of 'Size: Positive'Size is the minimum number of bits needed to store a value of subtype Positive; X'Size is the actual number of bits allocated to X. Since you're using GNAT,
Integer'Last = Positive'Last = 2 ** 31 - 1. X'Size = 32. Positive'Size = 31.
So, 2 ** (X'Size - 1) = 2 ** 31 > Positive'Last. You probably want to use Positive'Size instead of X'Size.
(Again, from the SPARK point of view; compilers are allowed to obtain the logically correct result.)
Aside: the short-circuit forms and then and or else should only be used when they're actually needed. Modern processors do all sorts of optimizations at the machine-code level that have to be turned off for short-circuit evaluation. Although they may look like optimizations, in practice they are often the opposite.
HTH.
(You might want to tag this with [ada]. I only saw it because you referenced it in c.l.ada.)
Preventing overflow in the post condition
Given the original function signature
function Floor_Log2 (X : Positive) return Natural with
Post => 2**Floor_Log2'Result <= X and then X < 2**(Floor_Log2'Result + 1);
I observe that I need to limit the domain of X in order to prevent overflow in the second term of the post condition. Given the definitions in Standard.ads, i.e.
type Integer is range -(2**31) .. +(2**31 - 1);
for Integer'Size use 32;
subtype Natural is Integer range 0 .. Integer'Last;
subtype Positive is Integer range 1 .. Integer'Last;
I conclude that, in order to prevent overflow,
X < 2**(Floor_Log2'Result + 1) <= 2**31 - 1
and therefore X <= 2**30 - 1. Hence, I changed the function signature to:
subtype Pos is Positive range 1 .. 2**30 - 1
function Floor_Log2 (X : Pos) return Natural with
Post => 2**Floor_Log2'Result <= X and then X < 2**(Floor_Log2'Result + 1);
First Approach
In principle, I could now proof the post condition as follows in GNAT CE 2019 (note that I use a different algorithm compared to the one stated in the question):
util.ads
package Util with SPARK_Mode is
subtype Pos is Positive range 1 .. 2**30 - 1
function Floor_Log2 (X : Pos) return Natural with
Post => 2**Floor_Log2'Result <= X and then X < 2**(Floor_Log2'Result + 1);
end Util;
util.adb
package body Util with SPARK_Mode is
----------------
-- Floor_Log2 --
----------------
function Floor_Log2 (X : Pos) return Natural is
L : Positive := 1;
H : Positive := L * 2;
I : Natural := 0;
begin
while not (L <= X and then X < H) loop
pragma Loop_Invariant
(L = 2 ** I and H = 2 ** (I+1));
pragma Loop_Invariant
(for all J in 0 .. I =>
not (2 ** J <= X and then X < 2 ** (J+1)));
L := H;
H := H * 2;
I := I + 1;
end loop;
return I;
end Floor_Log2;
end Util;
Unfortunately, however, the provers have difficulties with the non-linear arithmetic (i.e. exponentiation) and all proof sessions (on my computer) end with a timeout. In fact, if I run gnatprove with effort level 0, then I can only proof the post condition when I limit the upper bound of Pos to 2**7 - 1, i.e.
subtype Pos is Positive range 1 .. 2**7 - 1;
Increasing the effort level (or timeout) allows me to proof the post condition for larger values of Pos'Last.
Second Approach
In order to work around the limitation of the provers, I applied a little trick by redefining the exponentiation function. I could then use the following code to prove the post condition for the full range of Pos when I run gnatprove with effort level 1:
spark_exp.ads
generic
type Int is range <>;
Base : Int;
N_Max : Natural;
package SPARK_Exp with SPARK_Mode is
subtype Exp_T is Natural range 0 .. N_Max;
function Exp (N : Exp_T) return Int with Ghost;
private
type Seq_T is array (Exp_T range <>) of Int;
function Exp_Seq return Seq_T with
Ghost,
Post => (Exp_Seq'Result'First = 0)
and then (Exp_Seq'Result'Last = N_Max)
and then (Exp_Seq'Result (0) = 1)
and then (for all I in 1 .. N_Max =>
Exp_Seq'Result (I) = Base * Exp_Seq'Result (I - 1) and
Int'First < Exp_Seq'Result (I) and Exp_Seq'Result (I) < Int'Last);
function Exp (N : Exp_T) return Int is (Exp_Seq (N));
end SPARK_Exp;
spark_exp.adb
package body SPARK_Exp with SPARK_Mode is
-------------
-- Exp_Seq --
-------------
function Exp_Seq return Seq_T is
S : Seq_T (Exp_T'Range) := (others => 1);
begin
for I in 1 .. N_Max loop
pragma Loop_Invariant
(for all J in 1 .. I - 1 =>
S (J) = Base * S (J - 1) and
(Int'First / Base) < S (J) and S (J) < (Int'Last / Base));
S (I) := Base * S (I - 1);
end loop;
return S;
end Exp_Seq;
end SPARK_Exp;
util.ads
with SPARK_Exp;
package Util with SPARK_Mode is
subtype Pos is Positive range 1 .. 2**30 - 1;
package SPARK_Exp_2 is
new SPARK_Exp (Positive, 2, 30);
function Exp2 (N : SPARK_Exp_2.Exp_T) return Positive
renames SPARK_Exp_2.Exp;
function Floor_Log2 (X : Pos) return Natural with
Post => (Exp2 (Floor_Log2'Result) <= X) and then
(X < Exp2 (Floor_Log2'Result + 1));
end Util;
util.adb
package body Util with SPARK_Mode is
----------------
-- Floor_Log2 --
----------------
function Floor_Log2 (X : Pos) return Natural is
L : Positive := 1;
H : Positive := L * 2;
I : Natural := 0;
begin
while not (L <= X and then X < H) loop
pragma Loop_Invariant
(L = Exp2 (I) and H = Exp2 (I + 1));
pragma Loop_Invariant
(for all J in 0 .. I =>
not (Exp2 (J) <= X and then X < Exp2 (J + 1)));
L := H;
H := H * 2;
I := I + 1;
end loop;
return I;
end Floor_Log2;
end Util;
This implementation proves all checks within the body, but the preconditions are still not proven:
package body Util is
pragma SPARK_Mode (On);
function Floor_Log2 (X : Positive) return Natural is
I : Natural := 30;
Prod : Natural := 2**30;
type Large_Natural is range 0 .. 2**31;
Prev_Prod : Large_Natural := Large_Natural'Last with Ghost;
begin
while I > 0 loop
if X >= Prod then
pragma Assert (Large_Natural (X) < Prev_Prod);
return I;
end if;
pragma Loop_Invariant (I > 0);
pragma Loop_Invariant (Prod >= X and Prev_Prod >= Large_Natural (X));
-- pragma Loop_Invariant (2**I > X);
Prod := Prod / 2;
I := I - 1;
end loop;
pragma Assert (I = 0);
return 0;
end Floor_Log2;
end Util;
This gives the following output with gnatprove:
gnatprove -P/Users/pnoffke/projects/ada/spark/floor_log2/floor_log2.gpr -j0 --ide-progress-bar -u util.adb --level=2 --report=all
Phase 1 of 2: generation of Global contracts ...
Phase 2 of 2: flow analysis and proof ...
util.adb:10:13: info: initialization of "I" proved
util.adb:11:18: info: initialization of "Prod" proved
util.adb:12:28: info: assertion proved
util.adb:12:48: info: initialization of "Prev_Prod" proved
util.adb:13:20: info: initialization of "I" proved
util.adb:15:33: info: initialization of "I" proved
util.adb:15:33: info: loop invariant preservation proved
util.adb:15:33: info: loop invariant initialization proved
util.adb:16:33: info: initialization of "Prod" proved
util.adb:16:33: info: loop invariant preservation proved
util.adb:16:33: info: loop invariant initialization proved
util.adb:16:47: info: initialization of "Prev_Prod" proved
util.adb:18:18: info: initialization of "Prod" proved
util.adb:18:23: info: division check proved
util.adb:19:15: info: initialization of "I" proved
util.adb:19:17: info: range check proved
util.adb:22:22: info: initialization of "I" proved
util.adb:22:22: info: assertion proved
util.ads:5:15: info: overflow check proved
util.ads:5:44: medium: postcondition might fail, cannot prove X / 2 < 2**Floor_Log2'result (e.g. when Floor_Log2'Result = 0 and X = 2)
util.ads:5:46: info: division check proved
util.ads:5:53: medium: overflow check might fail (e.g. when Floor_Log2'Result = 30)
I don't understand why gnatprove cannot prove the commented Loop_Invariant. If I try to do so, I get the following additional output:
util.adb:17:33: medium: loop invariant might fail after first iteration, cannot prove 2**I > X (e.g. when I = 0 and X = 0)
util.adb:17:33: medium: loop invariant might fail in first iteration, cannot prove 2**I > X (e.g. when I = 30 and X = 1)
util.adb:17:34: medium: overflow check might fail (e.g. when I = 0)
In the counter-example, it says "when I = 0 and X = 0", but I cannot be 0 per the first Loop_Invariant.
Also if I initialize Prod to 2**I instead of 2**30, I get:
util.adb:6:26: medium: overflow check might fail (e.g. when I = 30 and Prod = 0)
I suspect gnatprove has some fundamental issue with the ** operator. I was hoping to use Prev_Prod to help prove your preconditions, but I don't see how to get there with the above issues I'm having.
I am trying to code a Periodic Vehicle Routing Problem with some inventory constraints in AMPL. I would like to add the subtour constraints dynamically. In order to do this i was inspired by this formulation for a TSP:
https://groups.google.com/d/msg/ampl/mVsFg4mAI1c/ZdfRHHRijfUJ
However, I can not get it to eliminate subtours in my model. I used the following in my model file.
param T; # Number of time-periods
param V; # Number of vehicles
param F; # Number of fuel types
set P ordered; # Number of gas stations
param hpos {P} >= 0;
param vpos {P} >= 0;
set PAIRS := {p in P, j in P};
param dist {(p,j) in PAIRS}
:= sqrt((hpos[j]-hpos[p])**2 + (vpos[j]-vpos[p])**2);
# A binary variable to determine if an arc is traversed.
var H{(p,j) in PAIRS, v in 1..V, t in 1..T} binary;
# A binary variable to determine if a delivery of fuel is made to a station in a given time period.
var StationUsed{p in P, f in 1..F, v in 1..V, t in 1..T} binary;
minimize TransportationCost:
sum {(p,j) in PAIRS} sum {v in 1..V, t in 1..T} dist[p,j] * H[p,j,v,t];
param nSubtours >= 0 integer;
set SUB {1..nSubtours} within P;
subject to Subtour_Elimination {k in 1..nSubtours, m in SUB[k], v in 1..V, t in 1..T, f in 1..F}:
sum {p in SUB[k], j in P diff SUB[k]}
if (p,j) in PAIRS then H[p,j,v,t] else H[j,p,v,t] >=2 * StationUsed[m,f,v,t] ;
I added the StationUsed variable, as my problem unlike TSP does not have to visit all nodes in every timeperiod. H is my binary decision variable declaring if vehicle travels the arc (p,j) in a time period.
Then I used a formulation similar to the TSP in my run file:
set NEWSUB;
set EXTEND;
let nSubtours := 0;
repeat {
solve;
let NEWSUB := {};
let EXTEND := {member(ceil(Uniform(0,card(P))),P)};
repeat {
let NEWSUB := NEWSUB union EXTEND;
let EXTEND := {j in P diff NEWSUB: exists {p in NEWSUB, v in 1..V, t in 1..T}
((p,j) in PAIRS and H[p,j,v,t] = 1 or (j,p) in PAIRS and H[j,p,v,t] = 1)};
} until card(EXTEND) = 0;
if card(NEWSUB) < card(P) then {
let nSubtours := nSubtours + 1;
let SUB[nSubtours] := NEWSUB;
display SUB;
} else break;
};
# Display the routes
display {t in 1..T, v in 1..V}: {(p,j) in PAIRS} H[p,j,v,t];
I am not sure if the above is applicable to my problem with multiple vehicles and multiple time periods. I have tried defining v and t in let EXTEND, at it is needed to use H, but I am not sure if this is a correct method. My models runs, when formulated as above, however it does not eliminate the subtours. Do you guys have any suggestions in this regard?
ADDED QUESTION:
I found some inspiration in this model formulated in SAS/OR:
(A bit extensive to read and not necessary for my questions)
http://support.sas.com/documentation/cdl/en/ormpex/67518/HTML/default/viewer.htm#ormpex_ex23_sect009.htm
It eliminates subtours dynamically over d days and I figured it could be translated to my problem with multiple vehicles and multiple periods (days).
To specify my problem a little. A node can only be visited by one vehicle once within a time period. All nodes does not have to be visited in every time period, which is a major difference from the TSP formulation, where all nodes are in the cycle.
I tried with the following approach:
The constraint in the model file is the same as before.
set P ordered; # Number of nodes
set PAIRS := {p in P, j in P: ord(p) != ord(j)};
param nSubtours >= 0 integer;
param iter >= 0 integer;
set SUB {1..nSubtours} within P;
subject to Subtour_Elimination {s in 1..nSubtours, k in SUB[s], f in F, v in V, t in T}:
sum {p in SUB[s], j in P diff SUB[s]}
if (p,j) in PAIRS then H[p,j,v,t] else H[j,p,v,t] >= 2 * StationUsed[k,f,v,t];
My run file looks like this:
let nSubtours := 0;
let iter := 0;
param num_components {V, T};
set P_TEMP;
set PAIRS_SOL {1..iter, V, T} within PAIRS;
param component_id {P_TEMP};
set COMPONENT_IDS;
set COMPONENT {COMPONENT_IDS};
param cp;
param cj;
# loop until each day and each vehicles support graph is connected
repeat {
let iter := iter + 1;
solve;
# Find connected components for each day
for {v in V, t in T} {
let P_TEMP := {p in P: exists {f in F} StationUsed[p,f,v,t] > 0.5};
let PAIRS_SOL[iter, v, t] := {(p,j) in PAIRS: H[p, j, v, t] > 0.5};
# Set each node to its own component
let COMPONENT_IDS := P_TEMP;
let num_components[v, t] := card(P_TEMP);
for {p in P_TEMP} {
let component_id[p] := p;
let COMPONENT[p] := {p};
};
# If p and j are in different components, merge the two component
for {(p,j) in PAIRS_SOL[iter, v, t]} {
let cp := component_id[p];
let cj := component_id[j];
if cp != cj then {
# update smaller component
if card(COMPONENT[cp]) < card(COMPONENT[cj]) then {
for {k in COMPONENT[cp]} let component_id[k] := cj;
let COMPONENT[cj] := COMPONENT[cj] union COMPONENT[cp];
let COMPONENT_IDS := COMPONENT_IDS diff {cp};
} else {
for {k in COMPONENT[cj]} let component_id[k] := cp;
let COMPONENT[cp] := COMPONENT[cp] union COMPONENT[cj];
let COMPONENT_IDS := COMPONENT_IDS diff {cj};
};
};
};
let num_components[v, t] := card(COMPONENT_IDS);
display num_components[v, t];
# create subtour from each component not containing depot node
for {k in COMPONENT_IDS: 1 not in COMPONENT[k]} { . #***
let nSubtours := nSubtours + 1;
let SUB[nSubtours] := COMPONENT[k];
display SUB[nSubtours];
};
};
display num_components;
} until (forall {v in V, t in T} num_components[v,t] = 1);
I get a lot of "invalid subscript discarded", when running the model:
Error at _cmdno 43 executing "if" command
(file amplin, line 229, offset 5372):
error processing set COMPONENT:
invalid subscript COMPONENT[4] discarded.
Error at _cmdno 63 executing "for" command
(file amplin, line 245, offset 5951):
error processing set COMPONENT:
invalid subscript COMPONENT[3] discarded.
(...)
Bailing out after 10 warnings.
I think the script is doing what I am looking for, but it stops, when it has discarded 10 invalid subscripts.
When trying to debug I tested the second for loop.
for {p in P_TEMP} {
let component_id[p] := p;
let COMPONENT[p] := {p};
display component_id[p];
display COMPONENT[p];
};
This is displaying correct, but not before a few errors with "invalid subscript discarded". It seems that p runs through some p not in P_TEMP. For example when P_TEMP is a set consisting of nodes "1 3 4 5", then I get "invalid subscript discarded" for component_id[2] and COMPONENT[2]. My guess is that something similar happens again later on in the IF-ELSE statement.
How do I avoid this?
Thank you,
Kristian
(previous answer text deleted because I misunderstood the implementation)
I'm not sure if this fully explains your issue, but I think there are a couple of problems with how you're identifying subtours.
repeat {
solve;
let NEWSUB := {};
let EXTEND := {member(ceil(Uniform(0,card(P))),P)};
repeat {
let NEWSUB := NEWSUB union EXTEND;
let EXTEND := {j in P diff NEWSUB: exists {p in NEWSUB, v in 1..V, t in 1..T}
((p,j) in PAIRS and H[p,j,v,t] = 1 or (j,p) in PAIRS and H[j,p,v,t] = 1)};
} until card(EXTEND) = 0;
if card(NEWSUB) < card(P) then {
let nSubtours := nSubtours + 1;
let SUB[nSubtours] := NEWSUB;
display SUB;
} else break;
};
What this does:
solves the problem
sets NEWSUB as empty
randomly picks one node from P as the starting point for EXTEND and adds this to NEWSUB
looks for any nodes not currently in NEWSUB which are connected to a node within NEWSUB by any vehicle journey on any day, and adds them to NEWSUB
repeats this process until there are no more to add (i.e. either NEWSUB equals P, the entire set of nodes, or until there are no journeys between NEWSUB and non-NEWSUB notedes)
checks whether NEWSUB is smaller than P (in which case it identifies NEWSUB as a new subtour, appends it to SUB, and goes back to the start).
if NEWSUB has the same size as P (i.e. is equal to P) then it stops.
This should work for a single-vehicle problem with only a single day, but I don't think it's going to work for your problem. There are two reasons for this:
If your solution has different subtours on different days, it may not recognise them as subtours.
For example, consider a single-vehicle problem with two days, where your cities are A, B, C, D, E, F.
Suppose that the day 1 solution selects AB, BC, CD, DE, EF, FA, and the day 2 solution selects AB, BC, CA, DE, EF, FD. Day 1 has no subtour, but day 2 has two length-3 subtours, so this should not be a legal solution.
However, your implementation won't identify this. No matter which node you select as the starting point for NEWSUB, the day-1 routes connect it to all other nodes, so you end up with card(NEWSUB) = card(P). It doesn't notice that Day 2 has a subtour so it will accept this solution.
I'm not sure whether your problem allows for multiple vehicles to visit the same node on the same day. If it does, then you're going to run into the same sort of problem there, where a subtour for vehicle 1 isn't identified because vehicle 2 links that subtour to the rest of P.
Some of this could be fixed by doing subtour checking separately for each day and for each vehicle. But for the problem as you've described it, there's another issue...
Once the program has identified a closed route (i.e. a set of nodes that are all linked to one another, and not to any other nodes) then it needs to figure out whether this subtour should be prohibited.
For the basic TSP, this is straightforward. We have one vehicle that needs to visit every node - hence, if the cardinality of the subtour is smaller than the cardinality of all nodes, then we have an illegal subtour. This is handled by if card(NEWSUB) < card(P).
However, you state:
my problem unlike TSP does not have to visit all nodes in every timeperiod
Suppose Vehicle 1 travels A-B-C-A and Vehicle 2 travels D-E-F-D. In this case, these routes will look like illegal subtours because ABC and DEF are each smaller than ABCDEF and there are no routes that link them. If you use if card(NEWSUB) < card(P) as your criterion for a subloop that should be forbidden, you'll end up forcing every vehicle to visit all nodes, which is fine for basic TSP but not what you want here.
This one can be fixed by identifying how many nodes vehicle v visits on day t, and then comparing the length of the subtour to that total: e.g. if there are 10 cities total, vehicle 1 only visits 6 of them on day 1, and a "subtour" for vehicle 1 visits 6 cities, then that's fine, but if it visits 8 and has a subtour that visits 6, that implies it's travelling two disjoint subloops, which is bad.
One trap to watch out for here:
Suppose Day 1 requires vehicle 1 to visit ABCDEF. If we get a "solution" that has vehicle 1 ABCA and DEFD on one day, we might identify ABCA as a subtour that should be prevented.
However, if Day 2 has different requirements, it might be that having vehicle 1 travel ABCA (and no other nodes) is a legitimate solution for day 2. In this case, you don't want to forbid it on day 2 just because it was part of an illegal solution for day 1.
Similarly, you might have a "subroute" that is a legal solution for one vehicle but illegal for another.
To avoid this, you might need to maintain a different list of prohibited subroutes for each vehicle x day, instead of using one list for all. Unfortunately this is going to make your implementation a bit more complex.
I'm trying to write a constraint but I get this error :
Main.mod:25: operand preceding < has invalid type
Context: ...es : sum { ta in Task_Names } Time_Source [ ti , ma , ta ] <MathProg model processing error
>Exit code: 1 Time: 0.241
this is the code :
set Tasks dimen 4;
set endTimes := setof{(t,s,e,d) in Tasks}e;
set Task_Names := setof{(t,s,e,d) in Tasks}t;
set Machines dimen 2;
set Machine_Names := setof{(m,a) in Machines}m;
set Task_Machines dimen 3;
param lastTime := max{t in endTimes}(t);
set time_slots := 1..lastTime;
var Start_Time{ti in time_slots,ta in Task_Names},binary;
var Time_Source{ti in time_slots,ma in Machine_Names,ta in Task_Names},integer;
s.t. c1{ti in time_slots,ma in Machine_Names,ta in Task_Names,(ta,ma,co) in Task_Machines}:Time_Source[ti,ma,ta] = co*Start_Time[ti,ta];
s.t. c2{ti in time_slots,(t,s,e,d) in Tasks : ti>=s and ti<=e and ti<=e-d}:Start_Time[ti,t]=1;
s.t. c3{ti in time_slots,(t,s,e,d) in Tasks : ti<s or ti>e or ti>e-d}:Start_Time[ti,t]=0;
s.t. c4{ti in time_slots,ma in Machine_Names,(ma,num) in Machines: sum{ta in Task_Names} Time_Source[ti,ma,ta]<num};
solve;
display:Start_Time;
display:Time_Source;
display:Machine_Names;
display:Task_Machines;
data;
set Tasks :=
(T1,3,20,1)
(T2,3,30,5)
(T3,12,40,10);
set Machines :=
(M1,2)
(M2,3)
(M3,2)
(M4,1);
set Task_Machines :=
(T1,M1,1)
(T1,M2,1)
(T2,M1,2)
(T3,M4,5);
end;
the error is for the "C4" constraint. My question is in "c4" how can I use a variable in its conditions?
how can I solve this error?
It looks to me as if there are only two minor typos in c4 (position of }, using <= instead of <):
s.t. c4{ ti in time_slots,
ma in Machine_Names,
(ma,num) in Machines}:
sum{ta in Task_Names} Time_Source[ti,ma,ta] <= num;
I copied the whole fixed example into a in-browser MathProg editor (with included GLPK.js solver).
What happens for a global variable when running in the parallel mode?
I have a global variable, "to_be_optimized_parameterIndexSet", which is a vector of indexes that should be optimized using gamultiobj and I have set its value only in the main script(nowhere else).
My code works properly in serial mode but when I switch to parallel mode (using "matlabpool open" and setting proper values for 'gaoptimset' ) the mentioned global variable becomes empty (=[]) in the fitness function and causes this error:
??? Error using ==> parallel_function at 598
Error in ==> PF_gaMultiFitness at 15 [THIS LINE: constants(to_be_optimized_parameterIndexSet) = individual;]
In an assignment A(I) = B, the number of elements in B and
I must be the same.
Error in ==> fcnvectorizer at 17
parfor (i = 1:popSize)
Error in ==> gamultiobjMakeState at 52
Score =
fcnvectorizer(state.Population(initScoreProvided+1:end,:),FitnessFcn,numObj,options.SerialUserFcn);
Error in ==> gamultiobjsolve at 11
state = gamultiobjMakeState(GenomeLength,FitnessFcn,output.problemtype,options);
E rror in ==> gamultiobj at 238
[x,fval,exitFlag,output,population,scores] = gamultiobjsolve(FitnessFcn,nvars, ...
Error in ==> PF_GA_mainScript at 136
[x, fval, exitflag, output] = gamultiobj(#(individual)PF_gaMultiFitness(individual, initialConstants), ...
Caused by:
Failure in user-supplied fitness function evaluation. GA cannot continue.
I have checked all the code to make sure I've not changed this global variable everywhere else.
I have a quad-core processor.
Where is the bug? any suggestion?
EDIT 1: The MATLAB code in the main script:
clc
clear
close all
format short g
global simulation_duration % PF_gaMultiFitness will use this variable
global to_be_optimized_parameterIndexSet % PF_gaMultiFitness will use this variable
global IC stimulusMoment % PF_gaMultiFitness will use these variables
[initialConstants IC] = oldCICR_Constants; %initialize state
to_be_optimized_parameterIndexSet = [21 22 23 24 25 26 27 28 17 20];
LB = [ 0.97667 0.38185 0.63529 0.046564 0.23207 0.87484 0.46014 0.0030636 0.46494 0.82407 ];
UB = [1.8486 0.68292 0.87129 0.87814 0.66982 1.3819 0.64562 0.15456 1.3717 1.8168];
PopulationSize = input('Population size? ') ;
GaTimeLimit = input('GA time limit? (second) ');
matlabpool open
nGenerations = inf;
options = gaoptimset('PopulationSize', PopulationSize, 'TimeLimit',GaTimeLimit, 'Generations', nGenerations, ...
'Vectorized','off', 'UseParallel','always');
[x, fval, exitflag, output] = gamultiobj(#(individual)PF_gaMultiFitness(individual, initialConstants), ...
length(to_be_optimized_parameterIndexSet),[],[],[],[],LB,UB,options);
matlabpool close
some other piece of code to show the results...
The MATLAB code of the fitness function, "PF_gaMultiFitness":
function objectives =PF_gaMultiFitness(individual, constants)
global simulation_duration IC stimulusMoment to_be_optimized_parameterIndexSet
%THIS FUNCTION RETURNS MULTI OBJECTIVES AND PUTS EACH OBJECTIVE IN A COLUMN
constants(to_be_optimized_parameterIndexSet) = individual;
[smcState , ~, Time]= oldCICR_CompCore(constants, IC, simulation_duration,2);
targetValue = 1; % [uM]desired [Ca]i peak concentration
afterStimulus = smcState(Time>stimulusMoment,14); % values of [Ca]i after stimulus
peak_Ca_value = max(afterStimulus); % smcState(:,14) is [Ca]i
if peak_Ca_value < 0.8 * targetValue
objectives(1,1) = inf;
else
objectives(1, 1) = abs(peak_Ca_value - targetValue);
end
pkIDX = peakFinder(afterStimulus);
nPeaks = sum(pkIDX);
if nPeaks > 1
peakIndexes = find(pkIDX);
period = Time(peakIndexes(2)) - Time(peakIndexes(1));
objectives(1,2) = 1e5* 1/period;
elseif nPeaks == 1 && peak_Ca_value > 0.8 * targetValue
objectives(1,2) = 0;
else
objectives(1,2) = inf;
end
end
Global variables do not get passed from the MATLAB client to the workers executing the body of the PARFOR loop. The only data that does get sent into the loop body are variables that occur in the text of the program. This blog entry might help.
it really depends on the type of variable you're putting in. i need to see more of your code to point out the flaw, but in general it is good practice to avoid assuming complicated variables will be passed to each worker. In other words anything more then a primitive may need to be reinitialized inside a parallel routine or may need have specific function calls (like using feval for function handles).
My advice: RTM
Given this algorithm, I would like to know if there exists an iterative version. Also, I want to know if the iterative version can be faster.
This some kind of pseudo-python...
the algorithm returns a reference to root of the tree
make_tree(array a)
if len(a) == 0
return None;
node = pick a random point from the array
calculate distances of the point against the others
calculate median of such distances
node.left = make_tree(subset of the array, such that the distance of points is lower to the median of distances)
node.right = make_tree(subset, such the distance is greater or equal to the median)
return node
A recursive function with only one recursive call can usually be turned into a tail-recursive function without too much effort, and then it's trivial to convert it into an iterative function. The canonical example here is factorial:
# naïve recursion
def fac(n):
if n <= 1:
return 1
else:
return n * fac(n - 1)
# tail-recursive with accumulator
def fac(n):
def fac_helper(m, k):
if m <= 1:
return k
else:
return fac_helper(m - 1, m * k)
return fac_helper(n, 1)
# iterative with accumulator
def fac(n):
k = 1
while n > 1:
n, k = n - 1, n * k
return k
However, your case here involves two recursive calls, and unless you significantly rework your algorithm, you need to keep a stack. Managing your own stack may be a little faster than using Python's function call stack, but the added speed and depth will probably not be worth the complexity. The canonical example here would be the Fibonacci sequence:
# naïve recursion
def fib(n):
if n <= 1:
return 1
else:
return fib(n - 1) + fib(n - 2)
# tail-recursive with accumulator and stack
def fib(n):
def fib_helper(m, k, stack):
if m <= 1:
if stack:
m = stack.pop()
return fib_helper(m, k + 1, stack)
else:
return k + 1
else:
stack.append(m - 2)
return fib_helper(m - 1, k, stack)
return fib_helper(n, 0, [])
# iterative with accumulator and stack
def fib(n):
k, stack = 0, []
while 1:
if n <= 1:
k = k + 1
if stack:
n = stack.pop()
else:
break
else:
stack.append(n - 2)
n = n - 1
return k
Now, your case is a lot tougher than this: a simple accumulator will have difficulties expressing a partly-built tree with a pointer to where a subtree needs to be generated. You'll want a zipper -- not easy to implement in a not-really-functional language like Python.
Making an iterative version is simply a matter of using your own stack instead of the normal language call stack. I doubt the iterative version would be faster, as the normal call stack is optimized for this purpose.
The data you're getting is random so the tree can be an arbitrary binary tree. For this case, you can use a threaded binary tree, which can be traversed and built w/o recursion and no stack. The nodes have a flag that indicate if the link is a link to another node or how to get to the "next node".
From http://en.wikipedia.org/wiki/Threaded_binary_tree
Depending on how you define "iterative", there is another solution not mentioned by the previous answers. If "iterative" just means "not subject to a stack overflow exception" (but "allowed to use 'let rec'"), then in a language that supports tail calls, you can write a version using continuations (rather than an "explicit stack"). The F# code below illustrates this. It is similar to your original problem, in that it builds a BST out of an array. If the array is shuffled randomly, the tree is relatively balanced and the recursive version does not create too deep a stack. But turn off shuffling, and the tree gets unbalanced, and the recursive version stack-overflows whereas the iterative-with-continuations version continues along happily.
#light
open System
let printResults = false
let MAX = 20000
let shuffleIt = true
// handy helper function
let rng = new Random(0)
let shuffle (arr : array<'a>) = // '
let n = arr.Length
for x in 1..n do
let i = n-x
let j = rng.Next(i+1)
let tmp = arr.[i]
arr.[i] <- arr.[j]
arr.[j] <- tmp
// Same random array
let sampleArray = Array.init MAX (fun x -> x)
if shuffleIt then
shuffle sampleArray
if printResults then
printfn "Sample array is %A" sampleArray
// Tree type
type Tree =
| Node of int * Tree * Tree
| Leaf
// MakeTree1 is recursive
let rec MakeTree1 (arr : array<int>) lo hi = // [lo,hi)
if lo = hi then
Leaf
else
let pivot = arr.[lo]
// partition
let mutable storeIndex = lo + 1
for i in lo + 1 .. hi - 1 do
if arr.[i] < pivot then
let tmp = arr.[i]
arr.[i] <- arr.[storeIndex]
arr.[storeIndex] <- tmp
storeIndex <- storeIndex + 1
Node(pivot, MakeTree1 arr (lo+1) storeIndex, MakeTree1 arr storeIndex hi)
// MakeTree2 has all tail calls (uses continuations rather than a stack, see
// http://lorgonblog.spaces.live.com/blog/cns!701679AD17B6D310!171.entry
// for more explanation)
let MakeTree2 (arr : array<int>) lo hi = // [lo,hi)
let rec MakeTree2Helper (arr : array<int>) lo hi k =
if lo = hi then
k Leaf
else
let pivot = arr.[lo]
// partition
let storeIndex = ref(lo + 1)
for i in lo + 1 .. hi - 1 do
if arr.[i] < pivot then
let tmp = arr.[i]
arr.[i] <- arr.[!storeIndex]
arr.[!storeIndex] <- tmp
storeIndex := !storeIndex + 1
MakeTree2Helper arr (lo+1) !storeIndex (fun lacc ->
MakeTree2Helper arr !storeIndex hi (fun racc ->
k (Node(pivot,lacc,racc))))
MakeTree2Helper arr lo hi (fun x -> x)
// MakeTree2 never stack overflows
printfn "calling MakeTree2..."
let tree2 = MakeTree2 sampleArray 0 MAX
if printResults then
printfn "MakeTree2 yields"
printfn "%A" tree2
// MakeTree1 might stack overflow
printfn "calling MakeTree1..."
let tree1 = MakeTree1 sampleArray 0 MAX
if printResults then
printfn "MakeTree1 yields"
printfn "%A" tree1
printfn "Trees are equal: %A" (tree1 = tree2)
Yes it is possible to make any recursive algorithm iterative. Implicitly, when you create a recursive algorithm each call places the prior call onto the stack. What you want to do is make the implicit call stack into an explicit one. The iterative version won't necessarily be faster, but you won't have to worry about a stack overflow. (do I get a badge for using the name of the site in my answer?
While it is true in the general sense that directly converting a recursive algorithm into an iterative one will require an explicit stack, there is a specific sub-set of algorithms which render directly in iterative form (without the need for a stack). These renderings may not have the same performance guarantees (iterating over a functional list vs recursive deconstruction), but they do often exist.
Here is stack based iterative solution (Java):
public static Tree builtBSTFromSortedArray(int[] inputArray){
Stack toBeDone=new Stack("sub trees to be created under these nodes");
//initialize start and end
int start=0;
int end=inputArray.length-1;
//keep memoy of the position (in the array) of the previously created node
int previous_end=end;
int previous_start=start;
//Create the result tree
Node root=new Node(inputArray[(start+end)/2]);
Tree result=new Tree(root);
while(root!=null){
System.out.println("Current root="+root.data);
//calculate last middle (last node position using the last start and last end)
int last_mid=(previous_start+previous_end)/2;
//*********** add left node to the previously created node ***********
//calculate new start and new end positions
//end is the previous index position minus 1
end=last_mid-1;
//start will not change for left nodes generation
start=previous_start;
//check if the index exists in the array and add the left node
if (end>=start){
root.left=new Node(inputArray[((start+end)/2)]);
System.out.println("\tCurrent root.left="+root.left.data);
}
else
root.left=null;
//save previous_end value (to be used in right node creation)
int previous_end_bck=previous_end;
//update previous end
previous_end=end;
//*********** add right node to the previously created node ***********
//get the initial value (inside the current iteration) of previous end
end=previous_end_bck;
//start is the previous index position plus one
start=last_mid+1;
//check if the index exists in the array and add the right node
if (start<=end){
root.right=new Node(inputArray[((start+end)/2)]);
System.out.println("\tCurrent root.right="+root.right.data);
//save the created node and its index position (start & end) in the array to toBeDone stack
toBeDone.push(root.right);
toBeDone.push(new Node(start));
toBeDone.push(new Node(end));
}
//*********** update the value of root ***********
if (root.left!=null){
root=root.left;
}
else{
if (toBeDone.top!=null) previous_end=toBeDone.pop().data;
if (toBeDone.top!=null) previous_start=toBeDone.pop().data;
root=toBeDone.pop();
}
}
return result;
}