I'm a bit confused as to whether the Sieve of Eratosthenes (implemented with an array for all the numbers and a loop marking the composite numbers) is an example of Dynamic Programming? A couple of friends were telling me the way it's implemented is an example of Bottom Up DP, but I'm having trouble seeing it. Exactly what are the subproblems and how would you implement SoE with Top-Down / Recursion? Thanks guys.
Sure, we could think of the Sieve of Eratosthenes as an example of dynamic programming. The subproblems would be all the composite numbers. Skipping over marked numbers is a perfect demonstration of the subproblems overlapping since if they did not overlap we wouldn't be skipping over them :)
One way we could formulate the sieve recursively could be: let f(N) represent the Nth prime and its associated sieve state. Then:
f(1) = (2, [4,6...])
f(N) = (q, join( Sieve, [q+q,q+q+q...]))
'''a pair, of the next number q above p
_not_ in Sieve, and the Sieve with
all the multiples of this number q
added into it (we'll place an upper
bound on this process, practically)'''
where (p, Sieve) = f(N - 1)
q = next_not_in(p, Sieve)
Let's test:
f(3) =
call f(2) =
call f(1) =
<-- return (2, [4,6...])
<-- return (3, [4,6,8,9...])
<-- return (5, [4,6,8,9,10...])
Related
The task is to find the Power Set of an array which is to find set of all subsets of an array for example for [1,2] it will be [[],[1],[2],[1,2]]. For an n length array, the power set is 2^n long. It is evident when I try to do it using recursion and 2 calls like below
class Solution:
def helperSubsets(self,output_li, nums, klist):
if len(nums)==0:
output_li.append(klist)
return
self.helperSubsets(output_li,nums[1:],klist+[nums[0]])
self.helperSubsets(output_li,nums[1:],klist)
def subsets(self, nums: List[int]) -> List[List[int]]:
output_li = []
self.helperSubsets(output_li,nums,[])
return output_li
Recurrence Relation is : T(n) = 2T(n-1) + O(1)
But when trying to do it using for loop in recursive call, how do I prove or find the time complexity of the recursion, what would be the Recurrence relation also if you know. I am not able to figure it out. Reference of doing it with for loop and recursion:
class Solution:
def helperSubsets(self,output_li, nums, klist):
output_li.append(klist)
for i in range(len(nums)):
self.helperSubsets(output_li,nums[i+1:],klist+[nums[i]])
def subsets(self, nums: List[int]) -> List[List[int]]:
output_li = []
self.helperSubsets(output_li,nums,[])
return output_li
In the first example, the element is either being picked or not, so the recursive calls are being divided twice at each element so complexity is 2^n. But for the second code which is doing the same thing, how do I find the complexity? (apart from knowing it's doing the same thing as the first so it'll be similar). Can you also give me the recurrence relation for it if you know?
Fibonacci series is 0 1 1 2 3 5 8... and so on. It can be obtained using swapping elements and displaying them whereas we can obtain it using array. I was asked to find it using recursion in interview and main logic for it,
int fib(int n){
if(n<1)
return 1;
else
return fib(n-1)+fib(n-2);}
It generate problem for stack for big number because we are increasing complexity here. So what is optimum way here?
Ironically, the method used above i.e. binary recursion computes the Fibonacci number by making two recursive calls in each non-base case. Unfortunately, such a direct implementation of the Fibonacci formula number in this way requires an exponential number of calls to the method.
We are tempted to use the bad recursive formulation because of the way the nth Fibonacci number, F(n), depends on the two previous values, F(n-2) and F(n-1). But notice that after computing F(n-2), the call to compute F(n-1) requires its own recursive call to compute F(n-2), as it does not have the knowledge of value of F(n-2) that was computed at the earlier level of recursion. That is duplicate work. Worse yet, both of these calls will need to (re)compute the value of F(n-3), as will the computation of F(n-1). This snowballing effect is what leads to the exponential running time of fib().
We can compute F(n) much more efficiently using linear recursion in which each invocation makes only one recursive call. To do so, we need to redefine the expectations of the method. Rather than having the method that returns a single value, which is the nth Fibonacci number, we define a recursive method that returns an array with two consecutive Fibonacci numbers {F(n), F(n-1)} using the convention F(-1)=0. Although it seems to be a greater burden to report two consecutive Fibonacci numbers instead of one, passing this extra information from one level of the recursion to the next makes it much easier to continue the process. (It allows us to avoid having to recompute the second value that was already known within the recursion.)
An implementation based on this strategy is clearly shown here.
Memoization. Create logic that calculates each fib numb only once.
static BigInteger[] fibNumbs = new BigInteger[10000];
public static void main(String[] args) {
fibNumbs[1] = BigInteger.ONE;
fibNumbs[2] = BigInteger.ONE;
System.out.println(fibOf(10000));
}
public static BigInteger fibOf(int n) {
if (n <= 1) {
return BigInteger.ONE;
}
if (fibNumbs[n - 1]==null) {
fibNumbs[n - 1] = fibOf(n - 1);
}
if (fibNumbs[n - 2]==null) {
fibNumbs[n - 2] = fibOf(n - 2);
}
return fibNumbs[n - 1].add(fibNumbs[n - 2]);
}
If I told you two consecutive fibonacci numbers, eg. a=3 and b=5, could you guess the next? It's the two summed so its 8. Now with a=5 and the newly computed number b=8 you can calculate the next? You start the iteration with the two first 0, 1 and the index of the number you'd like for each iteration you count down and when you hit zero a is your answer. This is a O(n) algorithm.
After a long search on google I couldn't find a clear answer of this:
In Prolog doing recursion by itself its easy. My main problem is understanding where to place accumulators and counters. Here is an example:
nXlist(N,X,[X|T]):-
N \=0,
N1 is N-1,
nXList(N1,X,T).
nXList(0,_,[]).
media([X|L], N, Soma):-
media(L, N1, Soma1),
N is N1 + 1,
Soma is Soma1 + X.
media([], 0, 0).
On the first example i used a counter BEFORE the recursion but in the second example I use it AFTER. The reason I have done that is the called try and see cause i really can't understand why sometimes is before and sometimes is after...
Maybe the central point of your question is in the preamble:
In Prolog doing recursion by itself its easy
It's not easy, it's mandatory. We don't have loops, because we don't have a way to control them. Variables are assign once.
So, I think the practical answer is rather simple: if the 'predicate' (like is/2) needs a variable value, you ground the variable before calling it.
To me, it helps to consider a Prolog program (a set of clauses) as grammar productions, and clause arguments as attributes, either inherited (values computed before the 'instruction pointer') or synthesized (values computed 'here', to be returned).
update: Most importantly, if the recursive call is not last, the predicate is not tail recursive. So, having anything after the recursive call should be avoided if possible. Notice that both definitions in the answer by user false are tail recursive, and that's precisely due to the fact that the arithmetic conditions there are placed before the recursive call, in both of them. That's so basic, that we have to make an effort to notice it explicitly.
Sometimes we count down, sometimes we count up. I discuss this in another answer at length. It talks of accumulators, befores and afters. :)
There's also this thing called "associativity" of an operation (say, +), where
a+(b+(c+....)) == (a+b)+(c+...)
that lets us regroup and (partially) calculate sooner rather than later. As soon as possible, but not sooner.
Short answer: you can place such arithmetical relations both before and thereafter. At least, if you are using constraints in place of (is)/2. The only difference may be in termination and errors.
So let's see how your predicates can be defined with constraints:
:- use_module(library(clpfd)).
nXList(0,_,[]).
nXList(N,X,[X|T]):-
N #> 0,
N1 #= N-1,
nXList(N1,X,T).
media([], 0, 0).
media([X|L], N, Soma):-
N #> 0,
N #= N1 + 1,
Soma #= Soma1 + X,
media(L, N1, Soma1).
You can now use these definitions in a much more general way, say:
?- nXList(3, X, T).
T = [X, X, X]
; false.
?- media(Xs, 3, S).
Xs = [_A, _B, _C], _D+_A#=S, _C+_B#=_D
; false.
... nXList/3 can be more compactly expressed by:
..., length(T, N), maplist(=(X), T), ...
I'm having difficulty determining the big O of simple recursive methods. I can't wrap my head around what happens when a method is called multiple times. I would be more specific about my areas of confusion, but at the moment I'm trying to answer some hw questions, and in lieu of not wanting to cheat, I ask that anyone responding to this post come up with a simple recursive method and provide a simple explanation of the big O of said method. (Preferably in Java... a language I'm learning.)
Thank you.
You can define the order recursively as well. For instance, let's say you have a function f. To calculate f(n) takes k steps. Now you want to calculate f(n+1). Lets say f(n+1) calls f(n) once, then f(n+1) takes k + some constant steps. Each invocation will take some constant steps extra, so this method is O(n).
Now look at another example. Lets say you implement fibonacci naively by adding the two previous results:
fib(n) = { return fib(n-1) + fib(n-2) }
Now lets say you can calculate fib(n-2) and fib(n-1) both in about k steps. To calculate fib(n) you need k+k = 2*k steps. Now lets say you want to calculate fib(n+1). So you need twice as much steps as for fib(n-1). So this seems to be O(2^N)
Admittedly, this is not very formal, but hopefully this way you can get a bit of a feel.
You might want to refer to the master theorem for finding the big O of recursive methods. Here is the wikipedia article: http://en.wikipedia.org/wiki/Master_theorem
You want to think of a recursive problem like a tree. Then, consider each level of the tree and the amount of work required. Problems will generally fall into 3 categories, root heavy (first iteration >> rest of tree), balanced (each level has equal amounts of work), leaf heavy (last iteration >> rest of tree).
Taking merge sort as an example:
define mergeSort(list toSort):
if(length of toSort <= 1):
return toSort
list left = toSort from [0, length of toSort/2)
list right = toSort from [length of toSort/2, length of toSort)
merge(mergeSort(left), mergeSort(right))
You can see that each call of mergeSort in turn calls 2 more mergeSorts of 1/2 the original length. We know that the merge procedure will take time proportional to the number of values being merged.
The recurrence relationship is then T(n) = 2*T(n/2)+O(n). The two comes from the 2 calls and the n/2 is from each call having only half the number of elements. However, at each level there are the same number of elements n which need to be merged, so the constant work at each level is O(n).
We know the work is evenly distributed (O(n) each depth) and the tree is log_2(n) deep, so the big O of the recursive function is O(n*log(n)).
Is it impossible to know if two functions are equivalent? For example, a compiler writer wants to determine if two functions that the developer has written perform the same operation, what methods can he use to figure that one out? Or can what can we do to find out that two TMs are identical? Is there a way to normalize the machines?
Edit: If the general case is undecidable, how much information do you need to have before you can correctly say that two functions are equivalent?
Given an arbitrary function, f, we define a function f' which returns 1 on input n if f halts on input n. Now, for some number x we define a function g which, on input n, returns 1 if n = x, and otherwise calls f'(n).
If functional equivalence were decidable, then deciding whether g is identical to f' decides whether f halts on input x. That would solve the Halting problem. Related to this discussion is Rice's theorem.
Conclusion: functional equivalence is undecidable.
There is some discussion going on below about the validity of this proof. So let me elaborate on what the proof does, and give some example code in Python.
The proof creates a function f' which on input n starts to compute f(n). When this computation finishes, f' returns 1. Thus, f'(n) = 1 iff f halts on input n, and f' doesn't halt on n iff f doesn't. Python:
def create_f_prime(f):
def f_prime(n):
f(n)
return 1
return f_prime
Then we create a function g which takes n as input, and compares it to some value x. If n = x, then g(n) = g(x) = 1, else g(n) = f'(n). Python:
def create_g(f_prime, x):
def g(n):
return 1 if n == x else f_prime(n)
return g
Now the trick is, that for all n != x we have that g(n) = f'(n). Furthermore, we know that g(x) = 1. So, if g = f', then f'(x) = 1 and hence f(x) halts. Likewise, if g != f' then necessarily f'(x) != 1, which means that f(x) does not halt. So, deciding whether g = f' is equivalent to deciding whether f halts on input x. Using a slightly different notation for the above two functions, we can summarise all this as follows:
def halts(f, x):
def f_prime(n): f(n); return 1
def g(n): return 1 if n == x else f_prime(n)
return equiv(f_prime, g) # If only equiv would actually exist...
I'll also toss in an illustration of the proof in Haskell (GHC performs some loop detection, and I'm not really sure whether the use of seq is fool proof in this case, but anyway):
-- Tells whether two functions f and g are equivalent.
equiv :: (Integer -> Integer) -> (Integer -> Integer) -> Bool
equiv f g = undefined -- If only this could be implemented :)
-- Tells whether f halts on input x
halts :: (Integer -> Integer) -> Integer -> Bool
halts f x = equiv f' g
where
f' n = f n `seq` 1
g n = if n == x then 1 else f' n
Yes, it is undecidable. This is a form of the halting problem.
Note that I mean that it's undecidable for the general case. Just as you can determine halting for sufficiently simple programs, you can determine equivalency for sufficiently simple functions, and it's not inconceivable that this could be of some use for an application. But you cannot make a general method for determining equivalency of any two possible functions.
The general case is undecidable by Rice's Theorem, as others have already said (Rice's Theorem essentially says that any nontrivial property of a Turing-complete formalism is undecidable).
There are special cases where equivalence is decidable, the best-known example is probably equivalence of finite state automata. If I remember correctly equivalence of pushdown automata is already undecidable by reduction to Post's Correspondence Problem.
To prove that two given functions are equivalent you would require as input a proof of the equivalence in some formalism, which you can then check for correctness. The essential parts of this proof are the loop invariants, as these cannot be derived automatically.
In the general case it's undecidable whether two turing machines have always the same output for the identical input. Since you can't even decide whether a tm will halt on the input, I don't see how it should be possible to decide whether both halt AND output the same result...
It depends on what you mean by "function."
If the functions you are talking about are guaranteed to terminate -- for example, because they are written in a language in which all functions terminate -- and operate over finite domains, it's "easy" (although it might still take a very, very long time): two functions are equivalent if and only if they have the same value at every point in their shared domain.
This is called "extensional" equivalence to distinguish it from syntactic or "intensional" equivalence. Two functions are extensionally equivalent if they are intensionally equivalent, but the converse does not hold.
(All the other people above noting that it is undecidable in the general case are quite correct, of course, this is a fairly uncommon -- and usually uninteresting in practice -- special case.)
Note that the halting problem is decidable for linear bounded automata. Real computers are always bounded, and programs for them will always loop back to a previous configuration after sufficiently many steps. If you are using an unbounded (imaginary) computer to keep track of the configurations, you can detect that looping and take it into account.
You could check in your compiler to see if they are "exactly" identical, sure, but determining if they return identical values would be difficult and time consuming. You would have to basically call that method and perform its routine over an infinite number of possible calls and compare the value with that from the other routine.
Even if you could do the above, you would have to account for what global values change within the function, what objects are destroyed / changed in the function that do not affect the outcome.
You can really only compare the compiled code. So compile the compiled code to refactor?
Imagine the run time on trying to compile the code with "that" compiler. You could spend a LOT of time on here answering questions saying: "busy compiling..." :)
I think if you allow side effects, you can show that the problem can be morphed into the Post correspondence problem so you can't, in general, show if two functions are even capable of having the same side effects.
Is it impossible to know if two functions are equivalent?
No. It is possible to know that two functions are equivalent. If you have f(x), you know f(x) is equivalent to f(x).
If the question is "it is possible to determine if f(x) and g(x) are equivalent with f and g being any function and for all functions g and f", then the answer is no.
However, if the question is "can a compiler determine that if f(x) and g(x) are equivalent that they are equivalent?", then the answer is yes if they are equivalent in both output and side effects and order of side effects. In other words, if one is a transformation of the other that preserves behavior, then a compiler of sufficient complexity should be able to detect it. It also means that the compiler can transform a function f into a more optimal and equivalent function g given a particular definition of equivalent. It gets even more fun if f includes undefined behavior, because then g can also include undefined (but different) behavior!