Are there algorithms that are only feasible in a recursive manner? If yes, how can one identify these algorithms?
Consider the following as a case study.
In order to approximate a exponential, one can use.
ef ≈ 1 + f(1 + f/2(1 + f/3(1 + f/4)))
Here is a function that computes this approximation at a level of n steps using recursion.
exp_approximation = function(f, n, i = 2)
{
if (n == 1)
{
return( 1 + f/i )
} else
{
1 + f/i * exp_approximation(f,n=n-1,i=i+1)
}
}
Is it possible to write the same algorithm using iteration instead of recursion?
I wrote my code in R but I welcome solution in pseudo code other some other commonly used language such as C, C++, Java, Python, ...
Yes, technically it's possible. See, for example, this question for a generic conversion method. That said, many algorithms are much more intuitive in the recursive form (and might be more efficient in functional languages but not in imperative ones). That's especially true for math-related problems involving recurrences, where recursion is in fact a representation of a recurrence relation.
Related
I read recently in Steven Skiena "Algorithms design manual book" that the harmonic Sum is of O(lg n). I understand how this is possible mathematically, but when i saw the recursive solution to it, which is the following:
public static double harmonic(int n) {
if(n == 1) {
return 1.0;
} else {
return (1.0 / n) + harmonic(n - 1);
}
}
I can not see how this is O(lg n), I see it as O(n). is this code above really lg(n) if so how ? if not can someone refer me to an example where this is the case ?
I believe the confusion here is that big-O notation can be used not only for running times, but for the growth of functions in general.
As kaya3 comments, the return value of the function above is O(lg n), which means it grows asymptotically as fast as lg n as you increase n. This is quite likely what the book is claiming. It is a mathematical property of the harmonic series.
The running time of the function, however, takes O(n) time. This is what you are claiming, and it is perfectly true (assuming constant-time arithmetics).
If you are still confused about how big-O notation can be used for purposes other than the running time of algorithms, I recommend you check its definition.
I have no idea if this is even remotely possible (I looked up "computing algebra" etc with discouraging results). How can one compute Algebra and find Derivatives with Unity?
For example, simplifying the distance formula with one variable (x unkown, some function f(x) known):
d = sqrt( (int-x)^2 + (int-f(x))^2 );
and then finding the derivative of this simplified expression?:
d=>d'
Thank you for your time and any light you can shed on this question. And once again, I have no idea if algebraic operations are even commonplace among most programs, let alone Unity-script specifically.
I have also noticed a few systems claiming algebra manipulation (e.g. http://coffeequate.readthedocs.org/en/latest/), but even if this is so how would one go about applying these systems to unity?
If you are writing in C#, you can pull off derivatives with delegates and the definition of a derivative, like this:
delegate double MathFunc(double d);
MathFunc derive(MathFunc f, float h) {
return (x) => (f(x+h) - f(x)) / h;
}
where f in the function you are taking the derivative of, and h determines how accurate your derivative is.
I've had this question on my mind for a really long time but I can't figure out the answer. The question is, if does every recursive function have an iterative function that does the same?
For example,
factorial(n) {
if (n==1) { return 1 }
else { return factorial(n-1) }
}
This can be easily rewritten iteratively:
factorial(n) {
result = 1;
for (i=1; i<=n; i++) {
result *= i
}
return result
}
But there are many other, more complicated recursive functions, so I don't know the answer in general. This might also be a theoretical computer science question.
Yes, a recursive function can always be written as an iteration, from a theoretical point of view - this has been discussed before. Quoting from the linked post:
Because you can build a Turing complete language using strictly iterative structures and a Turning complete language using only recursive structures, then the two are therefore equivalent.
Explaining a bit: we know that any computable problem can be solved by a Turing machine. And it's possible to construct a programming language A without recursion, that is equivalent to a Turing machine. Similarly, it's possible to build a programming language B without iteration, equal in computational power to a Turing machine.
Therefore, if both A and B are Turing-complete we can conclude that for any iterative program there must exist an equivalent recursive program, and vice versa. This is a theoretical result, in the sense that it doesn't give you any hints on how to derive one recursive program from an arbitrary iterative program, or vice versa.
Without going to theory, it is easy to convince oneself that any recursive function can have an iterative equivalent by observing that processors (such as Pentium) just run iteratively.
I'm trying to implement the Softmax regression algorithm to solve the K-classifier problem after watching Professor Andrew Ng's lectures on GLM. I thought I understood everything he was saying until it finally came to writing the code to implement the cost function for Softmax regression, which is as follows:
The problem I am having is trying to figure out a way to vectorize this. Again I thought I understood how to go about vectorizing equations like this since I was able to do it for linear and logistic regression, but after looking at that formula I am stuck.
While I would love to figure out a vectorized solution for this (I realize there is a similar question posted already: Vectorized Implementation of Softmax Regression), what I am more interested in is whether any of you can tell me a way (your way) to methodically convert equations like this into vectorized forms. For example, for those of you who are experts or seasoned veterans in ML, when you read of new algorithms in the literature for the first time, and see them written in similar notation to the equation above, how do you go about converting them to vectorized forms?
I realize I might be coming off as being like the student who is asking Mozart, "How do you play the piano so well?" But my question is simply motivated from a desire to become better at this material, and assuming that not everyone was born knowing how to vectorize equations, and so someone out there must have devised their own system, and if so, please share! Many thanks in advance!
Cheers
This one looks pretty hard to vectorize since you are doing exponentials inside of your summations. I assume you are raising e to arbitrary powers. What you can vectorize is the second term of the expression \sum \sum theta ^2 just make sure to use .* operator in matlab enter link description here to computer \theta ^2
Same goes for the inner terms of the ratio of the that goes into the logarithm. \theta ' x^(i) is vectorizable expression.
You might also benefit from a memoization or dynamic programming technique and try to reuse the results of computations of e^\theta' x^(i).
Generally in my experience the way to vectorize is first to get non-vectorized implementation working. Then try to vectorize the most obvious parts of your computation. At every step tweak your function very little and always check if you get the same result as non-vectorized computation. Also, having multiple test cases is very helpful.
The help files that come with Octave have this entry:
19.1 Basic Vectorization
To a very good first approximation, the goal in vectorization is to
write code that avoids loops and uses whole-array operations. As a
trivial example, consider
for i = 1:n
for j = 1:m
c(i,j) = a(i,j) + b(i,j);
endfor
endfor
compared to the much simpler
c = a + b;
This isn't merely easier to write; it is also internally much easier to
optimize. Octave delegates this operation to an underlying
implementation which, among other optimizations, may use special vector
hardware instructions or could conceivably even perform the additions in
parallel. In general, if the code is vectorized, the underlying
implementation has more freedom about the assumptions it can make in
order to achieve faster execution.
This is especially important for loops with "cheap" bodies. Often it
suffices to vectorize just the innermost loop to get acceptable
performance. A general rule of thumb is that the "order" of the
vectorized body should be greater or equal to the "order" of the
enclosing loop.
As a less trivial example, instead of
for i = 1:n-1
a(i) = b(i+1) - b(i);
endfor
write
a = b(2:n) - b(1:n-1);
This shows an important general concept about using arrays for
indexing instead of looping over an index variable.  Index Expressions.
Also use boolean indexing generously. If a condition
needs to be tested, this condition can also be written as a boolean
index. For instance, instead of
for i = 1:n
if (a(i) > 5)
a(i) -= 20
endif
endfor
write
a(a>5) -= 20;
which exploits the fact that 'a > 5' produces a boolean index.
Use elementwise vector operators whenever possible to avoid looping
(operators like '.*' and '.^').  Arithmetic Ops. For simple
inline functions, the 'vectorize' function can do this automatically.
-- Built-in Function: vectorize (FUN)
Create a vectorized version of the inline function FUN by replacing
all occurrences of '', '/', etc., with '.', './', etc.
This may be useful, for example, when using inline functions with
numerical integration or optimization where a vector-valued
function is expected.
fcn = vectorize (inline ("x^2 - 1"))
=> fcn = f(x) = x.^2 - 1
quadv (fcn, 0, 3)
=> 6
See also:  inline,  formula,
 argnames.
Also exploit broadcasting in these elementwise operators both to
avoid looping and unnecessary intermediate memory allocations.
 Broadcasting.
Use built-in and library functions if possible. Built-in and
compiled functions are very fast. Even with an m-file library function,
chances are good that it is already optimized, or will be optimized more
in a future release.
For instance, even better than
a = b(2:n) - b(1:n-1);
is
a = diff (b);
Most Octave functions are written with vector and array arguments in
mind. If you find yourself writing a loop with a very simple operation,
chances are that such a function already exists. The following
functions occur frequently in vectorized code:
Index manipulation
* find
* sub2ind
* ind2sub
* sort
* unique
* lookup
* ifelse / merge
Repetition
* repmat
* repelems
Vectorized arithmetic
* sum
* prod
* cumsum
* cumprod
* sumsq
* diff
* dot
* cummax
* cummin
Shape of higher dimensional arrays
* reshape
* resize
* permute
* squeeze
* deal
Also look at these pages from a Stanford ML wiki for some more guidance with examples.
http://ufldl.stanford.edu/wiki/index.php/Vectorization
http://ufldl.stanford.edu/wiki/index.php/Logistic_Regression_Vectorization_Example
http://ufldl.stanford.edu/wiki/index.php/Neural_Network_Vectorization
I'm interested in building a derivative calculator. I've racked my brains over solving the problem, but I haven't found a right solution at all. May you have a hint how to start? Thanks
I'm sorry! I clearly want to make symbolic differentiation.
Let's say you have the function f(x) = x^3 + 2x^2 + x
I want to display the derivative, in this case f'(x) = 3x^2 + 4x + 1
I'd like to implement it in objective-c for the iPhone.
I assume that you're trying to find the exact derivative of a function. (Symbolic differentiation)
You need to parse the mathematical expression and store the individual operations in the function in a tree structure.
For example, x + sin²(x) would be stored as a + operation, applied to the expression x and a ^ (exponentiation) operation of sin(x) and 2.
You can then recursively differentiate the tree by applying the rules of differentiation to each node. For example, a + node would become the u' + v', and a * node would become uv' + vu'.
you need to remember your calculus. basically you need two things: table of derivatives of basic functions and rules of how to derivate compound expressions (like d(f + g)/dx = df/dx + dg/dx). Then take expressions parser and recursively go other the tree. (http://www.sosmath.com/tables/derivative/derivative.html)
Parse your string into an S-expression (even though this is usually taken in Lisp context, you can do an equivalent thing in pretty much any language), easiest with lex/yacc or equivalent, then write a recursive "derive" function. In OCaml-ish dialect, something like this:
let rec derive var = function
| Const(_) -> Const(0)
| Var(x) -> if x = var then Const(1) else Deriv(Var(x), Var(var))
| Add(x, y) -> Add(derive var x, derive var y)
| Mul(a, b) -> Add(Mul(a, derive var b), Mul(derive var a, b))
...
(If you don't know OCaml syntax - derive is two-parameter recursive function, with first parameter the variable name, and the second being mathched in successive lines; for example, if this parameter is a structure of form Add(x, y), return the structure Add built from two fields, with values of derived x and derived y; and similarly for other cases of what derive might receive as a parameter; _ in the first pattern means "match anything")
After this you might have some clean-up function to tidy up the resultant expression (reducing fractions etc.) but this gets complicated, and is not necessary for derivation itself (i.e. what you get without it is still a correct answer).
When your transformation of the s-exp is done, reconvert the resultant s-exp into string form, again with a recursive function
SLaks already described the procedure for symbolic differentiation. I'd just like to add a few things:
Symbolic math is mostly parsing and tree transformations. ANTLR is a great tool for both. I'd suggest starting with this great book Language implementation patterns
There are open-source programs that do what you want (e.g. Maxima). Dissecting such a program might be interesting, too (but it's probably easier to understand what's going on if you tried to write it yourself, first)
Probably, you also want some kind of simplification for the output. For example, just applying the basic derivative rules to the expression 2 * x would yield 2 + 0*x. This can also be done by tree processing (e.g. by transforming 0 * [...] to 0 and [...] + 0 to [...] and so on)
For what kinds of operations are you wanting to compute a derivative? If you allow trigonometric functions like sine, cosine and tangent, these are probably best stored in a table while others like polynomials may be much easier to do. Are you allowing for functions to have multiple inputs,e.g. f(x,y) rather than just f(x)?
Polynomials in a single variable would be my suggestion and then consider adding in trigonometric, logarithmic, exponential and other advanced functions to compute derivatives which may be harder to do.
Symbolic differentiation over common functions (+, -, *, /, ^, sin, cos, etc.) ignoring regions where the function or its derivative is undefined is easy. What's difficult, perhaps counterintuitively, is simplifying the result afterward.
To do the differentiation, store the operations in a tree (or even just in Polish notation) and make a table of the derivative of each of the elementary operations. Then repeatedly apply the chain rule and the elementary derivatives, together with setting the derivative of a constant to 0. This is fast and easy to implement.