I'm running some performance sensitive code and looking to improve speed. I am using vnormdiff and findmax a lot and wondered whether these are the most efficient functions around? Any thoughts greatly appreciated.
Whenever you encounter a performance problem, it's good to look at your problem from two angles. First, is my overall algorithm the best it can be? If you're using an O(N^2) algorithm but an O(N) is available, that could make an enormous difference. It sounds like you're examining neighbors, so some of the more refined nearest-neighbor algorithms (which depend on dimensionality) might be of assistance.
Second, no discussion about optimization can really get started without profiling information. There's documentation on Julia's profiler here, and a graphical tool for inspecting it here.
Related
I have a very hard time with math yet I always seem to find myself in ever increasingly difficult math courses. Over the years I have amassed a small 'fortune' of various scripts and programs. They are written in a variety of languages, from C to Java to Python and even a few in Lisp (I like to fool around with different languages). These are great for getting fast answers or checking my homework, but not so great for showing how those answers were gotten. My question is how would I go about adding a 'solver' aspect to these various programs? It doesn't need to be language specific, but just a concept or maybe a specific word or definition would be of immense help. I basically want to do a Wolfram Alpha type program, where you punch in the problem and it shows a step by step solution. I have spent the last couple of days researching this, but I am always redirected towards an app or, frustratingly, to Wolfram Alpha itself. Any suggestions would be much appreciated. Thank you.
What categories of math are you talking about?
There does not seem to be a universal math solving program, what you mentioned sounds to me like the concept of a complicated math system(like MatLab).
And for, I think, a large scale of mathematical problems, the algorithms for computer programs to solve them differ from our ways to solve them.
Rather than a universal algorithm to achieve your goal, I think you can divide your problem into pieces, and achieve them with different sets of programs, there is a thesis Symbolic Integration by Joel Moses, the first chapters give a good view of how to divide a math problem into programmable cases(in the example of solving quadratic equations), and the last hundred pages is a set of algorithms for solving irrational integration, interpreted in LISP.
The question may be stupid but it really confuses me for a long time.
I read a lot of papers in wireless sensor network. Many researchers model their problems into the form of ILP. However, ILP is NP-Complete so it is not efficient for solving a problem.
So why people write their problems into the form of ILP? Do they do that to make their problem clear to see and easy to understand? Or do I make some mistakes understanding the relations between ILP and NPC?
I am really appreciated that you can help me to solve this question.
Although the question might be considered off-topic, there are basically a few points to address.
You are right that general integer linear programming is NP-hard.
If a specific problem needs to be solved and general integer linear programming is the most specific way to formulate it, then nothing can be done about it; some problems are just hard to solve.
In some cases, it is possible to use the LP relaxation instead, either as a heuristic or some approximation ratio can be proven.
The key point here is that integer linear programming is a widespread formalism for expressing problems. Basically I understand your question as the follwing one.
"Why do people use a model that is algorithmically hard to solve to
describe practical problems?"
Well, if that shortcoming could be circumvented in general, it would be a good idea to express every problem there is in terms of sorting, which is algorithmically easy.
NP-hard refers to the complexity of algorithms in the worst case. For most NP-hard problems, we have effective algorithms (heuristic or exact) that perform well most of the time, even if they do not perform well in the worst case. ILP is therefore a very useful tool in practice, even if there are some problems that it doesn't do well on.
I have a hammer. There are some jobs that my hammer is just no good for, or would take a very long time on. But it's still a very useful tool, because it can do a lot of jobs for me very well.
ILP is, in many ways, the same thing.
I'm giving TDD a serious try today and have found it really helpful, in line with all the praise it receives.
In my quest for exercises on which to learn both Python and TDD i have started to code SPOJ exercises using the TDD technique and i have arrived at a question:
Given that all of SPOJ's exercises are mostly math applied to programming; How does one test a math procedure as in the TDD Fashion? Sample known-to-be-correct data? Test against a known implementation?
I have found that using the sample data given in the problem itself is valuable but it feels overkill for something you can test so quickly using the console, not to mention the overhead to design your algorithms in a testable fashion (Proxying the stdout and stdin objects is nothing short of too much work for a really small reward), and while it is good because it forces you to think your solutions in testable terms i think i might be trying way too hard on this.
All guidance is welcome
test all edge cases. Your program is most likely to fail when the input is special for some reason: negative, or zero values, very large values, inputs in reverse order, empty inputs. You might also want some destructive tests to see how large the inputs can be before things break or grind to a halt.
Sphere online Judge may not be the best fit of TDD. For one the input data might be better behaved than what a real person might put in. Secondly there is a code size limit on some problems. An extensive test-suite might put you over that limit.
You might want to take a look at Uncle Bob's "Transformation Priority Premise." It offers some guidance on how to pick a sequence of tests to test drive an algorithm.
Use sample inputs for which you know the results (outputs). Use equivalence partitioning to identify a suitable set of test cases. With maths code you might find that you can not implement as incrementally as for other code: you might need several test cases for each incremental improvement. By that I mean that non maths code can typically be thought of as having a set of "features" and you can implement one feature at a time, but maths code is not much like that.
I've searched for meaning of heuristic function but everything I got is that it's function that ranks alternatives in search algorithms. But I suppose that it is not full definition of heuristics. As an example, heuristic of tree rank is used in Disjoint Union Set problem, but there's no searching!
I still don't understand, what does mean heuristic. Do you know any math definitions?
In its broadest sense, heuristics is the art of obtaining adequate results in short time from incomplete information.
As an example, the best known algorithms that solve the travelling salesman problem (i.e. finding the cheapest Hamiltonian cycle in a edge-weighted graph) have exponential time complexity. A heuristic algorithm to this problem is one that will often find a Hamiltonian cycle that is not much more expensive than the optimal solution, but will use only polynomial time.
A heuristic is a problem solving technique that comes from intuition,common sense,experience and is basically an easier and short method to solve problems.
It is applicable to human as well as machines.
Example: For example you choose to eat one of the fruits from the bucket because it looks fresh and ripe.(what you thought before picking up the fruit was an example of heuristic)
A heuristic function can be explained with an example of objective type question.
Suppose you are facing an objective test of 4 options of which only one is correct.
Now there is a question whose answer you do not know.
But then you look at the options and start thinking 1st is least probable and so is 3rd and so and so...
So basically you are rejecting least probable answers based on you experience and knwoledge related to those options
Thus it can be said that you are using a heuristic function.
Similarly for computers a heuristic function can be designed by keeping in mind the constraints of what you want from your program to do.
Assuming that I am interested in performance rather than portability of my linear algebra iterative multi-threaded solver and that I have the results of profiling my code in hand, how do I go about tuning my code to run optimally on that machine of my choice?
The algorithm involves Matrix-Vector multiplications, norms and dot-products. (FWIW, I am working on CG and GMRES).
I am working on codes which are of matrix size roughly equivalent to the full size of the RAM (~6GB). I'll be working on Intel i3 Laptop. I'll be linking my codes using Intel MKL.
Specifically,
Is there a good resource(PDF/Book/Paper) for learning manual tuning? There are numerous things that I learnt by doing for instance : Manual Unrolling isn't always optimal or about compiler flags but I would prefer a centralized resource.
I need something to translate profiler information to improved performance. For instance, my profiler tells me that my stacks of one processor are being accessed by another or that my mulpd ASM is taking too much time. I have no clue what these mean and how I could use this information for improving my code.
My intention is to spend as much time as needed to squeeze as much compute power as possible. Its more of a learning experience than for actual use or distribution as of now.
(I am concerned about manual tuning not auto-tuning)
Misc Details:
This differs from usual performance tuning since the major portions of the code are linked to Intel's proprietary MKL library.
Because of Memory Bandwidth issues in O(N^2) matrix-vector multiplications and dependencies, there is a limit to what I could manage on my own through simple observation.
I write in C and Fortran and I have tried both and as discussed a million times on SO, I found no difference in either if I tweak them appropriately.
Gosh, this still has no answers. After you've read this you'll still have no useful answers ...
You imply that you've already done all the obvious and generic things to make your codes fast. Specifically you have:
chosen the fastest algorithm for your problem (either that, or your problem is to optimise the implementation of an algorithm rather than to optimise the finding of a solution to a problem);
worked your compiler like a dog to squeeze out the last drop of execution speed;
linked in the best libraries you can find which are any use at all (and tested to ensure that they do in fact improve the performance of your program;
hand-crafted your memory access to optimise r/w performance;
done all the obvious little tricks that we all do (eg when comparing the norms of 2 vectors you don't need to take a square root to determine that one is 'larger' than another, ...);
hammered the parallel scalability of your program to within a gnat's whisker of the S==P line on your performance graphs;
always executed your program on the right size of job, for a given number of processors, to maximise some measure of performance;
and still you are not satisfied !
Now, unfortunately, you are close to the bleeding edge and the information you seek is not to be found easily in books or on web-sites. Not even here on SO. Part of the reason for this is that you are now engaged in optimising your code on your platform and you are in the best position to diagnose problems and to fix them. But these problems are likely to be very local indeed; you might conclude that no-one else outside your immediate research group would be interested in what you do, I know you wouldn't be interested in any of the micro-optimisations I do on my code on my platform.
The second reason is that you have stepped into an area that is still an active research front and the useful lessons (if any) are published in the academic literature. For that you need access to a good research library, if you don't have one nearby then both the ACM and IEEE-CS Digital Libraries are good places to start. (Post or comment if you don't know what these are.)
In your position I'd be looking at journals on 2 topics: peta- and exa-scale computing for science and engineering, and compiler developments. I trust that the former is obvious, the latter may be less obvious: but if your compiler already did all the (useful) cutting-edge optimisations you wouldn't be asking this question and compiler-writers are working hard so that your successors won't have to.
You're probably looking for optimisations which like, say, loop unrolling, were relatively difficult to find implemented in compilers 25 years ago and which were therefore bleeding-edge back then, and which themselves will be old and established in another 25 years.
EDIT
First, let me make explicit something that was originally only implicit in my 'answer': I am not prepared to spend long enough on SO to guide you through even a summary of the knowledge I have gained in 25+ years in scientific/engineering and high-performance computing. I am not given to writing books, but many are and Amazon will help you find them. This answer was way longer than most I care to post before I added this bit.
Now, to pick up on the points in your comment:
on 'hand-crafted memory access' start at the Wikipedia article on 'loop tiling' (see, you can't even rely on me to paste the URL here) and read out from there; you should be able to quickly pick up the terms you can use in further searches.
on 'working your compiler like a dog' I do indeed mean becoming familiar with its documentation and gaining a detailed understanding of the intentions and realities of the various options; ultimately you will have to do a lot of testing of compiler options to determine which are 'best' for your code on your platform(s).
on 'micro-optimisations', well here's a start: Performance Optimization of Numerically Intensive Codes. Don't run away with the idea that you will learn all (or even much) of what you want to learn from this book. It's now about 10 years old. The take away messages are:
performance optimisation requires intimacy with machine architecture;
performance optimisation is made up of 1001 individual steps and it's generally impossible to predict which ones will be most useful (and which ones actually harmful) without detailed understanding of a program and its run-time environment;
performance optimisation is a participation sport, you can't learn it without doing it;
performance optimisation requires obsessive attention to detail and good record-keeping.
Oh, and never write a clever piece of optimisation that you can't easily un-write when the next compiler release implements a better approach. I spend a fair amount of time removing clever tricks from 20-year old Fortran that was justified (if at all) on the grounds of boosting execution performance but which now just confuses the programmer (it annoys the hell out of me too) and gets in the way of the compiler doing its job.
Finally, one piece of wisdom I am prepared to share: these days I do very little optimisation that is not under one of the items in my first list above; I find that the cost/benefit ratio of micro-optimisations is unfavourable to my employers.