Related
What is the difference between the Maximal Information Coefficient and Hierarchical Agglomerative Clustering in identifying functional and non functional dependencies.
Which of them can identify duplicates better?
This question doesn't make a lot of sense, sorry.
The MIC and HAC have close to zero in common.
The MIC is a crippled form of "correlation" with a very crude heuristic search, and plenty of promotion video and news announcements, and received some pretty harsh reviews from statisticians. You can file it in the category "if it had been submitted to an appropriate journal (rather than the quite unspecific and overrated Science which probably shouldn't publish such topics at all - or at least, get better reviewers from the subject domains. It's not the first Science article of this quality....), it would have been rejected (as-is - better expert reviewers would have demanded major changes)". See, e.g.,
Noah Simon and Robert Tibshirani, Comment on “Detecting Novel Associations in Large Data Sets” by Reshef et al., Science Dec. 16, 2011
"As one can see from the Figure, MIC has lower power than dcor, in every case except the somewhat pathological high-frequency sine wave. MIC is sometimes less powerful than Pearson correlation as well, the linear case being particularly worrisome."
And "tibs" is a highly respected author. And this is just one of many surprised that such things get accepted in such a high reputation journal. IIRC, the MIC authors even failed to compare to "ancient" alternatives such as Spearman, to modern alternatives like dCor, or to properly conduct a test of statistical power of their method.
MIC works much worse than advertised when studied with statistical scrunity:
Gorfine, M., Heller, R., & Heller, Y. (2012). Comment on "detecting novel associations in large data sets"
"under the majority of the noisy functionals and non-functional settings, the HHG and dCor tests hold very large power advantages over the MIC test, under practical sample sizes; "
As a matter of fact, MIC gives wildly inappropriate results on some trivial data sets such as a checkerboard uniform distribution ▄▀, which it considers maximally correlated (as correlated as y=x); by design. Their grid-based design is overfitted to the rather special scenario with the sine curve. It has some interesting properties, but these are IMHO captured better by earlier approaches such as Spearman and dCor).
The failure by the MIC authors to compare to Spearman is IMHO a severe omission, because their own method is also purely rank-based if I recall correctly. Spearman is Pearson-on-ranks, yet they compare only to Pearson. The favorite example of MIC (another questionable choice) is the sine wave - which after rank transformation actually is busy a zigzag curve, not a sine anymore). I consider this to be "cheating" to make Pearson look bad, by not using the rank transformation with Pearson, too. Good reviewers would have demanded such a comparison.
Now all of these complaints are essentially unrelated to HAC. HAC is not trying to define any form if "correlation", but it can be used with any distance or similarity (including correlation similarity).
HAC is something completely different: a clustering algorithm. It analyzes a larger rows, not two (!) columns.
You could even combine them: if you compute the MIC foe every pair of variables (but I'd rather use Pearson correlation, Spearman correlation, or distance correlation dCor instead), you can use HAC to cluster variables.
For finding aftual duplicates, neither is a good choice. Just sort your data, and duplicates will follow each other. (Or, if you sort columns, next to each other).
I would like for this to become a sign-post for various time series breakout/change/disturbance detection methods in R. My question is to describe the motivation and differences in approaches with each of the following packages. That is, when does it make more sense to use one approach over the other, similarities/differences, etc.
Packages in question:
strucchange (example here)
changepoint (example here)
BreakoutDetection (link includes simple example)
qcc's Control Charts (tutorial here)
bfast
Perhaps (?) to a lesser extent: AnomalyDetection and mvOutlier
I am hopeful for targeted answers. Perhaps a paragraph for each method. It is easy to slap each of these across a time series but that can come at the cost of abusing/violating assumptions. There are resources that provide guidelines for ML supervised/unsupervised techniques. I (and surely others) would appreciate some guidepost/pointers around this area of time-series analysis.
Two very different motivations have led to time-series analysis:
Industrial quality control and detection of outliers, detecting deviations from a stable noise.
Scientific understanding of trends, where the understanding of trends and of their determinants is of central importance.
Of course both are to a large extent two sides of a same coin and the detection of outliers can be important for time series cleaning before trends analysis. I will nevertheless try hereafter to use this distinction as a red line to explain the diversity of packages offered by R to study time-series.
In quality control, the stability of the mean and standard deviation are of major importance as exemplified by the history of one of the first statistical efforts to maintain industrial quality, the control chart. In this respect, qcc is a reference implementation of the most classical quality control diagrams: Shewhart quality control, cusum and EWMA charts.
The old but still active mvoutlier and the more recent AnomalyDetection focus on outliers detection. mvoutlier mainly uses the Mahalanobis distance and can work with two dimensional datasets (rasters) and even multi-dimensional datasets using using the algorithm of Filzmoser, Maronna, and Werner (2007). AnomalyDetection uses the time series decomposition to identify both local anomalies (outlyers) and global anomalies (variations not explained by seasonal patterns).
and BreakoutDetection
As AnomalyDetection, BreakoutDetection have been open-sourced by twitter in 2014. BreakoutDetection, open-sourced in 2014 by Twitter, intends to detect breakouts it time series, that is groups of anomalies, using non-parametric statistics. The detection of breakouts comes very close to the detection of trends and understanding of patterns. In a similar optic, the brca package focuses on the analysis of irregularly sampled time-series, particularly to identify behavioral changes in animal movement.
Definitely shifting to determination of changes in trends changepoint implements multiple (simple) frequentist and non-parametric methods to detect single or multiple breaks in time series trends. strucchange allows to fit, plot and test trend changes using regression models. Finally, bfast builds on strucchange to analyze raster (e.g. satellite images) time series and handles missing data.
I am trying to build a basic Emotion detector from speech using MFCCs, their deltas and delta-deltas. A number of papers talk about getting a good accuracy by training GMMs on these features.
I cannot seem to find a ready made package to do the same. I did play around with scilearn in Python, Voicebox and similar toolkits in Matlab and Rmixmod, stochmod, mclust, mixtools and some other packages in R. What would be the best library to calculate GMMs from trained data?
Challenging problem is training data, which contains the emotion information, embedded in feature set. The same features that encapsulate emotions should be used in the test signal. The testing with GMM will only be good as your universal background model. In my experience typically with GMM you can only separate male female and a few unique speakers. Simply feeding the MFCC’s into GMM would not be sufficient, since GMM does not hold time varying information. Since emotional speech would contain time varying parameters such as pitch and changes in pitch over periods in addition to the frequency variations MFCC parameters. I am not saying it not possible with current state of technology but challenging in a good way.
If you want to use Python, here is the code in the famous speech recognition toolkit Sphinx.
http://sourceforge.net/p/cmusphinx/code/HEAD/tree/trunk/sphinxtrain/python/cmusphinx/gmm.py
Closed. This question is opinion-based. It is not currently accepting answers.
Want to improve this question? Update the question so it can be answered with facts and citations by editing this post.
Closed 1 year ago.
Improve this question
This is not a "is math required for programming?" question.
I always thought that for programming, a scary amount of complicated math would be involved (I only got as far as intermediate algebra in college because I'm bad with it).
However, I just got my first job as a developer and have found there is not a ton of math above basic arithmetic(as of yet). I also read on a question here in SO that math is more used to ensure the would-be developer can understand complex problems and solve them.
So I guess is there a different kind of programming where a math level above algebra is needed? My guess would be like geometry and other disciplines for video game programming where you create shapes in 3D and play with time and space in environments. What else requires a high level of math?
EDIT: Wow, lot of answers. One of which made me think of another similar question...say in programs like photoshop, what kind of math(or overall work) is involved in making something twist, crop, edit, and color things like images?
I think there are at least two types of answer to this question. Firstly, there are the sorts of programming which is problems which come from a field where maths is important. These include:
finance
science research, e.g. physical modelling
engineering implementations, e.g. stress analysis, chemical engineering
experimental science, e.g. physics, psychology
mathematics itself
cryptography
image processing
signal processing
And then there are the sorts of programming where the target is not necessarily mathematical, but the process of achieving that target needs some maths. These include:
games
optimisation processes
high-complexity systems, e.g. flight control software
high-availability systems, e.g. industrial process monitoring and/or safety
complex data transformations, e.g. compiler design
and so on. Various of these require various levels and aspects of mathematics.
Gaming and simulation are obvious answers. The math is not difficult, but it is clearly there.
For example, say you want to build some sort of asteroids game. You'll need to figure out the position of your space ship. That's an vector. Now you want the ship to travel in a certain direction a certain direction every frame. You'll need to add some sort of delta-x to x, and delta-y to y, so your motion is another vector: . In an asteroids game, you accelerate in the direction you're pointing, so whenever you hit the 'thrust' key, you'll need to calculate the delta of dx and dy, or an acceleration vector,
(yep, this is the same dx from calculus class, but now I'm building robot zombie opposums with it.)
But that's not all. If you act now, I'll throw in some trig. Normally you think of motion and acceleration as angle and distance(r and theta,) but the programming language normally needs these values in dx, dy format. Here's where you'll use some trig: dx = r * cos (theta) and dy = r * sin(theta)
But, let's say you want to have a planet with a gravitational pull? You'll need to approximate gravity in a way that gives you orbit behavior (elliptical orbits, firing changes the altitude of the other side of the orbit, and so on.) This is easiest to do if you understand Newton's law of universal gravitation: f = ((sqrt(m1 * m2))/d^2) * G. This tels you how much 'planetward' force to add to your ship every frame.Multiply this value by a normalized vector pointing from the spaceship to the planet, and add that as a new motion vector.
Believe it or not, I encourage people who don't like math to take game programming courses. Often they discover that math can be (dare I say it) kind of fun when they're using it on problems that involve exploding cows in space.
As another example, think about optimizing a sound wave. The analog wave has an infinite number of points. It's impossible to store them all in a digital signal, so we break the audio signal into a large number of small segments, then measure every segment. If we want a perfect representation, grab an infinitely large number of infinitely small time slices.
Draw this out, and you've created the Riemann sum, the foundational idea of Integration (and in fact of Calculus)
One more example:
A friend of mine (a biology professor) was trying to build a 'sim city'-style simulation of a lake ecosystem. He was a decent programmer, but he got all bogged down in the various calculations. He wanted the user to vary usage of the lake (outboard motors, fishing restrictions, and dumping restrictions) and then to see how that affected levels of Nitrogen and other key elements.
He tried all kinds of crazy if-then structures with nested conditions and ugly Boolean logic, but never had a clean solution.
We took actual data, ran it through Excel, and found a trendline that accurately reflected his data with a simple logarithmic formula.
Hundreds of lines of messy code were replaced with a simple formula.
Here's a few general places:
Graphics
Cryptography
Statistics
Compression
Optimization
There are also a lot of specific problem areas where complex math is required, but this is due more to the nature of the program and less about programming in general. Things like financial applications fall into this.
Any kind of numerical analysis, like in geophysics or petroleum exploration.
I once built a tool for accident investigators that required a lot of trigonometry.
In commercial programming, not so much math as arithmetic.
All programming requires math. I think that the difference between people with mathematical backgrounds and people with programming backgrounds is how they approach and answer problems. However, if you are advancing your programming skills you are likely unknowingly advancing your mathematical skills as well (and vise versa).
If you abstractly look at both programming and mathematics you'll see they're identical in their approaches: they both strive to answer problems using very fundamental building blocks.
There is a pretty famous essay by Edsger W. Dijkstra which he attempts to answer your exact question. It is called: On the Interplay Between Mathematics and Programming.
Game programming (especially 3-D, as you mentioned) has a lot of "more advanced" math. For that matter, any projects where you're modeling a system (e.g. physics simulation).
Crypto also uses different forms of math.
Robotics requires hardcore matrices, and AI requires all kinds of math.
Quite alot of complex(ish) math in the Finance sector. Other than that and Trig for 3d I can't honestly think of much else.
Im sure there are some though.
Many seemingly non-mathematical industries such as Pharmaceuticals (eg. BioInformatics), Agriculture, Marketing and in general, any "Business Intelligence" relies heavily on statistics. System performance, routing, scheduling, fault tolerance -- the list goes on....
Digital signal processing and AI/simulation/agents are others.
Animation via code, especially when you try to model real physics, also needs math.
I'm a Mathematics graduate and I have to say that the only places where I've really seen any Maths being used (above very basic arithmetic) is understanding / simplifying logical statements, so for example things like the equivalence of these two statements:
(!something) && (!otherThing)
!(something || otherThing)
Apart from that the only time that you would need more complex Maths is when you are working with computer graphics or some subject which is Maths based (e.g. finance or computations) - in which case knowing the Maths is more about understanding your subject than it is about the actual programming.
I work on software that's rather similar to CAD software, and a good grasp of geometry and at least an idea of computational geometry is necessary.
I work in computational chemistry. You need a lot of linear algebra and general understanding of techniques such as Taylor expansion, integrals, gradients, Hessians, Fourier transformation (and in general, expansion on a basis set), differential equations. It's not terribly complex math, but you have to know it.
Statistics is used heavily in businesses performing Quality Assurance and Quality Analysis. My first development job was on a contract at the USDA; these were standard "line-of-business" applications except their line-of-business happened to involve a lot of statistical analysis!
Image compression and image recognition both use Fourier series (including classic sine wave series and other orthogonal series such as wavelet transformations) which has some pretty heavy theory usually not covered until a graduate level course in mathematics or engineering.
Non-linear optimization, constrained optimization, and system estimation using hidden models likewise use a significant amount of advanced mathematical analysis.
Computer science is math. Programming is programmers job. They are related, but the two areas don't exactly overlap, so I see the point of your question.
Scientific computing and numerical analysis obviously require a solid base of linear algebra, geometry, advanced calculus and maybe more. And the whole study of algorithm, data structures and their complexity and properties makes use of discrete mathematics, graph theory, as well as calculus and probability. Behind the simple JPEG standard there's a lot of information theory, coding theory, fourier analysis... And these are only some examples.
Although a computer scientist could even work an entire life without writing down a single line of code, as well as the best programmer in the world could know just a little of math, the fact is that computers perform algorithms. And algorithms require math. I suggest you to take a look at Donald Knuth's "The Art of Computer Programming" to have an idea of what is underneath the "simple" programming thing.
I got my masters degree in meteorology, and I can tell you for that field and other applied physics fields, the kind of coding you will be doing requires an immense amount of mathematics. A lot of what you are coding are things like time differential of equations.
For things like writing code for games, however, you're not always going to be doing a lot math in your code. Gaming requires lots of logic. The part of game coding where math comes in is when you have write physics engines and things like that.
I'm not looking for a general discussion on if math is important or not for programming.
Instead I'm looking for real world scenarios where you have actually used some branch of math to solve some particular problem during your career as a software developer.
In particular, I'm looking for concrete examples.
I frequently find myself using De Morgan's theorem when as well as general Boolean algebra when trying to simplify conditionals
I've also occasionally written out truth tables to verify changes, as in the example below (found during a recent code review)
(showAll and s.ShowToUser are both of type bool.)
// Before
(showAll ? (s.ShowToUser || s.ShowToUser == false) : s.ShowToUser)
// After!
showAll || s.ShowToUser
I also used some basic right-angle trigonometry a few years ago when working on some simple graphics - I had to rotate and centre a text string along a line that could be at any angle.
Not revolutionary...but certainly maths.
Linear algebra for 3D rendering and also for financial tools.
Regression analysis for the same financial tools, like correlations between financial instruments and indices, and such.
Statistics, I had to write several methods to get statistical values, like the F Probability Distribution, the Pearson product moment coeficient, and some Linear Algebra correlations, interpolations and extrapolations for implementing the Arbitrage pricing theory for asset pricing and stocks.
Discrete math for everything, linear algebra for 3D, analysis for physics especially for calculating mass properties.
[Linear algebra for everything]
Projective geometry for camera calibration
Identification of time series / statistical filtering for sound & image processing
(I guess) basic mechanics and hence calculus for game programming
Computing sizes of caches to optimize performance. Not as simple as it sounds when this is your critical path, and you have to go back and work out the times saved by using the cache relative to its size.
I'm in medical imaging, and I use mostly linear algebra and basic geometry for anything related to 3D display, anatomical measurements, etc...
I also use numerical analysis for handling real-world noisy data, and a good deal of statistics to prove algorithms, design support tools for clinical trials, etc...
Games with trigonometry and AI with graph theory in my case.
Graph theory to create a weighted graph to represent all possible paths between two points and then find the shortest or most efficient path.
Also statistics for plotting graphs and risk calculations. I used both Normal distribution and cumulative normal distribution calculations. Pretty commonly used functions in Excel I would guess but I actully had to write them myself since there is no built-in support in the .NET libraries. Sadly the built in Math support in .NET seem pretty basic.
I've used trigonometry the most and also a small amount a calculus, working on overlays for GIS (mapping) software, comparing objects in 3D space, and converting between coordinate systems.
A general mathematical understanding is very useful if you're using 3rd party libraries to do calculations for you, as you ofter need to appreciate their limitations.
i often use math and programming together, but the goal of my work IS the math so use software to achive that.
as for the math i use; mostly Calculus (FFT's analysing continuous and discrete signals) with a slash of linar algebra (CORDIC) to do trig on a MCU with no floating point chip.
I used a analytic geometry for simple 3d engine in opengl in hobby project on high school.
Some geometry computation i had used for dynamic printing reports, where was another 90° angle layout than.
A year ago I used some derivatives and integrals for store analysis (product item movement in store).
Bot all the computation can be found on internet or high-school book.
Statistics mean, standard-deviation, for our analysts.
Linear algebra - particularly gauss-jordan elimination and
Calculus - derivatives in the form of difference tables for generating polynomials from a table of (x, f(x))
Linear algebra and complex analysis in electronic engineering.
Statistics in analysing data and translating it into other units (different project).
I used probability and log odds (log of the ratio of two probabilities) to classify incoming emails into multiple categories. Most of the heavy lifting was done by my colleague Fidelis Assis.
Real world scenarios: better rostering of staff, more efficient scheduling of flights, shortest paths in road networks, optimal facility/resource locations.
Branch of maths: Operations Research. Vague definition: construct a mathematical model of a (normally complex) real world business problem, and then use mathematical tools (e.g. optimisation, statistics/probability, queuing theory, graph theory) to interrogate this model to aid in the making of effective decisions (e.g. minimise cost, maximise efficency, predict outcomes etc).
Statistics for scientific data analyses such as:
calculation of distributions, z-standardisation
Fishers Z
Reliability (Alpha, Kappa, Cohen)
Discriminance analyses
scale aggregation, poling, etc.
In actual software development I've only really used quite trivial linear algebra, geometry and trigonometry. Certainly nothing more advanced than the first college course in each subject.
I have however written lots of programs to solve really quite hard math problems, using some very advanced math. But I wouldn't call any of that software development since I wasn't actually developing software. By that I mean that the end result wasn't the program itself, it was an answer. Basically someone would ask me what is essentially a math question and I'd write a program that answered that question. Sure I’d keep the code around for when I get asked the question again, and sometimes I’d send the code to someone so that they could answer the question themselves, but that still doesn’t count as software development in my mind. Occasionally someone would take that code and re-implement it in an application, but then they're the ones doing the software development and I'm the one doing the math.
(Hopefully this new job I’ve started will actually let me to both, so we’ll see how that works out)