I am plotting a model from lightgbm and am trying to view the plot tree. When I use plot.tree it works... however, the output of the tree does not match feature importance, nor does it match the # of leafs I have choosen in my optimzation of my parameters.
for example, Feature A is the most important feature in my feature importance plot, but this feature does not show up in my actual decision tree plot as a node to have a decision on. Also, one of my parameters is 22 leaves, but the tree plot has 24 leaves.
I am doing this within databricks environment using python.
any ideas what is happening?
I can't post code, sorry. anyone with a general idea of what is happening will help.
First of all, Ligthgbm is a Boosting ensemble method, which means that you create several tree in series.
So, which tree are you plotting? You have several trees, and only exploring one tree is not representative of how exactly the model works. For sure, if you check a few trees, your feature A should appear.
About different num_leaves, I don't have a clear answer. It makes no sense. I should have some code and output to analyze it (but I have seen in you comment that you can't provide it, don't worry). In theory, you shouldn't have any tree with more than 22 leaves if you specified this value... Anyway, you can try to use another hyperparameter: max_depth, which is quite similar, event better.
Related
In GBM model, following parameters are used -
col_sample_rate
col_sample_rate_per_tree
col_sample_rate_change_per_level
I understand how the sampling works and how many variables get considered for splitting at each level for every tree. I am trying to understand how many times each feature gets considered for making a decision. Is there a way to easily extract all sample of features used for making a splitting decision from the model object?
Referring to the explanation provided by H2O, http://docs.h2o.ai/h2o/latest-stable/h2o-docs/data-science/algo-params/col_sample_rate.html, is there a way to know 60 randomly chosen features for each split?
Thank you for your help!
If you want to see which features were used at a given split in a give tree you can navigate the H2OTree object.
For R see documentation here and here
For Python see documentation here
You can also take a look at this Blog (if this link ever dies just do a google search for H2OTree class)
I don’t know if I would call this easy, but the MOJO tree visualizer spits out a graphviz dot data file which is turned into a visualization. This has the information you are interested in.
http://docs.h2o.ai/h2o/latest-stable/h2o-genmodel/javadoc/overview-summary.html#viewing-a-mojo
I'm having a few issues running a simple decision tree within R using rpart.
I can't post my actual data for an example because of confidentiality, but here's the structure. I've blanked out a load of bits just because I've got my tin foil hat on today.
I've run the most basic model to predict MIX based on MIX_BEFORE and LIFESTAGE and I don't get a tree out of the end of it. I've tried using rpart.control and specifying the minsplit, it makes no difference.
Even when I add in a few more variables I still don't get a tree:
Yet... the second I remove the factor variables and attempt to build a tree using an integer, it works fine:
Any ideas at all?
Your data has a fairly strong class imbalance: 99% one class, 1% the other. So rpart can get 99% accuracy just by saying that everything is the majority class (which is what it is doing). Most variables will not be able to discriminate better than that, so you get trees with no branches like you did with the factor variables. Your _NBR variable happens to be more predictive for the small number of points with _NBR >= 7. But even your model that uses _NBR predicts almost all points are majority class. You may be able to get some help from This Cross Validated Post on how to deal with class imbalance.
I am interested in exploring surrogate based optimization. I am not yet writing opendao code, just trying to figure out to what extent OpenMDAO will support this work.
I see that it has a DOE driver to generate training data (http://openmdao.readthedocs.org/en/1.5.0/usr-guide/tutorials/doe-drivers.html), I see that it has several surrogate models that can be added to a meta model (http://openmdao.readthedocs.org/en/1.5.0/usr-guide/examples/krig_sin.html). Yet, I haven't found an example where the results of the DOE are passed as training data to the Meta-model.
In many of the examples/tutorials/forum-posts it seems that the training data is created directly on or within the meta model. So it is not clear how these things work together.
Could the developers explain how training data is passed from a DOE to a meta model? Thanks!
In openmdao 1.x, this kind of process isn't directly supported (yet) via a DOE, but it is definitely possible. There are two paths that you can take, which offer different benefits depending on your eventual goal.
I will separate the different scenarios based on a single high level classification:
1) You want to do gradient based optimization around the whole DOE/Metamodel combination. This would be the case if, for example, you wanted to use CFD to predict drag at a few key points, then use a meta-model to generate a drag polar for mission analysis. A great example of this kind of modeling can be found in this paper on simultaneous aircraft-mission design optimization..
2) You don't want to do gradient based optimization around the whole model. You might want to do gradient free optimization (like a Genetic algorithm). You might want to do gradient based optimization just around the surrogate itself, with fixed training data. Or you might not want to do optimization at all...
If you're use case falls under scenario 1 (or will eventually fall under this use case in the future), then you want to use a multi-point approach. You create one instance of your model for each training case, then you can mux the results into an array you pass into meta-model. This is necessary so that derivatives can
be propagated through the full model. The multi-point approach will work well, and is very parallelizable. Depending on the structure of the model you will use for generating the training data itself, you might also consider a slightly different multi-point approach with a distributed component or a series of distributed components chained together. If your model will support it, the distributed component approach is the most efficient model structure to use in this case.
If you're use case falls into scenario 2, you can still employ the multi-point approach if you like. It will work out of the box. However, you could also consider using a regular DOE to generate the training data. In order to do this, you'll need to use a nested-problem approach, where you put the DOE training data generation in a sub-problem. This will also work, though it will take a bit of extra coding on your part to get the array of results out of the DOE because thats not currently implemented.
If you wanted to use the DOE to generate the data, then pass it downstream to a surrogate that would get optimized on, you could use a pair of problem instances. This would not necessarily require that you make nested problems at all. Instead you just build a run-script that has one problem instance that uses a DOE, when its done you collect the data into an array. Then you could manually assign that to the training inputs of a meta-model in a second problem instance. Something like the following pseudo-code:
prob1 = Problem()
prob1.driver = DOE()
#set up the DOE variables and model ...
prob1.run()
training_data = prob1.driver.results
prob2 = Problem()
prob2.driver = Optimizer()
#set up the meta-model and optimization problem
prob2['meta_model.train:x'] = training_data
prob2.run()
I'd like to extract useful rules from Decision Trees/Random Forest in order to develop a more applicable way to handle the rules and predictions. So I need an application which makes the rules more understandable.
Any suggestions (e.g. visualizations, validation methods etc) for my purpose?
As far as WHY a particular split was chosen, the answer is always going to be: "Because that split created the best splitting of the target variable."
You referenced scikit-learn... Go ahead and briefly scan scikit-learn's documentation on Decision Trees... It has an example, which is exactly what you are asking for in the middle of the page. It looks like this:
The code to generate this plot is there also:
from sklearn.datasets import load_iris
from sklearn import tree
iris = load_iris()
clf = tree.DecisionTreeClassifier()
clf = clf.fit(iris.data, iris.target)
from sklearn.externals.six import StringIO
with open("iris.dot", 'w') as f:
f = tree.export_graphviz(clf, out_file=f)
There are several other graphical representations there also with accompanying code:
The SKL documentation is generally awesome and is very useful.
Hope this helps!
While this is certainly possible for Decision Trees and AN6U5 did a great job describing how, Random Forests use bundles of little trees that were trained using random subsets of the data and random subsets of the features. Thus each tree is optimal only in that limited setting of features and data. Since there are typically 100s or even 1000s of them, figuring out the context by examining the randomized data is going to be a thankless task. I don't think anyone does it.
However there are importance ranking for the features generated for Random Forests and pretty much all implementations will output them if requested. They turn out to be extremely useful.
Two of the most important ones are MDI (Mean Decrease Impurity) and MDA (Mean Decrease Accuracy). They are described in some detail in chapter 6 of this excellent work: http://arxiv.org/pdf/1407.7502v3.pdf
Again.
I need help one more time.
Nowadays I am getting into PHYLOCOM software for inferring characteristics of a phylogeny from different samples. This software allows you to calculate if your species are showing clustering or overdispersion within other populations in your analyses.
As input files you need a phylogenetic tree in NEWICK format and a sample file (.txt).
I have done two tests, one modifying my tree in R with 'ape' package this way:
compute.brlen(tree, main=expression(rho==10))
And the other one by this other 'ape' option:
tree$edge.length = tree$edge.length * 10
The first modification generates an output with an ultrametric tree while the second output is a non-ultrametric tree. If then I run PHYLOCOM itself by
phylocom comstruct
I get different results, not only in the values of the parameters, but also in the signification p-values.
My question is if anyone knows how should I run the PHYLOCOM to do these 'comstruct' analyses correctly, with an input of a ultrametric or non-ultrametric, and also what are the differences in running this in oone way or in another.
I know this is not a 'classical' question for stackoverflow forums, but maybe anyone that works with phylogeny could help me.
Thanks a lot.
I think I may be able to help, but unfortunately I cannot add comments to get more information so I will have to infer what you mean from the information given. I apologize if it doesnt help.
Firstly, you may want to consult the help for compute.brlen(). As there is no argument for "main" in this function. I think you have taken it from the example in the help file, but you may note that this is outside the compute.brlen function and in the plot function. It will give you a title in your plot.
To change the rho value in compute.brlen() you need to change the power argument.
For example:
compute.brlen(tree, power = 10)
This may be why you are getting different results for the different trees. Because there is no transformation being performed on your compute.brlen() tree.
I am not familiar with PHYLOCOM, so I can't help on that front. But ultrametric and non-ultra-metric trees will give different relationships between the tips of the tree, so I would not be surprised that they give different results. I should note that I am not super confident on the differences in the analysis of ultrametric and non-ultrametric trees, but from looking at the plotted differences I would assume that this is true.