Hidden Markov Model in R - Predict the next observation with RHmm - r

This is my first post on StackOverflow and I could use a little help... Please forgive me if I am not following the correct posting protocols.
There is another example in the StackOverflow for which I am heavily basing my work off of but I cant quite figure out how to adapt the code. Most importantly, I am looking at the solution to the question provided.
Here is the link:
Getting the next observation from a HMM gaussian mixture distribution
Some background:
RHmm - version 2.1.0 downloaded from R Forge.
RStudio - 0.98.953
R - 3.0.2 32 bit
I am trying to figure out the following issues with my code:
How do I amend the solution from the link above (prediction of the next observation) to work with my Baum-Welch model?
Ex. hm_model <- HMMFit(obs=TWII_Train, nStates=5)
The R / RStudio session aborts when I run the Baum-Welch version of the hm_model <- HMMFit(obs=TWII_Train, dis="MIXTURE", nStates=5, nMixt=4). Can you recreate the error and propose a workaround?
Here is my R code:
library(quantmod)
library(RHmm)
getSymbols("^TWII")
TWII_Subset <- window(TWII, start=as.Date("2012-01-01"), end = as.Date("2013-04-01"))
TWII_Train <- cbind(TWII_Subset$TWII.Close - TWII_Subset$TWII.Open,
TWII_Subset$TWII.Volume)
hm_model <- HMMFit(obs=TWII_Train, nStates=5)
VitPath <- viterbi(hm_model, TWII_Train)

I'm not a user of this package and this is not really an answer, but a comment would obscure some of the structures. It appears that the "proportion" value of your model is missing (so the structures are different. The "mean" value looks like this:
$ mean :List of 5
..$ : num [1:2] 6.72 3.34e+06
..$ : num [1:2] -12.4 2420174.5
..$ : num [1:2] -2.4 1832546.5
..$ : num [1:2] -10.4 1432636.1
..$ : num [1:2] 5.02 1.96e+06
I also suspect that you should be using 2 and 5 rather than 4 and 5 for m and n. Look at the rest of the model with:
str(hm_model)

Related

How to balance a dataset using SCUT and the scutr-package in R

I was given a machine learning project in R by a colleague who can no longer work on it. I am currently trying to balance the used dataset with the SCUT function in the scutr package and I keep running into the following Problem:
The project I am working with contains the base dataset, formatted as a standard dataframe that contains different information on different YouTube channels (URL, name, description, etc.) and also a classification of 4 classes (hkgeschlecht). The classification is numerical, some of the other information as well, but the channel description for example is a text:
'data.frame': 199 obs. of 6 variables:
$ ctitle : chr "Gaming Kati" "EinfallsReich" "Frank / Generation - E" "Gladiator Glubschi" ...
$ cdescr : chr "Dieser Kanal ist einfach ein Kanal von einem Mädel, welches einfach im Animal Crossing hype ist <U+0001F61D><U+"| __truncated__ "Kurze und EinfallsReiche Fakten Videos mit folgenden Themen:\n\n "| __truncated__ "Ich bin Frank aus Hamburg...\n\n...glücklicher Ehemann und Vater von zwei fantastischen Jungs. \n\nZu meinen gr"| __truncated__ "Gladiator Glubschi\n- Ein Glubschi\n- Zwei krasse Kanäle\n- Drei Unterhaltung!\n\nUnd damit erstmal danke fürs "| __truncated__ ...
$ cthumbnailurl: chr "https://yt3.ggpht.com/a/AATXAJwsWCPoVZ6g-uk_9UbMU3NqOU-QuoQyunPoYg=s240-c-k-c0xffffffff-no-rj-mo" "https://yt3.ggpht.com/a/AATXAJxunaT5qD2CbS7AQodCYq-HDOVee87NYBnRnw=s240-c-k-c0xffffffff-no-rj-mo" "https://yt3.ggpht.com/a/AATXAJzaeY6aZJuWpCsa8ul1CXHmQ1bC6reTWk9mTw=s240-c-k-c0xffffffff-no-rj-mo" "https://yt3.ggpht.com/a/AATXAJx0pmglui0v3YZblGuT1yOdNTm33qVP7mLXxQ=s240-c-k-c0xffffffff-no-rj-mo" ...
$ cviews : int 1348087 2764 229744 15556 1884 1077314 158044 113570 25495 2364116 ...
$ csubscriber : int 13000 0 1140 320 0 7940 623 823 406 34700 ...
$ hkgeschlecht : num 2 99 1 1 1 2 1 1 1 2 ...
The project uses a Naive Bayes Classifier and thus the channel description (sdescr) in the dataframe is transformed into a document feature matrix dfm which then is split into a training dataset and test dataset. This all works out fine and the model gives me decent predictions.
However the main dataset is unbalanced as one class is much more dominant than the others. I now want to balance this dataset using the SCUT-method so that the prediction of the minority classes improves. I had planned on using the scutr package and the SCUT function in it since it is seems fairly straight forward.
Now my problem is, if I apply the function to main dataset like this:
ret <- SCUT(mldata, "hkgeschlecht", oversample = oversample_smote, undersample = undersample_hclust,)
I get this error:
Error in get.knnx(data, query, k, algorithm) : Data non-numeric
I assume that is due to the differently formated variables in the dataframe.
But if I try to apply it only to the training dataset like this:
ret <- SCUT(testdfm1, testdfm1#docvars$docvars, oversample = oversample_smote, undersample = undersample_hclust,)
I get this error:
Error in validate_dataset(data, cls_col) :
Column not found in data: 22211299112111312111122333311133211111111
Which I assume is due to the SCUT function needing a dataframe format and not a document feature matrix.
My question thus is: How I can apply the SCUT method in this case? Is there a way to make the function work with a document feature matrix, say to get it to recognize the column with the classification? Would that even make sense? Or do I have to go about it in a completely different way?

Plotting Timeseries with map() from purrr

I am trying to generate a plot that is similar to this:
A walkthrough is provided here -> https://medium.com/#erickramer/beautiful-data-science-with-functional-programming-and-r-a3f72059500b
However the code supplied on this website isn't generating a plot for me, instead I get this error:
> forecasts1 = tsdf %>%
+ map(auto.arima) %>%
+ map(forecast, h=10)
Error in is.constant(x) :
(list) object cannot be coerced to type 'double'
This is despite the fact that I have replicated their data formatting precisely. Here are our datasets for comparison:
> str(tsdf)
Classes ‘tbl_df’, ‘tbl’ and 'data.frame': 89 obs. of 1 variable:
$ time_series:List of 89
..$ 1_1 : Time-Series from 2013 to 2017: 8981338 10707490 11410597 10816217 12263765 ...
..$ 1_10 : Time-Series from 2013 to 2017: 12645212 13510638 13133558 13542970 16074675 ...
..$ 1_2 : Time-Series from 2013 to 2017: 19028892 20626896 19952328 20865263 22547313 ...
..$ 1_3 : Time-Series from 2013 to 2017: 7081624 8317481 8374427 8330653 9643845 ...
..$ 1_4 : Time-Series from 2013 to 2017: 25421637 30934941 30756101 27977317 32417608 ...
And the provided example data (upon which the code did work, according to the website):
> str(time_series)
List of 9
$ Germany : Time-Series [1:52] from 1960 to 2011: 684721 716424 749838 ...
$ Singapore : Time-Series [1:52] from 1960 to 2011: 7208 7795 8349 ...
$ Finland : Time-Series [1:37] from 1975 to 2011: 85842 86137 86344 ...
I can't seem to figure it out, though it may have something to do with the fact that their timeseries has one solid endpoint, yet my timeseries have several different monthly endpoints.
Any help with this is greatly appreciated!
* UPDATE *
After applying Akruns suggestion I stored exclusively the time-series vector in a list like so:
tsdf <- akrun %>%
select(time_series)
I then fit the model like this:
tsdf$time_series %>% map(auto.arima) %>%
map(forecast, h=12)
...and then the plot...
... looks awful.
Do I need to convert y_axis scale? Or do some sort of differencing to the data before plotting the arima? Really appreciate any suggestions!

Read cell values without formatting into R with googlesheets

Would like to be able to read Google Sheets cell values into R with googlesheets package, but without any cell formatting applied (e.g. comma separators, percentage conversion, etc.).
Have tried gs_read() without specifying a range, which uses gs_read_csv(), which will "request the data from the Sheets API via the exportcsv link". Can't find a way to tell it to provide underlying cell value without formatting applied.
Similarly, tried gs_read() and specifying a range, which uses gs_read_cellfeed(). But can't find a way to indicate that I want un-formatted cell values.
Note: I'm not after the formulas in any cells, just the values without any formatting applied.
Example:
(looks like I'm not able to post image images)
Here's a screenshot of an example Google Sheet:
https://www.dropbox.com/s/qff05u8nn3do33n/Screenshot%202015-07-26%2008.42.58.png?dl=0
First and third columns are numeric with no formatting applied, 2nd column applies comma separators for thousands, 4th column applies percentage formatting.
Reading this sheet with the following code:
library(googlesheets)
gs <- gs_title("GoogleSheets Test")
ws <- gs_read(gs, ws = "Sheet1")
yields:
> str(ws)
Classes ‘tbl_df’, ‘tbl’ and 'data.frame': 3 obs. of 4 variables:
$ Number : int 123456 123457 123458
$ Number_wFormat : chr "123,456" "123,457" "123,458"
$ Percent : num 0.123 0.234 0.346
$ Percent_wFormat: chr "12.34%" "23.45%" "34.56%"
Would like to be able to read a worksheet that has formatting applied (ala columns 2 and 4), but read the unformatted values (ala columns 1 and 3).
At this point, I think your best bet is to fix the imported data like so:
> ws$Number_fixed <- type.convert(gsub(',', '', ws$Number_wFormat))
> ws$Percent_fixed <- type.convert(gsub('%', '', ws$Percent_wFormat)) / 100
> str(ws)
Classes ‘tbl_df’, ‘tbl’ and 'data.frame': 3 obs. of 6 variables:
$ Number : int 123456 123457 123458
$ Number_wFormat : chr "123,456" "123,457" "123,458"
$ Percent : num 0.123 0.234 0.346
$ Percent_wFormat: chr "12.34%" "23.45%" "34.56%"
$ Number_fixed : int 123456 123457 123458
$ Percent_fixed : num 0.123 0.234 0.346
I had some hope that post-processing with functions from readr would be a decent answer, but it looks like percentages and "currency" style numbers are open issues there too.
I have opened an issue to solve this better in googlesheets, one way or another.

text searching in R

I am trying to make a query to use in a R package named RISmed, which will search and downloaded relevant journal article information from pubmed database. I want to search two words always together, for example :
query= "gene sequencing"
search<-EUtilsSummary(query,type="esearch",db = "pubmed",mindate=2014, maxdate=2014, retmax=20)
If I use, above command, it will search gene and sequencing separately, then both gene and sequencing,that means if in whole text gene and sequencing exists, my command captures them but I want to search in such a way, that it will consider "Gene sequencing", two words always together. How can I write that query? Would anyone please help me?
Thanks in advance !
I would try this:
query <- '"gene sequencing"[Title/Abstract]'
The pubmed search engine does accept quoted strings and you just need to know how to preserve them within R. Using surrounding single quotes is one method. Using back-slashed quotes would be another. Notice that the returned value from my experiment with your code shows that escape-backslashing is how the implemeters of that package do it:
> str(search)
Formal class 'EUtilsSummary' [package "RISmed"] with 6 slots
..# db : chr "pubmed"
..# count : num 542
..# retmax : num 20
..# retstart : num 0
..# PMID : chr [1:20] "25548628" "25543043" "25542841" "25540641" ...
..# querytranslation: chr "\"gene sequencing\"[Title/Abstract] AND 2014[EDAT] : 2014[EDAT]"

Regarding creating a data set in accordance with a given data format

I am learning to use topicmodels package and R as well, and explored one of its example data set by using
str(testdata)
'data.frame': 3104 obs. of 5 variables:
$ Article_ID: int 41246 41257 41268 41279 41290 41302 41314 41333 41344 41355 ...
$ Date : chr "1-Jan-96" "2-Jan-96" "3-Jan-96" "4-Jan-96" ...
$ Title : chr "Nation's Smaller Jails Struggle To Cope With Surge in Inmates" "FEDERAL IMPASSE SADDLING STATES WITH INDECISION" "Long, Costly Prelude Does Little To Alter Plot of Presidential Race" "Top Leader of the Bosnian Serbs Now Under Attack From Within" ...
$ Subject : chr "Jails overwhelmed with hardened criminals" "Federal budget impasse affect on states" "Contenders for 1996 Presedential elections" "Bosnian Serb leader criticized from within" ...
$ Topic.Code: int 12 20 20 19 1 19 1 1 20 15 ...
If I want to create a data set according to the above format in R, how to do that?
test.data is a data.frame, one of the few fundamental R objects. You should probably start here: http://cran.r-project.org/doc/manuals/R-intro.pdf.
Some functions for creating data.frames are data.frame, read.table, read.csv. For each of these you can access their documentation by typing ?data.frame for example. Good luck.

Resources