I have a client that needs/wants to upgrade their Drupal 7 site to Drupal 8. The site has over 50,000 articles, and most of the content was generated using the original "article" node/content type...but that original node has been customized to include a few more fields. The client wants me to create another custom content type for a new site feature, but I'd prefer to get the site upgraded to Drupal 8 before I begin work on the new features so I don't have to go through even more migration hassles in the future.
I have read through all the Drupal 6/7 to 8 Migration docs starting with this page: https://www.drupal.org/docs/8/upgrade/upgrading-from-drupal-6-or-7-to-drupal-8
I have tried to use the migration tools as per this write up with both the webui version, and the drush version. Both methods appear to use the same migration code, as both methods generate the same errors (see below):
First, here's what "drush migrate-status" gives us:
serverpilot#XXX.XXX.XXX.XXX:~/apps/therooster-d8/public$ drush migrate-status
Group: Import from Drupal 7 (migrate_drupal_7) Status Total Imported Unprocessed Last imported
upgrade_action_settings Idle 0 0 0
upgrade_block_content_type Idle 1 0 1
upgrade_d7_action Idle 15 0 15
upgrade_d7_aggregator_settings Idle 0 0 0
upgrade_d7_dblog_settings Idle 1 0 1
upgrade_d7_file_private Idle 0 0 0
upgrade_d7_filter_settings Idle 1 0 1
upgrade_d7_global_theme_settings Idle 1 0 1
upgrade_d7_image_settings Idle 0 0 0
upgrade_d7_image_styles Idle 13 0 13
upgrade_d7_node_settings Idle 1 0 1
upgrade_d7_search_settings Idle 0 0 0
upgrade_d7_system_authorize Idle 0 0 0
upgrade_d7_system_cron Idle 0 0 0
upgrade_d7_system_date Idle 1 0 1
upgrade_d7_system_file Idle 1 0 1
upgrade_d7_system_mail Idle 0 0 0
upgrade_d7_system_performance Idle 1 0 1
upgrade_d7_theme_settings Idle 2 0 2
upgrade_d7_tracker_settings Idle 0 0 0
upgrade_d7_url_alias Idle 10890 0 10890
upgrade_d7_user_flood Idle 0 0 0
upgrade_d7_user_mail Idle 1 0 1
upgrade_file_settings Idle 0 0 0
upgrade_menu_settings Idle 0 0 0
upgrade_search_page Idle 0 0 0
upgrade_statistics_settings Idle 1 0 1
upgrade_system_image Idle 1 0 1
upgrade_system_image_gd Idle 1 0 1
upgrade_system_logging Idle 1 0 1
upgrade_system_maintenance Idle 0 0 0
upgrade_system_rss Idle 0 0 0
upgrade_system_site Idle 1 0 1
upgrade_taxonomy_settings Idle 0 0 0
upgrade_text_settings Idle 0 0 0
upgrade_update_settings Idle 1 0 1
upgrade_block_content_body_field Idle 1 0 1
upgrade_block_content_entity_display Idle 1 0 1
upgrade_block_content_entity_form_display Idle 1 0 1
upgrade_d7_filter_format Idle 5 0 5
upgrade_d7_custom_block Idle 10 0 10
upgrade_d7_user_role Idle 6 0 6
upgrade_d7_block Idle 345 0 345
upgrade_d7_field Idle 46 0 46
upgrade_d7_node_type Idle 10 0 10
upgrade_d7_comment_type Idle 10 0 10
upgrade_d7_taxonomy_vocabulary Idle 3 0 3
upgrade_d7_field_instance Idle 64 0 64
upgrade_d7_file Idle 50826 0 50826
upgrade_user_picture_field Idle 1 0 1
upgrade_user_picture_field_instance Idle 1 0 1
upgrade_user_picture_entity_display Idle 1 0 1
upgrade_user_picture_entity_form_display Idle 1 0 1
upgrade_d7_user Idle 107 0 107
upgrade_d7_comment_field Idle 10 0 10
upgrade_d7_comment_field_instance Idle 10 0 10
upgrade_d7_node_ad Idle 204 0 204
upgrade_d7_node_article Idle 9933 0 9933
upgrade_d7_node_background_ad Idle 1 0 1
upgrade_d7_node_establishment Idle 391 0 391
upgrade_d7_node_green_guide Idle 6 0 6
upgrade_d7_node_happy_hour Idle 176 0 176
upgrade_d7_node_homepage_config Idle 2 0 2
upgrade_d7_node_page Idle 7 0 7
upgrade_d7_node_sidebar_config Idle 1 0 1
upgrade_d7_node_webform Idle 1 0 1
upgrade_d7_comment Idle 0 0 0
upgrade_d7_comment_entity_display Idle 10 0 10
upgrade_d7_comment_entity_form_display Idle 10 0 10
upgrade_d7_comment_entity_form_display_subject Idle 10 0 10
upgrade_d7_view_modes Idle 5 0 5
upgrade_d7_field_formatter_settings Idle 97 0 97
upgrade_d7_field_instance_widget_settings Idle 64 0 64
upgrade_d7_google_analytics_settings Idle 1 0 1
upgrade_d7_google_analytics_user_settings Idle 107 0 107
upgrade_d7_menu Idle 7 0 7
upgrade_d7_menu_links Idle 88 0 88
upgrade_d7_node_revision_ad Idle 0 0 0
upgrade_d7_node_revision_article Idle 0 0 0
upgrade_d7_node_revision_background_ad Idle 0 0 0
upgrade_d7_node_revision_establishment Idle 0 0 0
upgrade_d7_node_revision_green_guide Idle 0 0 0
upgrade_d7_node_revision_happy_hour Idle 0 0 0
upgrade_d7_node_revision_homepage_config Idle 0 0 0
upgrade_d7_node_revision_page Idle 0 0 0
upgrade_d7_node_revision_sidebar_config Idle 0 0 0
upgrade_d7_node_revision_webform Idle 0 0 0
upgrade_d7_node_title_label Idle 10 0 10
upgrade_d7_shortcut_set Idle 1 0 1
upgrade_d7_shortcut Idle 4 0 4
upgrade_d7_shortcut_set_users Idle 0 0 0
upgrade_d7_taxonomy_term_category Idle 56 0 56
upgrade_d7_taxonomy_term_tags Idle 0 0 0
upgrade_d7_taxonomy_term_vicinity Idle 12 0 12
Group: Beer Imports (beer) Status Total Imported Unprocessed Last imported
beer_term Idle 3 0 3
beer_user Idle 4 0 4
beer_node Idle 3 0 3
beer_comment Idle 5 0 5
Group: Wine Imports (wine) Status Total Imported Unprocessed Last imported
weather_soap Idle 50 0 50
wine_role_json Idle 2 0 2
wine_role_xml Idle 2 0 2
wine_terms Idle 23 0 23
wine_variety_multi_xml Idle 4 0 4
Group: Default (default) Status Total Imported Unprocessed Last imported
block_content_type Idle 1 1 0
block_content_body_field Idle 1 1 0
block_content_entity_display Idle 1 1 0
block_content_entity_form_display Idle 1 1 0
user_picture_field Idle 1 0 1
d6_upload_field Idle 1 0 1
And this is what happens when I execute "drush migrate-import --all":
serverpilot#XXX.XXX.XXX.XXX:~/apps/therooster-d8/public$ drush migrate-import --all
Processed 1 item (1 created, 0 updated, 0 failed, 0 ignored) - done with 'upgrade_action_settings' [status]
Processed 1 item (1 created, 0 updated, 0 failed, 0 ignored) - done with 'upgrade_block_content_type' [status]
The "pathauto_node_update_action" plugin does not exist. [error]
(/srv/users/serverpilot/apps/therooster-d8/public/core/lib/Drupal/Component/Plugin/Discovery/DiscoveryTrait.php:52)
The "pathauto_taxonomy_term_update_action" plugin does not exist. [error]
(/srv/users/serverpilot/apps/therooster-d8/public/core/lib/Drupal/Component/Plugin/Discovery/DiscoveryTrait.php:52)
The "pathauto_user_update_action" plugin does not exist. [error]
(/srv/users/serverpilot/apps/therooster-d8/public/core/lib/Drupal/Component/Plugin/Discovery/DiscoveryTrait.php:52)
Processed 15 items (11 created, 0 updated, 3 failed, 1 ignored) - done with 'upgrade_d7_action' [status]
upgrade_d7_action Migration - 3 failed. [error]
Processed 1 item (1 created, 0 updated, 0 failed, 0 ignored) - done with 'upgrade_d7_aggregator_settings' [status]
Processed 1 item (1 created, 0 updated, 0 failed, 0 ignored) - done with 'upgrade_d7_dblog_settings' [status]
Processed 0 items (0 created, 0 updated, 0 failed, 0 ignored) - done with 'upgrade_d7_file_private' [status]
Processed 1 item (1 created, 0 updated, 0 failed, 0 ignored) - done with 'upgrade_d7_filter_settings' [status]
Processed 1 item (1 created, 0 updated, 0 failed, 0 ignored) - done with 'upgrade_d7_global_theme_settings' [status]
Processed 1 item (1 created, 0 updated, 0 failed, 0 ignored) - done with 'upgrade_d7_image_settings' [status]
Processed 13 items (13 created, 0 updated, 0 failed, 0 ignored) - done with 'upgrade_d7_image_styles' [status]
Processed 1 item (1 created, 0 updated, 0 failed, 0 ignored) - done with 'upgrade_d7_node_settings' [status]
Processed 1 item (0 created, 0 updated, 0 failed, 1 ignored) - done with 'upgrade_d7_search_settings' [status]
Processed 1 item (1 created, 0 updated, 0 failed, 0 ignored) - done with 'upgrade_d7_system_authorize' [status]
Processed 1 item (1 created, 0 updated, 0 failed, 0 ignored) - done with 'upgrade_d7_system_cron' [status]
Processed 1 item (1 created, 0 updated, 0 failed, 0 ignored) - done with 'upgrade_d7_system_date' [status]
Processed 1 item (0 created, 0 updated, 0 failed, 1 ignored) - done with 'upgrade_d7_system_file' [status]
Processed 1 item (0 created, 0 updated, 0 failed, 1 ignored) - done with 'upgrade_d7_system_mail' [status]
Processed 1 item (1 created, 0 updated, 0 failed, 0 ignored) - done with 'upgrade_d7_system_performance' [status]
Processed 0 items (0 created, 0 updated, 0 failed, 0 ignored) - done with 'upgrade_d7_theme_settings' [status]
Processed 1 item (1 created, 0 updated, 0 failed, 0 ignored) - done with 'upgrade_d7_tracker_settings' [status]
Processed 10890 items (10890 created, 0 updated, 0 failed, 0 ignored) - done with 'upgrade_d7_url_alias' [status]
Processed 1 item (1 created, 0 updated, 0 failed, 0 ignored) - done with 'upgrade_d7_user_flood' [status]
Processed 1 item (1 created, 0 updated, 0 failed, 0 ignored) - done with 'upgrade_d7_user_mail' [status]
Processed 1 item (1 created, 0 updated, 0 failed, 0 ignored) - done with 'upgrade_file_settings' [status]
Processed 1 item (1 created, 0 updated, 0 failed, 0 ignored) - done with 'upgrade_menu_settings' [status]
Processed 1 item (1 created, 0 updated, 0 failed, 0 ignored) - done with 'upgrade_search_page' [status]
Processed 1 item (1 created, 0 updated, 0 failed, 0 ignored) - done with 'upgrade_statistics_settings' [status]
Processed 1 item (1 created, 0 updated, 0 failed, 0 ignored) - done with 'upgrade_system_image' [status]
Processed 1 item (1 created, 0 updated, 0 failed, 0 ignored) - done with 'upgrade_system_image_gd' [status]
Processed 1 item (1 created, 0 updated, 0 failed, 0 ignored) - done with 'upgrade_system_logging' [status]
Processed 1 item (1 created, 0 updated, 0 failed, 0 ignored) - done with 'upgrade_system_maintenance' [status]
Processed 1 item (1 created, 0 updated, 0 failed, 0 ignored) - done with 'upgrade_system_rss' [status]
Processed 1 item (1 created, 0 updated, 0 failed, 0 ignored) - done with 'upgrade_system_site' [status]
Processed 1 item (1 created, 0 updated, 0 failed, 0 ignored) - done with 'upgrade_taxonomy_settings' [status]
Processed 1 item (1 created, 0 updated, 0 failed, 0 ignored) - done with 'upgrade_text_settings' [status]
Processed 1 item (1 created, 0 updated, 0 failed, 0 ignored) - done with 'upgrade_update_settings' [status]
Processed 1 item (1 created, 0 updated, 0 failed, 0 ignored) - done with 'upgrade_block_content_body_field' [status]
Processed 1 item (1 created, 0 updated, 0 failed, 0 ignored) - done with [status]
'upgrade_block_content_entity_display'
Processed 1 item (1 created, 0 updated, 0 failed, 0 ignored) - done with [status]
'upgrade_block_content_entity_form_display'
Missing filter plugin: filter_null. [error]
Processed 5 items (5 created, 0 updated, 0 failed, 0 ignored) - done with 'upgrade_d7_filter_format' [status]
Processed 10 items (10 created, 0 updated, 0 failed, 0 ignored) - done with 'upgrade_d7_custom_block' [status]
Processed 6 items (6 created, 0 updated, 0 failed, 0 ignored) - done with 'upgrade_d7_user_role' [status]
Processed 345 items (38 created, 0 updated, 0 failed, 307 ignored) - done with 'upgrade_d7_block' [status]
Attempt to create a field storage field_establishment with no type. [error]
(/srv/users/serverpilot/apps/therooster-d8/public/core/modules/field/src/Entity/FieldStorageConfig.php:252)
Attempt to create a field storage field_daily_menu with no type. [error]
(/srv/users/serverpilot/apps/therooster-d8/public/core/modules/field/src/Entity/FieldStorageConfig.php:252)
Attempt to create a field storage field_articles with no type. [error]
(/srv/users/serverpilot/apps/therooster-d8/public/core/modules/field/src/Entity/FieldStorageConfig.php:252)
Attempt to create a field storage field_articles with no type. [error]
(/srv/users/serverpilot/apps/therooster-d8/public/core/modules/field/src/Entity/FieldStorageConfig.php:252)
Attempt to create a field storage field_trending_articles with no type. [error]
(/srv/users/serverpilot/apps/therooster-d8/public/core/modules/field/src/Entity/FieldStorageConfig.php:252)
Processed 46 items (41 created, 0 updated, 5 failed, 0 ignored) - done with 'upgrade_d7_field' [status]
upgrade_d7_field Migration - 5 failed. [error]
Processed 10 items (10 created, 0 updated, 0 failed, 0 ignored) - done with 'upgrade_d7_node_type' [status]
Processed 10 items (10 created, 0 updated, 0 failed, 0 ignored) - done with 'upgrade_d7_comment_type' [status]
Processed 3 items (3 created, 0 updated, 0 failed, 0 ignored) - done with 'upgrade_d7_taxonomy_vocabulary' [status]
Attempt to create a field field_establishment that does not exist on entity type node. [error]
(/srv/users/serverpilot/apps/therooster-d8/public/core/modules/field/src/Entity/FieldConfig.php:293)
Attempt to create a field field_daily_menu that does not exist on entity type node. [error]
(/srv/users/serverpilot/apps/therooster-d8/public/core/modules/field/src/Entity/FieldConfig.php:293)
Attempt to create a field field_articles that does not exist on entity type node. [error]
(/srv/users/serverpilot/apps/therooster-d8/public/core/modules/field/src/Entity/FieldConfig.php:293)
Attempt to create a field field_articles that does not exist on entity type node. [error]
(/srv/users/serverpilot/apps/therooster-d8/public/core/modules/field/src/Entity/FieldConfig.php:293)
Attempt to create a field field_articles that does not exist on entity type taxonomy_term. [error]
(/srv/users/serverpilot/apps/therooster-d8/public/core/modules/field/src/Entity/FieldConfig.php:293)
Attempt to create a field field_trending_articles that does not exist on entity type node. [error]
(/srv/users/serverpilot/apps/therooster-d8/public/core/modules/field/src/Entity/FieldConfig.php:293)
Processed 64 items (58 created, 0 updated, 6 failed, 0 ignored) - done with 'upgrade_d7_field_instance' [status]
upgrade_d7_field_instance Migration - 6 failed. [error]
Processed 50826 items (50817 created, 0 updated, 9 failed, 0 ignored) - done with 'upgrade_d7_file' [status]
upgrade_d7_file Migration - 9 failed. [error]
Processed 1 item (1 created, 0 updated, 0 failed, 0 ignored) - done with 'upgrade_user_picture_field' [status]
Processed 1 item (1 created, 0 updated, 0 failed, 0 ignored) - done with [status]
'upgrade_user_picture_field_instance'
Processed 1 item (1 created, 0 updated, 0 failed, 0 ignored) - done with [status]
'upgrade_user_picture_entity_display'
Processed 1 item (1 created, 0 updated, 0 failed, 0 ignored) - done with [status]
'upgrade_user_picture_entity_form_display'
InvalidArgumentException: Passed variable is not an array or object in [error]
/srv/users/serverpilot/apps/therooster-d8/public/core/modules/migrate/src/Plugin/migrate/process/Flatten.php:52
Stack trace:
#0
/srv/users/serverpilot/apps/therooster-d8/public/core/modules/migrate/src/Plugin/migrate/process/Flatten.php(52):
ArrayIterator->__construct(NULL)
#1 /srv/users/serverpilot/apps/therooster-d8/public/core/modules/migrate/src/MigrateExecutable.php(380):
Drupal\migrate\Plugin\migrate\process\Flatten->transform(NULL,
Object(Drupal\migrate_tools\MigrateExecutable), Object(Drupal\migrate\Row), 'permissions')
#2
/srv/users/serverpilot/apps/therooster-d8/public/core/modules/migrate/src/Plugin/migrate/process/MigrationLookup.php(223):
Drupal\migrate\MigrateExecutable->processRow(Object(Drupal\migrate\Row), Array)
#3 /srv/users/serverpilot/apps/therooster-d8/public/core/modules/migrate/src/MigrateExecutable.php(366):
Drupal\migrate\Plugin\migrate\process\MigrationLookup->transform(Array,
Object(Drupal\migrate_tools\MigrateExecutable), Object(Drupal\migrate\Row), 'roles')
#4 /srv/users/serverpilot/apps/therooster-d8/public/core/modules/migrate/src/MigrateExecutable.php(203):
Drupal\migrate\MigrateExecutable->processRow(Object(Drupal\migrate\Row))
#5 /srv/users/serverpilot/apps/therooster-d8/vendor/drush/drush/includes/drush.inc(721):
Drupal\migrate\MigrateExecutable->import()
#6 /srv/users/serverpilot/apps/therooster-d8/vendor/drush/drush/includes/drush.inc(712):
drush_call_user_func_array(Array, Array)
#7
/srv/users/serverpilot/apps/therooster-d8/public/modules/contrib/migrate_tools/migrate_tools.drush.inc(303):
drush_op(Array)
#8 [internal function]: _drush_migrate_tools_execute_migration(Object(Drupal\user\Plugin\migrate\User),
'upgrade_d7_user', Array)
#9
/srv/users/serverpilot/apps/therooster-d8/public/modules/contrib/migrate_tools/migrate_tools.drush.inc(267):
array_walk(Array, '_drush_migrate_...', Array)
#10 /srv/users/serverpilot/apps/therooster-d8/vendor/drush/drush/includes/command.inc(422):
drush_migrate_tools_migrate_import()
#11 /srv/users/serverpilot/apps/therooster-d8/vendor/drush/drush/includes/command.inc(231):
_drush_invoke_hooks(Array, Array)
#12 /srv/users/serverpilot/apps/therooster-d8/vendor/drush/drush/includes/command.inc(199): drush_command()
#13 /srv/users/serverpilot/apps/therooster-d8/vendor/drush/drush/lib/Drush/Boot/BaseBoot.php(67):
drush_dispatch(Array)
#14 /srv/users/serverpilot/apps/therooster-d8/vendor/drush/drush/includes/preflight.inc(66):
Drush\Boot\BaseBoot->bootstrap_and_dispatch()
#15 /srv/users/serverpilot/apps/therooster-d8/vendor/drush/drush/drush.php(12): drush_main()
#16 {main}
Basically, the nearly 10K articles don't migrate over. That's primarily what I need. I've tried matching and enabling all the modules that I could match from Drupal 7 to 8...but perhaps I'm missing something that a trained eye is able to see from the errors above? Your help is much appreciated. Thank you
I don't enjoy being the bearer of bad news, but in my experience there are no one-click, automatic solutions that will upgrade and migrate from D7 to D8.
You'll end up with a far more stable product—and most likely save time and santity—if you rebuild the site in D8 and carry out a migration using the Migrate API.
I suggest doing it gradually and in pieces not all as you indicated by the drush command.
By using drush, first do the configuration using the switch [drush..... --configure-only]
This will create the needed configuration at the target site/dB including Articles.
You can also do everything except the problem portions by using the manifest file switch (see docs in that link above about the manifest file with drush).
Update:
Install the features module which integrates with drush and export /import the Articles content type. Then attempt the content migration using backup and migrate.
D7: https://www.drupal.org/docs/7/contrib/features/features-drush
D8: https://www.drupal.org/docs/8/modules/features
Fitting a linear-regression model using least squares on my training dataset works fine.
library(Matrix)
library(tm)
library(glmnet)
library(e1071)
library(SparseM)
library(ggplot2)
trainingData <- read.csv("train.csv", stringsAsFactors=FALSE,sep=",", header = FALSE)
testingData <- read.csv("test.csv",sep=",", stringsAsFactors=FALSE, header = FALSE)
lm.fit = lm(as.factor(V42)~ ., data = trainingData)
linearMPrediction = predict(lm.fit,newdata = testingData, se.fit = TRUE)
mean((linearMPrediction$fit - testingData[,20:41])^2)
linearMPrediction$residual.scale
However, when i try to fit a ridge-regression model on my training dataset as,
x = model.matrix(as.factor(V42)~., data = trainingData)
y = as.factor(trainingData$V42)
ridge = glmnet(x, y, family = "multinomial", alpha = 1, lambda.min.ratio = 1e-2)
I am having the following error for both multinomial and binomial distributions.
Error in lognet(x, is.sparse, ix, jx, y, weights, offset, alpha, nobs, :
one multinomial or binomial class has 1 or 0 observations; not allowed
Am I missing something? Any comment would be greatly appreciated. Here is a portion of how my data looks like by the way.
> trainingData$V42[1:50]
[1] "normal" "normal" "neptune" "normal" "normal" "neptune" "neptune" "neptune" "neptune" "neptune" "neptune"
[12] "neptune" "normal" "warezclient" "neptune" "neptune" "normal" "ipsweep" "normal" "normal" "neptune" "neptune"
[23] "normal" "normal" "neptune" "normal" "neptune" "normal" "normal" "normal" "ipsweep" "neptune" "normal"
[34] "portsweep" "normal" "normal" "normal" "neptune" "normal" "neptune" "neptune" "neptune" "normal" "normal"
[45] "normal" "neptune" "teardrop" "normal" "warezclient" "neptune"
> x
(Intercept) V1 V2tcp V2udp V3bgp V3courier V3csnet_ns V3ctf V3daytime V3discard V3domain V3domain_u V3echo V3eco_i V3ecr_i V3efs V3exec V3finger V3ftp
1 1 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
2 1 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
3 1 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
4 1 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
5 1 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
6 1 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
7 1 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
8 1 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
9 1 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
10 1 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
> y[1:50]
[1] normal normal neptune normal normal neptune neptune neptune neptune neptune neptune neptune normal
[14] warezclient neptune neptune normal ipsweep normal normal neptune neptune normal normal neptune normal
[27] neptune normal normal normal ipsweep neptune normal portsweep normal normal normal neptune normal
[40] neptune neptune neptune normal normal normal neptune teardrop normal warezclient neptune
22 Levels: back buffer_overflow ftp_write guess_passwd imap ipsweep land loadmodule multihop neptune nmap normal phf pod portsweep rootkit satan smurf spy ... warezmaster
> table(y)
y
back buffer_overflow ftp_write guess_passwd imap ipsweep land loadmodule multihop neptune
196 6 1 10 5 710 1 1 2 8282
nmap normal phf pod portsweep rootkit satan smurf spy teardrop
301 13449 2 38 587 4 691 529 1 188
warezclient warezmaster
181 7
You have single observations for some of the classes (like ftp_write with only 1 observation), which is not allowed (and clearly stated in the error).
I am studying how HTK Tools works with handwriting recognition. Following the ICFHR–2010 TUTORIAL I run examples for "Spanish-Numbers" corpus and received the resulting HMMs (files stored in folder hmm and listed in HMMsList), and res32.mlf with results of recognition received with HVite. Also I have master label file SamplesRef.mlf.
And now I want to see recognition results statistics, i.e. studying HResults tool.
When I run HResults as
HResults -I SamplesRef.mlf HMMsList res32.mlf
I see
====================== HTK Results Analysis =======================
Date: Tue Mar 31 15:21:11 2015
Ref : SamplesRef.mlf
Rec : res32.mlf
------------------------ Overall Results --------------------------
SENT: %Correct=0.00 [H=0, S=2, N=2]
WORD: %Corr=77.78, Acc=77.78 [H=7, D=0, S=2, I=0, N=9]
===================================================================
But if I add option -p in order to have confusion matrix I see the following error message:
~/icfhr$ HResults -p -I SamplesRef.mlf HMMsList res32.mlf
ERROR [+3331] Index: Label millones not in list[0 of 19]
FATAL ERROR - Terminating program HResults
I understand that message means that there is no HMM with name "millones" and I found that in my res32.mlf samples looks like:
"’*’/210341.rec"
mil
seiscientos
cincuenta
y
siete
millones
.
If I change res32.mlf with text editor to res33.mlf with content like:
"’*’/210341.rec"
m
i
l
s
e
i
s
c
i
... and so on.
And use samples.mlf (instead of SamplesRef.mlf) which inside looks like:
"*/210341.lab"
m
i
l
#
q
u
i
n
i
e
n
t
o
s
#
c
... and so on.
I have the desired result:
~/icfhr$ HResults -p -I samples.mlf HMMsList res33.mlf
====================== HTK Results Analysis =======================
Date: Tue Mar 31 15:35:42 2015
Ref : samples.mlf
Rec : res33.mlf
------------------------ Overall Results --------------------------
SENT: %Correct=0.00 [H=0, S=2, N=2]
WORD: %Corr=79.63, Acc=77.78 [H=43, D=5, S=6, I=1, N=54]
------------------------ Confusion Matrix -------------------------
a c d e i l m n o s t u v y Del [ %c / %e]
# 0 0 0 0 0 1 1 0 0 0 0 0 0 0 5 [ 0.0/3.7]
a 2 0 0 0 0 0 0 0 0 0 0 0 0 0 0
c 0 2 0 0 0 0 0 0 0 0 0 0 0 0 0
d 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0
e 0 0 0 6 0 0 0 0 0 0 0 0 0 0 0
i 0 0 0 0 6 0 0 0 0 0 0 0 0 0 0
l 0 0 0 0 0 3 0 0 0 0 0 0 0 0 0
m 0 0 0 0 0 0 2 0 0 0 0 0 0 0 0
n 0 1 0 0 0 0 0 6 0 0 0 0 0 0 0 [85.7/1.9]
o 0 0 0 0 0 0 0 0 4 0 0 0 0 0 0
q 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 [ 0.0/1.9]
s 0 0 0 0 0 0 0 0 0 4 0 0 0 0 0
t 0 0 0 0 0 0 0 0 0 0 4 0 0 0 0
u 0 0 0 1 0 0 0 0 0 0 0 1 0 0 0 [50.0/1.9]
v 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0
y 0 0 0 0 1 0 0 0 0 0 0 0 0 1 0 [50.0/1.9]
Ins 0 0 0 0 0 0 0 0 0 1 0 0 0 0
===================================================================
So, the main question is:
What is the simplest way (without text editor) to make mlf-files adapted for making confusion matrix?
(I suppose I miss some option of some HTK tool… but which tool and which option?)
Any useful ideas would be highly appreciated.
In order to use the -p option, you need to provide the labels list of the classes not your HMMs, (i.e. if you're trying to recognize the words Yes, No, Never) then your "HMMsList" file should be written as:
Yes
No
Never
Regardless of the HMMs that actually constitutes the words.
Your "HMMsList" file should be "LabelsList"
Performing an snmpwalk on OID .1.3.6.1.4.1.2021.11 produces the following CpuRaw values:
UCD-SNMP-MIB::ssCpuRawUser.0 = Counter32: 3191634181
UCD-SNMP-MIB::ssCpuRawNice.0 = Counter32: 2586628
UCD-SNMP-MIB::ssCpuRawSystem.0 = Counter32: 480833488
UCD-SNMP-MIB::ssCpuRawIdle.0 = Counter32: 3578238833
UCD-SNMP-MIB::ssCpuRawWait.0 = Counter32: 461331879
UCD-SNMP-MIB::ssCpuRawKernel.0 = Counter32: 422462005
UCD-SNMP-MIB::ssCpuRawInterrupt.0 = Counter32: 7890770
UCD-SNMP-MIB::ssCpuRawSoftIRQ.0 = Counter32: 50480713
I note that all values, with the exception of idle + kernel, have matching values (close enough) in /proc/stat:
cpu 3191634876 2586629 422462086 7873206561 461331924 7890771 50480723 0
cpu0 1551975573 184783 190766514 1008267162 200070032 7243827 44073977 0
cpu1 610948559 324197 73381486 2228315579 87505437 51905 2054732 0
cpu2 494534866 1024716 75891701 2342123809 86260984 289810 2089023 0
cpu3 534175876 1052931 82422383 2294500009 87495469 305228 2262989 0
intr 8208380331 4267093007 3 0 4 4 0 0 0 3 0 0 0 4 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 3538 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 328178999 0 0 0 0 0 0 0 997463093 0 0 0 0 0 0 0 2374098089 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 241543587 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
ctxt 163836165157
btime 1351100022
processes 224876777
procs_running 5
procs_blocked 0
I'd like to know what the difference is between ssCpuRawSystemand ssCpuRawKernel and how this relates to the contents of /proc/stat
ssCpuRawSystem
The number of 'ticks' (typically 1/100s) spent
processing system-level code.
On a multi-processor system, the 'ssCpuRaw*'
counters are cumulative over all CPUs, so their
sum will typically be N*100 (for N processors).
This object may sometimes be implemented as the
combination of the 'ssCpuRawWait(54)' and
'ssCpuRawKernel(55)' counters, so care must be
taken when summing the overall raw counters.
ssCpuRawKernel
The number of 'ticks' (typically 1/100s) spent
processing kernel-level code.
This object will not be implemented on hosts where
the underlying operating system does not measure
this particular CPU metric. This time may also be
included within the 'ssCpuRawSystem(52)' counter.
On a multi-processor system, the 'ssCpuRaw*'
counters are cumulative over all CPUs, so their
sum will typically be N*100 (for N processors).