I have a migration written that executes just fine on my test database - did dozens of tests runs without issue. I run it on a clone of my prod database and all of a sudden I'm having all sorts of problems. I'm beginning to think its a database config or permissions issue, but I'm logged into this clone as root, so I'm not even sure where to start looking...
If I copy the mysql statement from the error (...and fix the missing data) the statement executes without issue.
ALTER TABLE `retail_items-tmp_09-10-2020` CHANGE original_inventory initial_inventory INT DEFAULT NULL;
The offending line:
Schema::table('retail_items-tmp_09-10-2020', function($table) {
$table->renameColumn('original_inventory','initial_inventory');
});
The Error:
[PDOException (42000)]
SQLSTATE[42000]: Syntax error or access violation: 1064 You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near '-tmp_09-10-2020 CHANGE original_inventory initial_inventory INT DEFAULT NULL' at line 1
The migration:
public function up()
{
/* 1. Backup Existing Tables */
DB::statement('CREATE TABLE `retail_items-tmp_09-10-2020` LIKE `retail_items`; ');
DB::statement('CREATE TABLE `ARCH__retail_items_09-10-2020` LIKE `retail_items`; ');
DB::statement('INSERT INTO `retail_items-tmp_09-10-2020` SELECT * FROM `retail_items`; ');
DB::statement('INSERT INTO `ARCH__retail_items_09-10-2020` SELECT * FROM `retail_items`;');
/* 2. Update structure */
Schema::table('retail_items-tmp_09-10-2020', function($table) {
$table->renameColumn('original_inventory','initial_inventory');
});
Schema::table('retail_items-tmp_09-10-2020', function($table) {
$table->integer('event_ID')->length(11)->unsigned()->nullable();
$table->foreign('event_ID')->references('event_ID')->on('events');
});
/* 3. Update structure that would have been destructive prior to step 3 */
// When I had this piece of code included, it resulted in the same error "Syntax error or access violation..." this worked in testing, but throws errors on Prod, changed to DB:statement below with success.
// Schema::table('retail_items-tmp_09-10-2020', function($table) {
// $table->smallInteger('flag')->unsigned()->nullable(false)->default(0)->change();
// });
$query = "ALTER TABLE `retail_items-tmp_09-10-2020` CHANGE `flag` `flag` SMALLINT(11) DEFAULT 0 NOT NULL; ";
DB::statement($query);
}
I've been stuck on this for a bit, so any theories, places to look, etc would be appreciated.
(I have another migration that renames the temp table at the end of this. I have a few migrations and data operations that all together take ~ 10+ min to execute with the piece of code that I'm launching this with, so the temp tables are necessary to prevent downtime when launching in production)
I still ended up having more troubleshooting/ other roadblocks on this issue, so this only removed one...
Even though they were properly quoted, I read a number of S.O. posts on the subject and as Webtect suggests I did end up simply removing all the dashes and things worked much more consistently and reliably. I'm still feel like it's a mystery as to why this happened because I've used hyphens in dates like this before - properly quoted - without issue. But this one sure threw me for a loop. I think I will entirely stop the practice of using hyphens in dated-table-names going forward. They seem to cause more headaches than the improved readability warrants.
Related
I'm trying to set up some unit tests in a legacy Laravel project. This project has a lot of migration scripts creating and updating the database. This migrations work fine on a MySQL database, however I don't want to persist a test database so I'm using the standard .env.testing configuration of a Sqlite database:
One of the migration files contains the following:
public function up()
{
Schema::table('product_variants', function (Blueprint $table) {
$table->dropColumn(['sku']);
});
}
While this migration works in MySQL, when I try to run the migration in test mode (i.e. in Sqlite) I get the following message:
Reading the Laravel Docs, it's not supported to remove multiple columns from a Sqlite connection, so I converted the array param to a string (['sku'] into 'sku') but that did not change anything.
How do I get the migration to work in Sqlite?
Edit
Following the comment by BkiD (the column is indexed), I have attempted to identify what the index name of the column is using the following code:
$sm = Schema::getConnection()->getDoctrineSchemaManager();
$indexesFound = $sm->listTableIndexes('product_variants');
var_dump($indexesFound);die;
I was expecting an array of generated names, specifically I was expecting to see product_variants_sku_unique. What I actually got was an empty array: array(0){}
Edit2
Tried manually dropping the index anyway:
Schema::table('product_variants', function (Blueprint $table) {
$table->dropUnique('product_variants_sku_unique');
});
Any other name throws a message that the index does not exist. Since this is not throwing an error, I assume the index exists, even if the previous code does not return it. So. I'm "dropping" the index then trying to drop the sku column, but still getting the same message.
Have you tried something like this?
Schema::table('products', function (Blueprint $table) {
$table->dropColumn('name', 'description', 'price');
});
or this, seem's like not the best practice but hopefully works:
collect([
['old_a', 'new_a'],
['old_b', 'new_b'],
['old_c', 'new_c']])->map(function ($old_new) {
Schema::table('thetable', function (Blueprint $table) use ($old_new) {
$table->renameColumn($old_new[0], $old_new[1]);
});
});
So from what I understand, you're using Laravel 9.x and running sqlite 3.37.2 and you definitely have the Doctrine/dbal package installed.
My first question is - is this the complete migration? If not, please post the entire thing here.
If yes - please check your other migrations to look for references to this particular column.
If you look at the error, the error is coming from Index::_addColumn() , which means you probably have - an index declared for usage in mysql.
The doctrine/dbal package cannot rename/handle/drop SQLite columns that have an index
So your solution could be to either -
Drop the index in the laravel migration (provided you don't need it for MySQL as well - probably the case since you're dropping the column)
OR
Use a condition like this (in the migration file where the index was first created - incase you want to be able to rename or continue using the index in mysql) -
if (config('DB_CONNECTION') === 'mysql') {
$table->index('sku');
}
PS: The doctrine/dbal github repo has issues like these that have since been closed since the package has fixed this in newer versions - I'm guessing you probably aren't on the latest by default.
There is a warning in the laravel doc's that:
Dropping or modifying multiple columns within a single migration while using a SQLite database is not supported.
So, try to split your migrations.
public function up()
{
Schema::table('product_variants', function (Blueprint $table) {
$table->dropColumn('sku');
});
Schema::table('product_variants', function (Blueprint $table) {
$table->dropColumn('vku');
});
}
I know that it is a bad idea, but this can be a possible solution for your current issue. Hope, it works for you.
While trying to understand why my view is not displaying, I noticed the following error in the log:
I do not think it is possible to delete the URL alias from Taxonomy terms. At least I cannot find how to do this.
I have, however gone through ALL of my taxonomy terms and removed the value for this field.
I have also done the following with Pathauto:
Also, I have checked the report located at admin/reports/fields and can confirm that there are no entities that use a field called URL alias.
I have gone through each content item and ensured that they have the following setting (anyone know how to do this in bulk?). But still the error remains.
Anyone know then how I can fix this strange error?
Im not entirely sure what this command does, but it fixed the error:
drush updb --entity-updates
Since https://www.drupal.org/node/2554097, the magic in Drupal core that took care of updating entity definitions is gone. drush updb --entiy-updates is an alternative to this but it is not a silver bullet. Instead, it is safer to write database updates.
Taking the screenshot at the top as an example, here is a database update that would delete those two field definitions:
/**
* Fix taxonomy and node field definitions.
*
*/
function mymodule_update_8101() {
$manager = \Drupal::entityDefinitionUpdateManager();
if ($field = $manager->getFieldStorageDefinition('alias', 'node')) {
$manager->uninstallFieldStorageDefinition($field);
}
if ($field = $manager->getFieldStorageDefinition('alias', 'term')) {
$manager->uninstallFieldStorageDefinition($field);
}
}
Have a look at the rest of the available methods at https://www.drupal.org/node/2554097 in order to write database updates for each scenario.
use the entity_update module or the devel_entity_updates module
Background: I downloaded a *.sql backup of my WordPress site's database, and replaced all instances of the old database table prefix with a new one (e.g. from the default wp_ to something like asdfghjkl_).
I've just learnt that WordPress uses serialized PHP strings in the database, and what I did will have messed with the integrity of the serialized string lengths.
The thing is, I deleted the backup file just before I learnt about this (as my website was still functioning fine), and installed a number of plugins since. So, there's no way I can revert back, and I therefore would like to know two things:
How can I fix this, if at all possible?
What kind of problems could this cause?
(This article states that, a WordPress blog for instance, could lose its settings and widgets. But this doesn't seem to have happened to me as all the settings for my blog are still intact. But I have no clue as to what could be broken on the inside, or what issues it'd pose in the future. Hence this question.)
Visit this page: http://unserialize.onlinephpfunctions.com/
On that page you should see this sample serialized string: a:1:{s:4:"Test";s:17:"unserialize here!";}. Take a piece of it-- s:4:"Test";. That means "string", 4 characters, then the actual string. I am pretty sure that what you did caused the numeric character count to be out of sync with the string. Play with the tool on the site mentioned above and you will see that you get an error if you change "Test" to "Tes", for example.
What you need to do is get those character counts to match your new string. If you haven't corrupted any of the other encoding-- removed a colon or something-- that should fix the problem.
I came to this same problem after trying to change the domain from localhost to the real URL. After some searching I found the answer in Wordpress documentation:
https://codex.wordpress.org/Moving_WordPress
I will quote what is written there:
To avoid that serialization issue, you have three options:
Use the Better Search Replace or Velvet Blues Update URLs plugins if you can > access your Dashboard.
Use WP-CLI's search-replace if your hosting provider (or you) have installed WP-CLI.
Run a search and replace query manually on your database. Note: Only perform a search and replace on the wp_posts table.
I ended up using WP-CLI which is able to replace things in the database without breaking serialization: http://wp-cli.org/commands/search-replace/
I know this is an old question, but better late than never, I suppose. I ran into this problem recently, after inheriting a database that had had a find/replace executed on serialized data. After many hours of researching, I discovered that this was because the string counts were off. Unfortunately, there was so much data with lots of escaping and newlines and I didn't know how to count in some cases and I had so much data that I needed something automated.
Along the way, I stumbled across this question and Benubird's post helped put me on the right path. His example code did not work in production use on complex data, containing numerous special characters and HTML, with very deep levels of nesting, and it did not properly handle certain escaped characters and encoding. So I modified it a bit and spent countless hours working through additional bugs to get my version to "fix" the serialized data.
// do some DB query here
while($res = db_fetch($qry)){
$str = $res->data;
$sCount=1; // don't try to count manually, which can be inaccurate; let serialize do its thing
$newstring = unserialize($str);
if(!$newstring) {
preg_match_all('/s:([0-9]+):"(.*?)"(?=;)/su',$str,$m);
# preg_match_all("/s:([0-9]+):(\"[^\"\\\\]*(?:\\\\.[^\"\\\\]*)*\")(?=;)/u",$str,$m); // alternate: almost works but leave quotes in $m[2] output
# print_r($m); exit;
foreach($m[1] as $k => $len) {
/*** Possibly specific to my case: Spyropress Builder in WordPress ***/
$m_clean = str_replace('\"','"',$m[2][$k]); // convert escaped double quotes so that HTML will render properly
// if newline is present, it will output directly in the HTML
// nl2br won't work here (must find literally; not with double quotes!)
$m_clean = str_replace('\n', '<br />', $m_clean);
$m_clean = nl2br($m_clean); // but we DO need to convert actual newlines also
/*********************************************************************/
if($sCount){
$m_new = $m[0][$k].';'; // we must account for the missing semi-colon not captured in regex!
// NOTE: If we don't flush the buffers, things like <img src="http://whatever" can be replaced with <img src="//whatever" and break the serialize count!!!
ob_end_flush(); // not sure why this is necessary but cost me 5 hours!!
$m_ser = serialize($m_clean);
if($m_new != $m_ser) {
print "Replacing: $m_new\n";
print "With: $m_ser\n";
$str = str_replace($m_new, $m_ser, $str);
}
}
else{
$m_len = (strlen($m[2][$k]) - substr_count($m[2][$k],'\n'));
if($len != $m_len) {
$newstr='s:'.$m_len.':"'.$m[2][$k].'"';
echo "Replacing: {$m[0][$k]}\n";
echo "With: $newstr\n\n";
$str = str_replace($m_new, $newstr, $str);
}
}
}
print_r($str); // this is your FIXED serialized data!! Yay!
}
}
A little geeky explanation on my changes:
I found that trying to count with Benubird's code as a base was too inaccurate for large datasets, so I ended up just using serialize to be sure the count was accurate.
I avoided the try/catch because, in my case, the try would succeed but just returned an empty string. So, I check for empty data instead.
I tried numerous regex's but only a mod on Benubird's would accurately handle all cases. Specifically, I had to modify the part that checked for the ";" because it would match on CSS like "width:100%; height:25px;" and broke the output. So, I used a positive lookahead to only match when the ";" was outside of the set of double quotes.
My case had lots of newlines, HTML, and escaped double quotes, so I had to add a block to clean that up.
There were a couple of weird situations where data would be replaced incorrectly by the regex and then the serialize would count it incorrectly as well. I found NOTHING on any sites to help with this and finally thought it might be related to caching or something like that and tried flushing the output buffer (ob_end_flush()), which worked, thank goodness!
Hope this helps someone... Took me almost 20 hours including the research and dealing with weird issues! :)
This script (https://interconnectit.com/products/search-and-replace-for-wordpress-databases/) can help to update an sql database with proper URLs everywhere, without encountering serialized data issues, because it will update the "characters count" that could throw your URLs out of sync whenever serialized data occurs.
The steps would be:
if you already have imported a messed up database (widgets not
working, theme options not there, etc), just drop that database
using PhpMyAdmin. That is, remove everything on it. Then export and
have at hand an un-edited dump of the old database.
Now you have to import the (un-edited) old database into the
newly created one. You can do this via an import, or copying over
the db from PhpMyAdmin. Notice that so far, we haven't done any
search and replace yet; we just have an old database content and
structure into a new database with its own user and password. Your site will be probably unaccessible at this point.
Make sure you have your WordPress files freshly uploaded to the
proper folder on the server, and edit your wp-config.php to make it
connect with the new database.
Upload the script into a "secret" folder - just for security
reasons - at the same level than wp-admin, wp-content, and wp-includes. Do not forget to remove it all once the search and
replace have taken place, because you risk to offer your DB details
open to the whole internet.
Now point your browser to the secret folder, and use the script's fine
interface. It is very self-explanatory. Once used, we proceed to
completely remove it from the server.
This should have your database properly updated, without any serialized data issues around: the new URL will be set everywhere, and serialized data characters counts will be accordingly updated.
Widgets will be passed over, and theme settings as well - two of the typical places that use serialized data in WordPress.
Done and tested solution!
If the error is due to the length of the strings being incorrect (something I have seen frequently), then you should be able to adapt this script to fix it:
foreach($strings as $key => $str)
{
try {
unserialize($str);
} catch(exception $e) {
preg_match_all('#s:([0-9]+):"([^;]+)"#',$str,$m);
foreach($m[1] as $k => $len) {
if($len != strlen($m[2][$k])) {
$newstr='s:'.strlen($m[2][$k]).':"'.$m[2][$k].'"';
echo "len mismatch: {$m[0][$k]}\n";
echo "should be: $newstr\n\n";
$strings[$key] = str_replace($m[0][$k], $newstr, $str);
}
}
}
}
I personally don't like working in PHP, or placing my DB credentials in an public file. I created a ruby script to fix serializations that you can run locally:
https://github.com/wsizoo/wordpress-fix-serialization
Context Edit:
I approached fixing serialization by first identifying serialization via regex, and then recalculating the byte size of the contained data string.
$content_to_fix.gsub!(/s:([0-9]+):\"((.|\n)*?)\";/) {"s:#{$2.bytesize}:\"#{$2}\";"}
I then update the specified data via an escaped sql update query.
escaped_fix_content = client.escape($fixed_content)
query = client.query("UPDATE #{$table} SET #{$column} = '#{escaped_fix_content}' WHERE #{$column_identifier} LIKE '#{$column_identifier_value}'")
This error suddenly occurred when Pressflow was added to our Drupal installation. It is coming from a custom module that, prior to Pressflow seemed to work fine. After the addition of Pressflow, running Drush causes this error to display on screen.
The function that is identified as the source of the error looks like this:
function user_search_location_get(&$user) {
if (count($user->user_location_pref)) { // This line is causing the error.
return $user->user_location_pref;
}
// …
}
The error message is the following:
WD php: Notice: Undefined property: stdClass::$user_location_pref in user_search_location_get()
Short answer, in your custom module, you should check if that property exists before you count it. That, or make sure the $user object has that property before you use it.
if (isset($user->user_location_pref) && count($user->user_locaion_pref) > 0) {
return $user->user_locaion_pref;
}
While it is a little more work, when you start developing with notices turned on, you will find errors in your code that otherwise would have not appeared till later, and would have been more difficult to track down.
In your previous environment or install, the PHP error reporting was probably set to not show notices. While I recommend keeping notices on and making your code work with them, you can turn them off through the Drupal 7 UI. Configuration -> Development -> Logging and Errors. Then set the value to 'Errors and Warnings' ... Otherwise, you can set your error reporting level in your php.ini to report all except notices.
Note that Drupal 6 did not force notice reporting on, while Drupal 7 does. That prompts this type of question a lot.
If this is your only notice issue though, it makes more sense to just correct your custom module.
Stripped down, what I trying to do it the following
set selectQuery "SELECT col1, col2 FROM tableName"
db1 eval $selectQuery {
set insertQuery "INSERT INTO tableName VALUES($col1, $col2)"
db2 eval $insertQuery {
# Do trivial stuff to show progress
}
}
which basically copies the content of db1.tableName to db2.tableName.
The problem is that in my case, most of the content of db1.tableName already exists in db2.tableName. So basically, I just want to copy what doesn't exist yet so I thought I would just insert everything and let the inserts fail when the data's already there. This doesn't work because the whole script stops as soon as one insert fails. I tried using catch to ignore the failure and allow the script to continue, but I haven't been successful. Any ideas?
Also, there may be a better way to copy the contents of db1.tableName to db2.tableName without copying what's already there... Any help will be appreciated!
Shawn
P.S If you have ideas for a better title, that would be useful as well..
Sqlite has a INSERT OR REPLACE INTO construct which may be useful in this case.
http://www.sqlite.org/lang_insert.html
While I'm not familiar with your database api commands, so I can't comment on how you're looping throught the query results, but I can suggest a few things.
First, try catch again. I always find the usage a bit strange, but eventually you get used to it. Here's an example:
if { [catch { db2 eval $insertQuery} errmsg] } {
#There was an error - it is stored in $errmsg"
} else {
#success! Congratulations.
}
In your case, I imagine you just want to ignore any insert errors, so you can simply do:
set selectQuery "SELECT col1, col2 FROM tableName"
db1 eval $selectQuery {
set insertQuery "INSERT INTO tableName VALUES($col1, $col2)"
if { ![catch { db2 eval $insertQuery} errmsg] } {
# Do trivial stuff to show progress
}
}
You could also try doing the query first and putting the query results in a TCL list (or list of lists) - then do a foreach on the list, doing the inserts. Sometimes databases can be funny about mutating data, although it doesn't look at all like that is happening in your case.