I have recently enabled outOfOrder in my Flyway config to solve some merge conflicts.
The problem is when I run migrate all my scripts get executed and in status it shows OutOfOrder.
I want to know does OutOfOrder mean success state ?
Yes, that is a successfully applied migration.
See MigrationState#OUT_OF_ORDER for a bit more detail.
/**
* <p>This migration succeeded.</p>
* <p>
* This migration succeeded, but it was applied out of order.
* Rerunning the entire migration history might produce different results!
* </p>
*/
OUT_OF_ORDER("OutOrdr", true, true, false)
/**
* Creates a new MigrationState.
*
* #param displayName The name suitable for display to the end-user.
* #param resolved Flag indicating if this migration is available on the classpath or not.
* #param applied Flag indicating if this migration has been applied or not.
* #param failed Flag indicating if this migration has failed when it was applied or not.
*/
MigrationState(String displayName, boolean resolved, boolean applied, boolean failed) {
Related
Looking at the Drupal queue documentation it is not clear what the difference is between a DatabaseQueue and a BatchQueue? Are both FIFO? Will they expire? Serial meaning not concurrent processing?
From the source of Batch.php:
/**
* Defines a batch queue handler used by the Batch API.
*
* This implementation:
* - Ensures FIFO ordering.
* - Allows an item to be repeatedly claimed until it is actually deleted (no
* notion of lease time or 'expire' date), to allow multipass operations.
*
* Stale items from failed batches are cleaned from the {queue} table on cron
* using the 'created' date.
*
* #ingroup queue
*/
class Batch extends DatabaseQueue {
Batch extends DatabaseQueue. In my experience, batch is most often used with a form, as it will automatically process the batch after it's built.
Yes, FIFO.
No, items don't expire.
And yes, serial processing.
I have an Application that has a relationship to ApplicationFile:
/**
* #ORM\OneToMany(
* targetEntity="AppBundle\Entity\ApplicationFile",
* mappedBy="application",
* cascade={"remove"},
* orphanRemoval=true
* )
*/
private $files;
A file entity has a field that stores binary data, and can be up to 2MB in size. When iterating over a large list of applications and their files, PHP memory usage grows. I want to keep it down.
I've tried this:
$applications = $this->em->getRepository('AppBundle:Application')->findAll();
foreach ($applications as $app) {
...
foreach ($app->getFiles() as $file) {
...
$this->em->detach($file);
}
$this->em->detach($app);
}
Detaching the object should tell the entity manager to stop caring about this object and de-referencing it, but it surprisingly has no effect on the amount of memory usage - it keeps increasing.
Instead, I have to manually load the application files (instead of retrieving them through the association method), and the memory usage does not increase. This works:
$applications = $this->em->getRepository('AppBundle:Application')->findAll();
foreach ($applications as $app) {
...
$appFiles = $this
->em
->getRepository('AppBundle:ApplicationFile')
->findBy(array('application' => $application));
foreach ($appFiles as $file) {
...
$this->em->detach($file);
}
$this->em->detach($app);
}
I used xdebug_debug_zval to track references to the $file object. In the first example, there's an extra reference somewhere, which explains why memory is ballooning - PHP is not able to garbage collect it!
Does anyone know why this is? Where is this extra reference and how do I remove it?
EDIT: Explicitly calling unset($file) at the end of its loop has no effect. There are still TWO references to the object at this point (proven with xdebug_debug_zval). One contained in $file (which I can unset), but there's another somewhere else that I cannot unset. Calling $this->em->clear() at the end of the main loop has no effect either.
EDIT 2: SOLUTION: The answer by #origaminal led me to the solution, so I accepted his answer instead of providing my own.
In the first method, where I access the files through the association on $application, this has a side effect of initializing the previously uninitialized $files collection on the $application object I'm iterating over in the outer loop.
Calling $em->detach($application) and $em->detach($file) only tells Doctrine's UOW to stop tracking the objects, but it doesn't affect the array of $applications I'm iterating on, which now have populated collection of $files which eat up memory.
I have to unset each $application object after I'm done with it to remove all references to the loaded $files. To do this, I modified the loops as such:
$applications = $em->getRepository('AppBundle:Application')->findAll();
$count = count($applications);
for ($i = 0; $i < $count; $i++) {
foreach ($applications[$i]->getFiles() as $file) {
$file->getData();
$em->detach($file);
unset($file);
}
$em->detach($applications[$i]);
unset($applications[$i]);
// Don't NEED to force GC, but doing so helps for testing.
gc_collect_cycles();
}
Cascade
EntityManager::detach should indeed remove all references Doctrine has to the enities. But it does not do the same for associated entities automatically.
You need to cascade this action by adding detach the cascade option of the association:
/**
* #ORM\OneToMany(
* targetEntity="AppBundle\Entity\ApplicationFile",
* mappedBy="application",
* cascade={"remove", "detach"},
* orphanRemoval=true
* )
*/
private $files;
Now $em->detach($app) should be enough to remove references to the Application entity as well as its associated ApplicationFile entities.
Find vs Collection
I highly doubt that loading the ApplicationFile entities through the association, in stead of using the repository to findBy() them, is the source of your issue.
Sure that when loaded through the association, the Collection will have a reference to those child-entities. But when the parent entity is dereferenced, the entire tree will be garbage collected, unless there are other references to those child entities.
I suspect the code you show is pseudo/example code, not the actual code in production. Please examine that code thoroughly to find those other references.
Clear
Sometimes it worth clearing the entire EntityManager and merging a few entities back in. You could try $em->clear() or $em->clear('AppBundle\Entity\ApplicationFile').
Clear has no effect
You're saying that clearing the EntityManager has no effect. This means the references you're searching for are not within the EntityManager (of UnitOfWork), because you've just cleared that.
Doctrine but not Doctrine
Are you using any event-listeners or -subscribers? Any filters? Any custom mapping types? Multiple EntityManagers? Anything else that could be integrated into Doctrine or its life-cycle, but is not necessarily part of Doctrine itself?
Especially event-listeners/subscribers are often overlooked when searching for the source of issues. So I'd suggest you start to look there.
If we are speaking about your first implementation you have extra links to the collection in the PersistentCollection::coll of the Application::files property - this object is created by Doctrine on Application instantiation.
With detach you are just deleting UoW links to the object.
There are different ways to fix this but a lot of hacks should be applied. Most nice way probably to detach also Application object and unset it.
But it is still preferable to use more advanced ways for a batch processing: some were listed in the other answer. The current way forces doctrine to make use proxies and throws extra queries to DB to get the files of the current object.
Edit
The difference between the first and the second implementation is that there are no circular references in the second case: Application::files stays with uninitialized PersistenceCollection (with no elements in coll).
To check this - can you try to drop the files association explicitly?
The trick is in PHP's garbage collector, that works a bit odd. First off all each time the scripts need memory it will allocate memory from RAM, even if you use unset(), $object = null, or other tricks to free the memory, the allocated memory will not be returned to Operating System till the script is not finished and the process associated with it killed.
How to fix that ?
Usually is done on Linux Systems
Create commands that runs the needed script with limit, offset parameters and re-run the needed scripts in small batches more times. In this way, the script will use less memory, and each time the memory will be freed when the script will be finished.
Get rid of Doctrine it balloons by himself the memory, PDO is much faster and less costly.
For that kind of task where objects in memory could lead to that leaks, you should use Doctrine2 iterators
$appFiles = $this
->em
->getRepository('AppBundle:ApplicationFile')
->findBy(array('application' => $application));
should be refactored in order to return a query object and not an ArrayCollection, then from that query object, you can easily call iterate() method and clean the memory after every object inspection
Edit
You have "hidden references" because detach operation will not delete the object in memory, it only tells to EntityManager not to handle it anymore. This is why you should use my solution or unset() the object with php function.
I work on a project which uses Google's closure compiler with advanced optimizations turned on. I would like to include moment.js in the compilation, however all of my attempts have been fruitless.
I have tried exporting the moment function, but there are still run time problems, and a some compile errors.
Has anyone successfully compiled moment.js with advanced optimizations, or know how to do so?
The only solution I can come up with, is to concatenate the minified file to the compiled source and use externs for every function I use from moment.js. But this is not an ideal solution.
I saw two issues with the code which would have to be corrected before momentjs would be compatible with ADVANCED_OPTIMIZATIONS. There may be more, but these were the glaring ones:
Using an alias for the prototype: All references to .fn would need to be replaced with .prototype.
Using a helper function to add methods: the extend method hides definitions from the compiler. All uses of the extend helper function would have to be refactored so that they do not hide the property assignments from the compiler.
I can't get it to work either as of 26 March 2015, but the existence of this suggests that it's possible. Here are the externs
You've gotta write your own externs file for moment.js for what you use from it (or the entire object, but I find that a bit of extra work for no reason).
For example, I've got this snippet to test if an input's date is within 14 days from now
$checkout.find('.date-input').on('input', /** #this {Element} */ function () {
const $this = $(this);
const Days = Number($this.attr('data-days'));
if (Days > 0 && moment(/** #type {string} */($this.val())).diff(moment(), 'days') < Days) {
$checkout.find('.date-warning').removeClass('d-none');
} else {
$checkout.find('.date-warning').addClass('d-none');
}
});
And the only way I'd get that to compile correctly with advanced mode is by creating this extern.
/**
* #fileoverview Externs for moment
*
* #externs
*/
/**
* #param {string=} date
* #constructor
* #return {!moment}
*/
function moment(date) {}
/**
* #param {!moment} m
* #param {string} unit
* #return {number}
*/
moment.diff = function (m, unit) {};
moment.prototype.diff = moment.diff;
Now clearly that description of the moment function isn't perfect; it's missing some parameters that the moment function has, but I'm not using them so it doesn't matter to me.
But that's how I start my externs. I start basic as the need arises and then I continue to grow the externs file with the more functions I need from a library.
And don't forget to tell Closure Compiler where your extern is located with the flag --externs 'externs/moment.js'.
I am trying to compile my code generated by browserify with Google Closure Compiler using Advanced Optimization.
Tried running browserify with different flags, no success so far.
Have any one have some experience with that?
I had to change the file: /node_modules/browserify/node_modules/browser-pack/_prelude.js with google closure annotations and add the externs files as
/**
* #param {*=}o
* #param {*=}u
*/
function require(o,u){}
What errors/warnings do you get?
In the SUPPLY automatically-generated function modules, there can be seen the following comments:
* General Notes
* =============
* A common scenario for a supply method is to aquire key
* informations from the parameter <parent_element> and then
* to invoke a data provider.
* A free navigation thru the context, especially to nodes on
* the same or deeper hierachical level is strongly discouraged,
* because such a strategy may easily lead to unresolvable
* situations!!
*
** data declaration
* DATA lt_nod TYPE wd_this->Elements_nod.
* DATA ls_nod LIKE LINE OF lt_nod.
** #TODO compute values
** e.g. call a data providing **FuBa**
I understand the dangers of navigating through nodes that have an associated Supply Function but haven't been initialized yet - this basically leads to dead locks.
What i'd like to know is what's a FuBa, or data provider and how to use that - all the examples i've found only supply data for a node in a trivial manner, and don't tackle this problem.
Is that some way to register the nodes to be updated later... or... dunno ?
In this case, data provider is not a technical term, it's just some coding that provides the data you want to add to the context. Whatever that may be depends on your application context - anything from a local or remote function module or method call, a call to your assistance class or even - if you really want to adopt bad coding habits - to a direct database access.
FuBa is an abbreviation of Funktionsbaustein = function module.