I have created two migration SQL files for my Java web application: V1 and V2.
Upon the first start of the application, everything looks and works fine. Here's what I'm seeing in the log:
23-Aug-2020 09:31:35.923 INFO [localhost-startStop-1] org.flywaydb.core.internal.command.DbMigrate.info Current version of schema `cloudregs`: << Empty Schema >>
23-Aug-2020 09:31:36.074 INFO [localhost-startStop-1] org.flywaydb.core.internal.command.DbMigrate.info Migrating schema `cloudregs` to version 1 - initial
23-Aug-2020 09:31:36.118 WARNING [localhost-startStop-1] org.flywaydb.core.internal.sqlscript.DefaultSqlScriptExecutor.warn DB: Changing sql mode 'NO_AUTO_CREATE_USER' is deprecated. It will be removed in a future release. (SQL State: HY000 - Error Code: 3090)
23-Aug-2020 09:31:37.312 WARNING [localhost-startStop-1] org.flywaydb.core.internal.sqlscript.DefaultSqlScriptExecutor.warn DB: Changing sql mode 'NO_AUTO_CREATE_USER' is deprecated. It will be removed in a future release. (SQL State: HY000 - Error Code: 3090)
23-Aug-2020 09:31:37.419 INFO [localhost-startStop-1] org.flywaydb.core.internal.command.DbMigrate.info Migrating schema `cloudregs` to version 2 - user-invite
23-Aug-2020 09:31:37.459 WARNING [localhost-startStop-1] org.flywaydb.core.internal.sqlscript.DefaultSqlScriptExecutor.warn DB: Changing sql mode 'NO_AUTO_CREATE_USER' is deprecated. It will be removed in a future release. (SQL State: HY000 - Error Code: 3090)
23-Aug-2020 09:31:40.742 WARNING [localhost-startStop-1] org.flywaydb.core.internal.sqlscript.DefaultSqlScriptExecutor.warn DB: Changing sql mode 'NO_AUTO_CREATE_USER' is deprecated. It will be removed in a future release. (SQL State: HY000 - Error Code: 3090)
23-Aug-2020 09:31:40.812 INFO [localhost-startStop-1] org.flywaydb.core.internal.command.DbMigrate.info Successfully applied 2 migrations to schema `cloudregs` (execution time 00:04.912s)
And here's the corresponding flyway_schema_history:
+----------------+---------+-------------+------+---------------------+----------+--------------+---------------------+----------------+---------+
| installed_rank | version | description | type | script | checksum | installed_by | installed_on | execution_time | success |
+----------------+---------+-------------+------+---------------------+----------+--------------+---------------------+----------------+---------+
| 2 | 2 | user-invite | SQL | V2__user-invite.sql | 39331208 | root | 2020-08-23 09:31:40 | 3343 | 1 |
+----------------+---------+-------------+------+---------------------+----------+--------------+---------------------+----------------+---------+
It looks that only the second migration has been saved to flyway_schema_history.
When re-starting the application, I'm getting the following error:
Validate failed:
Detected resolved migration not applied to database: 1
What am I missing?
Edit: the issue seems to be fixed by upgrading from Flyway version 6.4.2 to 6.5.5
Related
I am referencing to this doc and this article in linking a Postgres database to Airflow.
Particularly, I added this line to the file airflow.cfg:
sql_alchemy_conn = postgresql+psycopg2://airflowadmin:airflowadmin#localhost/airflowdb
where airflowadmin is both the username and password for the postgres user and password, and airflowdb is a postgres db created, with airflowadmin having all the privileges.
Now, when I initialize the database with airflow db init, however, I still see sqlite being the linked database. Full output:
DB: sqlite:////home/userxxxx/airflow/airflow.db
[2021-09-07 12:43:53,827] {db.py:702} INFO - Creating tables
INFO [alembic.runtime.migration] Context impl SQLiteImpl.
INFO [alembic.runtime.migration] Will assume non-transactional DDL.
WARNI [airflow.models.crypto] empty cryptography key - values will not be stored encrypted.
WARNI [unusual_prefix_xxxxxxxxxxxxxxxxxxxxxxxxx_example_kubernetes_executor_config] Could not import DAGs in example_kubernetes_executor_config.py: No module named 'kubernetes'
WARNI [unusual_prefix_xxxxxxxxxxxxxxxxxxxxxxxxxxxxx_example_kubernetes_executor_config] Install kubernetes dependencies with: pip install apache-airflow['cncf.kubernetes']
Initialization done
What am I missing?
Make sure the airflow.cfg file you are changing is the same that is actually being loaded by Airflow. From the CLI run:
airflow info
Search under the Paths info section and compare it with the path of the folder with the airflow.cfg file that you are modifying.
airflow info:
ache Airflow
version | 2.1.2
executor | SequentialExecutor
task_logging_handler | airflow.utils.log.file_task_handler.FileTaskHandler
sql_alchemy_conn | sqlite:////home/vagrant/airflow/airflow.db
dags_folder | /home/vagrant/airflow/dags
plugins_folder | /home/vagrant/airflow/plugins
base_log_folder | /home/vagrant/airflow/logs
remote_base_log_folder |
System info
OS
...
...
Paths info
airflow_home | /home/vagrant/airflow
...
When not defined during the local installation process, the default value of airflow_homeis AIRFLOW_HOME=~/airflow, so I guess that may be the cause of your problem.
I am trying to install three node airflow cluster. Each node has airflow scheduler, airflow worker, airflow webserver, also it has celery, RabbitMQ cluster and Postgres multi master cluster(implemented with Bucardo). Versions of software:
Airflow 2.0.1
Postregsql 13.2
Ubuntu 20.04
pyhton 3.8.5
celery 4.4.7
bucardo 5.6.0
RabbitMQ 3.8.2
And I occur the problem starting airflow scheduler.
When I launch the first one(database is empty), it successfully starts.
But then when I'm launching another scheduler on another machine(I tried to launch on the same machine too), it fails with the following:
sqlalchemy.exc.IntegrityError: (psycopg2.errors.UniqueViolation) duplicate key value violates unique constraint "job_pkey"
DETAIL: Key (id)=(25) already exists.
[SQL: INSERT INTO job (dag_id, state, job_type, start_date, end_date, latest_heartbeat, executor_class, hostname, unixname) VALUES (%(dag_id)s, %(state)s, %(job_type)s, %(start_date)s, %(end_date)s, %(latest_heartbeat)s, %(executor_class)s, %(hostname)s, %(unixname)s) RETURNING job.id]
[parameters: {'dag_id': None, 'state': 'running', 'job_type': 'SchedulerJob', 'start_date': datetime.datetime(2021, 4, 21, 7, 39, 20, 429478, tzinfo=Timezone('UTC')), 'end_date': None, 'latest_heartbeat': datetime.datetime(2021, 4, 21, 7, 39, 20, 429504, tzinfo=Timezone('UTC')), 'executor_class': 'CeleryExecutor', 'hostname': 'hostname', 'unixname': 'root'}]
(Background on this error at: http://sqlalche.me/e/13/gkpj)
After trying to launch a few times eventually scheduler is working. I am assuming id is incremented and then data is successfully added into database:
airflow=> select * from job order by state;
id | dag_id | state | job_type | start_date | end_date | latest_heartbeat | executor_class | hostname | unixname
----+--------+---------+--------------+-------------------------------+-------------------------------+-------------------------------+----------------+------------------------------+----------
26 | | running | SchedulerJob | 2021-04-21 07:39:22.243721+00 | | 2021-04-21 07:39:22.243734+00 | CeleryExecutor | machine name | root
25 | | running | SchedulerJob | 2021-04-21 07:39:14.515009+00 | | 2021-04-21 07:39:19.632811+00 | CeleryExecutor | machine name | root
There is a warning with log tables as well(If the second and subsequent schedulers successfully started):
WARNING - Failed to log action with (psycopg2.errors.UniqueViolation) duplicate key value violates unique constraint "log_pkey"
DETAIL: Key (id)=(40) already exists.
I understand why scheduler cannot insert data into table, but how should it work correctly, how to launch multiple schedulers? Official documentation tells no additional configuration required. Hope I explained very clear. Thanks!
Looks like there is a race condition between the Airflow Schedulers and Bucardo.
Probably the easiest way to fix it is to query all servers sequentially with a connection string like this in your airflow.cfg (the same on all nodes):
[core]
sql_alchemy_conn=postgresql://USER:PASS#/DB?host=node1:port1&host=node2B&host=node3
For this to work you'll need sqlalchemy >= 1.3
Why this happens
There is a race condition between your schedulers and bucardo trying to read and write data from the table in different hosts. Changes does not propagate as quickly as they should and server writes to the table fail.
Even if you treat all your nodes as "multimaster", making all nodes look first at the same server will remediate this problem. In case of failure, they will use the second one.
I asked Airflow developers. The problem is in Bucardo since it does not support
'SELECT ... FOR UPDATE' :
I suspect that the problem is with Bucardo, which does not support record locking properly. We have high expectations, because it is a key protection mechanism against running the same task by many schedulers.
http://airflow.apache.org/docs/apache-airflow/stable/scheduler.html#database-requirements
If that doesn't work you will have problems with duplicate keys.
Thanks!
With Karaf 4.1.3 and Karaf 4.1.4 when I connect as a client.sh (with the log level set to debug) and perform a log:tail I get the following...
2018-01-05 13:38:00,348 | DEBUG | 0]-nio2-thread-1 | Nio2Session | 48 - org.apache.sshd.core - 1.6.0 | handleCompletedWriteCycle(Nio2Session[local=/127.0.0.1:8101, remote=/127.0.0.1:50379]) finished writing len=68
screams out on to the console, and unless I control C the whole karaf container hangs and in the server window I get the following message:
2018-01-05 13:37:58,819 RMI Scheduler(0) ERROR Recursive call to appender PaxOsgi
2018-01-05 13:37:58,819 RMI Scheduler(0) ERROR Recursive call to appender PaxOsgi
2018-01-05 13:37:58,819 RMI Scheduler(0) ERROR Recursive call to appender PaxOsgi
2018-01-05 13:37:58,819 RMI Scheduler(0) ERROR Recursive call to appender PaxOsgi
This is with a clean unzip running oracle 1.8 on both windows and ubuntu.
Any ideas?
Confirmed a bug and ticket raised:
https://issues.apache.org/jira/browse/KARAF-5559
I've just started my journey with DSE graph (had fair bit of understanding of Titan earlier). I've set-up DSE graph with Datastax 5.0.3.
When trying to create a search index for a property, I am getting following exception.
schema.vertexLabel('Employee').index('search').search().by('story').asText().add()
org.apache.tinkerpop.gremlin.driver.exception.ResponseException: Cannot create search index with workload: Analytics
I was able to create properties, materialized and secondary indices. But when I tried to create the search index, I am facing this issue.
I realized that while bringing up my single node cluster, I've had to turn off -s flag because that is not letting me to bring up the DSE server. There was some exception when bringing up the node for the first time I was not supposed to set the -s flag as per some Datastax developer QAs.
entrypoint: ["/usr/local/bin/dse-entrypoint", "-k", "-g"]
Now when I tried to enable the -s flag, my node is not coming up and I am getting the following exception.
dse | WARN 12:54:28,038 CLibrary.java:163 - Unable to lock JVM memory (ENOMEM). This can result in part of the JVM being swapped out, especially with mmapped I/O enabled. Increase RLIMIT_MEMLOCK or run Cassandra as root.
dse | WARN 12:54:28,038 StartupChecks.java:118 - jemalloc shared library could not be preloaded to speed up memory allocations
dse | WARN 12:54:28,039 StartupChecks.java:150 - JMX is not enabled to receive remote connections. Please see cassandra-env.sh for more info.
dse | WARN 12:54:28,047 SigarLibrary.java:174 - Cassandra server running in degraded mode. Is swap disabled? : false, Address space adequate? : true, nofile limit adequate? : true, nproc limit adequate? : true
dse | ERROR 12:54:28,710 CassandraDaemon.java:709 - Cannot start node if snitch's data center (SearchGraphAnalytics) differs from previous data center (GraphAnalytics). Please fix the snitch configuration, decommission and rebootstrap this node or use the flag -Dcassandra.ignore_dc=true.
dse | INFO 12:54:28,717 DseDaemon.java:556 - DSE shutting down...
dse | INFO 12:54:28,718 PluginManager.java:104 - All plugins are stopped.
dse | Oct 27, 2016 12:54:28 PM org.apache.coyote.http11.Http11Protocol pause
dse | INFO: Pausing Coyote HTTP/1.1 on http-172.19.0.3-8983
dse | Oct 27, 2016 12:54:29 PM org.apache.catalina.core.StandardService stop
dse | INFO: Stopping service Solr
dse | INFO 12:54:29,907 SolrHttpAuditLogFilter.java:225 - Shutting down Solr audit logging filter
dse | INFO 12:54:29,924 RepeatablePOSTQueryFilter.java:81 - Shutting down com.datastax.bdp.search.solr.RepeatablePOSTQueryFilter filter
dse | Oct 27, 2016 12:54:29 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
dse | SEVERE: The web application [/solr] appears to have started a thread named [Thread-3] but has failed to stop it. This is very likely to create a memory leak.
dse | Oct 27, 2016 12:54:29 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
dse | SEVERE: The web application [/solr] appears to have started a thread named [NonPeriodicTasks:1] but has failed to stop it. This is very likely to create a memory leak.
dse | Oct 27, 2016 12:54:29 PM org.apache.coyote.http11.Http11Protocol destroy
dse | INFO: Stopping Coyote HTTP/1.1 on http-172.19.0.3-8983
dse | INFO 12:54:33,191 MessageServer.java:129 - internode-messaging message server finished shutting down.
dse | INFO 12:54:37,209 MessageServer.java:129 - internode-messaging message server finished shutting down.
dse | Exception in thread "Daemon shutdown" java.lang.AssertionError
dse | at org.apache.cassandra.gms.Gossiper.addLocalApplicationStateInternal(Gossiper.java:1427)
dse | at org.apache.cassandra.gms.Gossiper.addLocalApplicationStates(Gossiper.java:1451)
dse | at org.apache.cassandra.gms.Gossiper.addLocalApplicationState(Gossiper.java:1441)
dse | at com.datastax.bdp.gms.DseState.setActiveStatusSync(DseState.java:241)
dse | at com.datastax.bdp.server.DseDaemon.preStop(DseDaemon.java:576)
dse | at com.datastax.bdp.server.DseDaemon.safeStop(DseDaemon.java:587)
dse | at com.datastax.bdp.server.DseDaemon.lambda$getShutdownHook$226(DseDaemon.java:905)
dse | at java.lang.Thread.run(Thread.java:745)
Please suggest how I can rectify this situation and be able to add the searchIndex to my properties.
This error is stating that you are starting up the cluster with a different name than what the cluster expects based on a previous configuration/start.
By default, unless you override the cluster name in the cassandra.yaml file, your cluster will startup with a name based on the workloads you enable, i.e. -s, -t. In your case, since you started the cluster as an Analytics cluster and then restarted it as a SearchAnalytics cluster, the cluster name is defaulting to the new cluster name, which doesn't match the old cluster name.
The easiest thing to do here is to wipe your cassandra commit log, caches, and data directory and restart the node. That will wipe out the old cluster name from your system tables and will allow the cluster to start. Doing this will wipe out any data you had in the cluster.
Our Nexus server filled its disk, and when we restarted it I got the following error:
jvm 1 | 2015-08-18 09:44:13,660+1000 ERROR [jetty-main-1] *SYSTEM com.bolyuba.nexus.plugin.npm.service.internal.orient.OrientMetadataStore - Life-cycle operation failed
jvm 1 | com.orientechnologies.orient.core.exception.OStorageException: Cannot open local storage '/nexus/db/npm' with mode=rw
jvm 1 | at com.orientechnologies.orient.core.storage.impl.local.OAbstractPaginatedStorage.open(OAbstractPaginatedStorage.java:220) ~[nexus-npm-repository-plugin-2.11.3-01/:na]
jvm 1 | at com.orientechnologies.orient.core.db.document.ODatabaseDocumentTx.open(ODatabaseDocumentTx.java:244) ~[nexus-npm-repository-plugin-2.11.3-01/:na]
jvm 1 | at com.bolyuba.nexus.plugin.npm.service.internal.orient.OrientMetadataStore.doStart(OrientMetadataStore.java:107) ~[nexus-npm-repository-plugin-2.11.3-01/:na]
jvm 1 | at org.sonatype.sisu.goodies.lifecycle.LifecycleSupport$Handler.doStart(LifecycleSupport.java:70) ~[goodies-lifecycle-1.9.jar:1.9]
jvm 1 | at org.sonatype.sisu.goodies.lifecycle.LifecycleHandlerContext$MainMap_Starting.started(LifecycleHandlerContext.java:255) ~[goodies-lifecycle-1.9.jar:1.9]
jvm 1 | at org.sonatype.sisu.goodies.lifecycle.LifecycleHandlerContext.started(LifecycleHandlerContext.java:57) ~[goodies-lifecycle-1.9.jar:1.9]
jvm 1 | at org.sonatype.sisu.goodies.lifecycle.LifecycleSupport.start(LifecycleSupport.java:129) ~[goodies-lifecycle-1.9.jar:1.9]
jvm 1 | at com.bolyuba.nexus.plugin.npm.service.internal.orient.OrientMetadataStoreLifecycle.on(OrientMetadataStoreLifecycle.java:51) [nexus-npm-repository-plugin-2.11.3-01/:na]
jvm 1 | Caused by: com.orientechnologies.orient.core.exception.OStorageException: File with name internal.pcl does not exist in storage npm
jvm 1 | at com.orientechnologies.orient.core.index.hashindex.local.cache.OWOWCache.openFile(OWOWCache.java:249) ~[nexus-npm-repository-plugin-2.11.3-01/:na]
jvm 1 | at com.orientechnologies.orient.core.index.hashindex.local.cache.OReadWriteDiskCache.openFile(OReadWriteDiskCache.java:159) ~[nexus-npm-repository-plugin-2.11.3-01/:na]
jvm 1 | at com.orientechnologies.orient.core.storage.impl.local.paginated.base.ODurableComponent.openFile(ODurableComponent.java:145) ~[nexus-npm-repository-plugin-2.11.3-01/:na]
jvm 1 | at com.orientechnologies.orient.core.storage.impl.local.paginated.OPaginatedCluster.open(OPaginatedCluster.java:203) ~[nexus-npm-repository-plugin-2.11.3-01/:na]
jvm 1 | at com.orientechnologies.orient.core.storage.impl.local.OAbstractPaginatedStorage.open(OAbstractPaginatedStorage.java:190) ~[nexus-npm-repository-plugin-2.11.3-01/:na]
jvm 1 | ... 61 common frames omitted
How do I fix this?
I couldn't find any info on this error as it related to Nexus specifically, so I thought I would share my solution.
First, you need to shut down Nexus and remove the /db/npm directory. Then boot Nexus and run the Rebuild hosted npm metadata task. That fixed the issue for me.