I'm trying to run a migration script in a Micronaut app configured with Flyway integration. The app runs as expected and applies the migration scripts without any Flyway placeholders. However, whenever I add a Flyway placeholder to a migration script the application does not start anymore due to a FlywayException:
org.flywaydb.core.api.FlywayException: No value provided for placeholder expressions: ${my_placeholder}. Check your configuration!
I've tried to configure the placeholder in application.yml using the Micronaut Flyway placeholders configuration property (as described here https://micronaut-projects.github.io/micronaut-flyway/latest/guide/index.html#io.micronaut.configuration.dbmigration.flyway.FlywayConfigurationProperties) but the placeholder simply isn't picked up. The application.yml file looks like this:
flyway:
datasources:
default:
locations: classpath:migrations
placeholders:
my_placeholder: "some value"
I've also tried creating a flyway.properties file with the placeholder defined according to what is described in this SO answer https://stackoverflow.com/a/9420671/2185719 but that did not work either
# flyway.properties
flyway.placeholders.my_placeholder=some value
While debugging Flyway startup (specifically in PlaceholderReplacingLine) I noticed that the placeholderReplacer object held a placeholder where the _ (underscores) had been replaced by - (dashes). Changing my_placeholder to my-placeholder in the migration script fixed the issue.
Related
I am having trouble getting quarkus.flyway.placeholders working in my Quarkus app.
I have this line defined in my application.properties file
quarkus.flyway.placeholders.myuser=my_user
in my sql file I have this line
GRANT DELETE, INSERT, SELECT, UPDATE ON survey.answers TO ${myuser};
the error I'm getting is
org.flywaydb.core.api.FlywayException: No value provided for placeholder: ${myuser}. Check your configuration!
Here are the things I've tried:
upgraded to Quarkus 1.13.6.Final
Tried setting
quarkus.flyway.placeholder-prefix=#[
quarkus.flyway.placeholder-suffix=]
As shown in the integration test:
https://github.com/quarkusio/quarkus/tree/main/integration-tests/flyway
Thank you
Matthew
I see you have a typo in your configuration or sql script.
With the following settings:
quarkus.flyway.placeholder-prefix=#[
quarkus.flyway.placeholder-suffix=]
In your sql file the placeholder should be defined as a #[myuser], not $[myuser]
You can also change the placeholder-prefix definition in your application.properties file to support prefix you already have in the sql file.
At one time I have two flyway users.
quarkus.flyway.owner this one has create privileges
quarkus.flyway.user this one has fewer privileges.
This item was not associated with the user role.
quarkus.flyway.placeholders.myuser=my_user
After changing it to quarkus.flyway.owner.placeholders.myuser=my_user
it started working.
So, I'm trying out EF Code First, so that I can have the code drive updates to the database. I'm working on a ClickOnce app using LocalDB, so figured this may be the best solution for me, since otherwise changes to the MDF file will cause it to be overwritten on the client when deployed, thus losing everything entered before.
However, I'm now having my fair share of all new problems around Code First Migrations. I've followed through a Code First Migrations on MSDN, and I've managed to get the initial Configuration created, as well as the initial database creation.
The problems begin when I try to make my first actual migration. I added one single field to one of my models, and tried to make an explicit migration to handle that schema change for the next time I publish. Well...
PM> Add-Migration AddIsPercentField
Unable to generate an explicit migration because the following explicit migrations are pending: [201601052011180_InitialCreate]. Apply the pending explicit migrations before attempting to generate a new explicit migration.
Ok... I'll run update and try again:
PM> Update-Database
Specify the '-Verbose' flag to view the SQL statements being applied to the target database.
Applying explicit migrations: [201601052011180_InitialCreate].
Applying explicit migration: 201601052011180_InitialCreate.
Unable to update database to match the current model because there are pending changes and automatic migration is disabled. Either write the pending model changes to a code-based migration or enable automatic migration. Set DbMigrationsConfiguration.AutomaticMigrationsEnabled to true to enable automatic migration.
You can use the Add-Migration command to write the pending model changes to a code-based migration.
PM> Add-Migration AddIsPercentField
Unable to generate an explicit migration because the following explicit migrations are pending: [201601052011180_InitialCreate]. Apply the pending explicit migrations before attempting to generate a new explicit migration.
That's familiar, as that's the error (blatant lie?) it just told me earlier. Well, maybe if I undo my changes and update again, it will move to a valid state:
PM> Update-Database
Specify the '-Verbose' flag to view the SQL statements being applied to the target database.
Applying explicit migrations: [201601052011180_InitialCreate].
Applying explicit migration: 201601052011180_InitialCreate.
Running Seed method.
Ok, no warning this time. Should be golden. Field added back, project rebuilt. Here we go:
PM> Add-Migration AddIsPercentField
Unable to generate an explicit migration because the following explicit migrations are pending: [201601052011180_InitialCreate]. Apply the pending explicit migrations before attempting to generate a new explicit migration.
So... is there actually a working way to generate explicit migrations for any changes beyond the first?
EDIT: I made some forward progress on this, I believe. I did notice that the __MigrationHistory table was not generated in my .mdf after running Update-Database, even though it said everything completed just fine. I believe the issue is actually around how the local database works within the application. The connection string references AttachDbFilename=|DataDirectory|. What I think is going on is that it is deploying the .mdf temporarily, updating that temporary deployment, thus ultimately not committing the changes.
I'm working on a solution I have in mind, which is to have migrations work against a copy of the blank .mdf put in a static location, so that the static .mdf will be used to track and determine changes, while the blank .mdf will be what goes out to clients with the deployment.
I found that the root of my problem was that the console commands were not actually able to make changes, thus track migrations, to my data file. This was due to the connection string to the data file referencing a deployed location, so the file being updated was merely temporary.
This was in part a good thing, because the whole point of using Code First Migrations in my project was to avoid a hash signature change to my .mdf (which should have simply remained blank, as a placeholder) when publishing, so that the data from previous versions would never be overridden and discarded. However, that also introduced the obvious (in retrospect) problem that EF could not track changes due to there never being a __MigrationHistory table.
The solution upon which I arrived was to have two .mdf files. The blank one, for deployments, and a second one, to which I would interact with Code First Migrations. So, I have the initial MyData.mdf of Build Action Content, and a second MyDataDesignTime.mdf of Build Action None. (The "Design Time" migration database shouldn't be deployed.)
Using this approach, I found that I could now work successfully with migrations, calling Update-Database and Add-Migration, making sure to pass the -ConnectionString parameter with AttachDbFilename pointed to the full path to my design time database.
Later, getting lazy to supply a long -ConnectionString parameter on every migration command, I added the design time path to my config connection strings, and updated my DbContext so that it uses the design time path initially, but which I would change at the beginning of run-time to use my actual target data file:
public partial class MyData : DbContext
{
public const string DesignTimeConnection = "MyDataConnectionStringDesignTime";
public static string ConnectionName { get; set; } = DesignTimeConnection;
public MyData()
: base("name=" + ConnectionName)
{
}
...
}
And at application initialization:
MyData.ConnectionName = "MyDataConnectionString";
This works, and it makes things simpler on me. However, the one minor issue I'm left with is that I have a full static path which applies only to my environment left in the app.config file. Not currently an issue, as I'm the only dev on this project, but it's a code smell that I'm not happy with. Is there some path variable that I can use, such that it still points to the actual design time data (not any temporary, deployed file), but does so relative to the active, open project?
Is it possible to override url that is given in .conf file when invoking migrate from command line?
flyway -flyway.url=jdbc:jtds:sqlserver://test_sqlserver:1433/mydatabase migrate
This above does not seem to work.
Yes. Command-line options are not prefixed with flyway.
In your case this would mean
flyway -url=jdbc:jtds:sqlserver://test_sqlserver:1433/mydatabase migrate
To expand on that comment:
-D sets a system property (in this case flyway.baselineVersion), it's a parameter to java, not to flyway.
-baselineVersion=1 is a parameter to flyway
Both happen to work only because flyway uses both command line arguments and system properties (in addition to the configuration file), they are not otherwise related.
I have a spring boot application that uses the actuator, auto-configuration and JPA. I want to be able to use an in-memory DB in my test profile, a MySQL DB configuration during development and a separate production DB configuration when the app is deployed in production. Presumably from the java command line I should be able to specify the environment and the right configuration file or config block in application.properties (or .yml) will be picked up.
I have not found a good post with example describing how to do this switching so I thought I'd ask if anyone has a good example. My main aim is to pre-define the spring.datasource and spring.jpa properties at build time and then at run-time switch the app config per environment "dynamically" using the java command line argument. Secondary goal would be to do the same with the management configurations, etc.
Thank you.
Thanks to #Richard for the mention of spring.profiles.active JVM variable. Since my question was specific to the way Spring Boot does this and since there is much more to the answer, I am inclined to answer this myself and include all the details of how I arrived at the answer in the hopes that it will save others time.
First, you can indeed pick the correct profile on the java command line by adding -Dspring.profiles.active=profile_name when you are running your Spring Boot app. (this is assuming your deployment preference is an uber jar with embedded container - Tomcat in my case)
I wanted to leave MySQL datasource configurations under the default profile and put H2 in-memory datasource configuration under a test profile. However, the way Spring Boot picks the right datasource based on profile is not so obvious. Even though I had MySQL details under the default profile and I had the in-memory H2 datasource details under the test profile, it would still pick H2 as the datasource even when spring.profiles.active was omitted from the command line. This was contrary to my assumption that default profile will be picked, well, by default :-)
I ended up having to put H2 configuration under the default profile and then create a local profile that included the MySQL datasource configuration. Here's what I ended up with in my application.yml
spring:
profiles: default
spring:
datasource:
driverClassName: org.h2.Driver
url: jdbc:h2:mem:sampletest;MODE=MySQL
---
spring:
profiles: test
spring.jpa:
hibernate:
ddl-auto: create-drop
---
spring:
profiles: local
spring.datasource:
driverClassName: com.mysql.jdbc.Driver
url: jdbc:mysql://127.0.0.1/sampledev
username: sample
password: sample
spring.jpa:
hibernate:
dialect: org.hibernate.dialect.MySQLInnoDBDialect
ddl-auto: update
This worked. I was able to switch between default profiles and the local profile by omitting or adding the -Dspring.profiles.active=local on the java command line. Because test profile inherits from default it is also using H2
One more nuance: I added ddl-auto: create-drop to the test profile which uses the in-memory DB to facilitate automatic table creation / teardown for unit tests. But for the local profile which uses MySQL I changed it to update. Implication being that for the local profile I have to first create the database outside of the application.
this article shows how to use spring profiles, available in spring 3.1 and later. It will do exactly what you want.
http://chariotsolutions.com/blog/post/spring-3-1-environment-profiles-2/
set a JVM variable like this: spring.profiles.active=development
then in your configuration xml you can wrap environment specific xml with the profile tags
<beans profiles="development">
<bean id="dataSource" class="..."></bean>
<bean id="messagingProvider" class="..."></bean>
</beans>
You can also set the profile on annotation-driven classes with #Profile("development") at the beginning of the class. That class will only be autowired if the profile matches.
For unit tests you can set the active profile on a test class with #ActiveProfile(profiles = "test", "CI"), it will run using test and CI resources
I'm trying to generate a full SQL script to deploy my application for the first time. I know that this command from the package manager console is supposed to create a full SQL script (see this SO question)
Update-Database -Script -SourceMigration:0
I've also read that $InitialDatabase is supposed to work instead of '0' for the source migration.
However, I have two configurations in my migrations folder. I'm using the following syntax for it:
Update-Database -Script -SourceMigration:0 -ConfigurationTypeName MyConfig
I'm getting back an empty script for all my configurations.
As a note, I have automatic migrations enabled and haven't added any explicit migration to it.
Having two migrations configurations in the same namespace is an unsupported scenario. We closed a bug similar to this as "by design" recently. The workaround is to have your migrations configurations in separate namespaces.
The most reliable solution to this issue would just be to put your configurations in different folders and use the MigrationsDirectory variable as shown below, then any explicit migrations generated for the configuration should be placed in the same folder
internal sealed class Configuration : DbMigrationsConfiguration<TestMultipleMigrationsScript.BlogContext>
{
public Configuration()
{
AutomaticMigrationsEnabled = false;
MigrationsDirectory = #"directory";
}
}