Issue
When deploying a Ktor Kotlin application to AppEngine per the Ktor tutorial, the Firestore server authentication is not working, thus data is not being written to the specified Firestore database.
Data is written as expected to Firestore both when the app is run directly in the IntelliJ IDE as well as when it is run with ktor's implementation via the gradle appengineRun command.
There are two sets of AppEngine/Firebase projects for both a staging and production environment. Prior to deploying with the gradle appengineDeploy command the correct SDK configuration been activated and verified via the command gcloud config configurations list.
The strange part is that a few of the apps deployed with these strategies did write to Firestore, however upon deploying the app again Firestore did not show new data being written to it.
Implementation
Ktor Setup
I have the standard ktor required files. I also have an old MANIFEST.MF file from an older implementation. Could that be causing issues?
src/main/resources/application.conf
ktor {
application {
modules = [ Initialization.main ]
}
}
src/main/resources/webapp/WEB-INF/
appengine-web.xml
<?xml version="1.0" encoding="utf-8"?>
<appengine-web-app xmlns="http://appengine.google.com/ns/1.0">
<threadsafe>true</threadsafe>
<runtime>java8</runtime>
</appengine-web-app>
web.xml
<?xml version="1.0" encoding="ISO-8859-1" ?>
<web-app xmlns="http://java.sun.com/xml/ns/javaee" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://java.sun.com/xml/ns/javaee http://java.sun.com/xml/ns/javaee/web-app_3_0.xsd" version="3.0">
<servlet>
<display-name>KtorServlet</display-name>
<servlet-name>KtorServlet</servlet-name>
<servlet-class>io.ktor.server.servlet.ServletApplicationEngine</servlet-class>
<!-- path to application.conf file, required -->
<init-param>
<param-name>io.ktor.config</param-name>
<param-value>application.conf</param-value>
</init-param>
</servlet>
<servlet-mapping>
<servlet-name>KtorServlet</servlet-name>
<url-pattern>/</url-pattern>
</servlet-mapping>
logging.properties
.level = INFO
src/main/META-INF/MANIFEST>MF
Manifest-Version: 1.0
Main-Class: Initialization
Dependencies
For authentication strategies outlined below #1-3 the Firebase Admin library is used: compile 'com.google.firebase:firebase-admin:6.5.0'
For authentication strategy #4 the Google Cloud Firestore library is used: compile 'com.google.cloud:google-cloud-firestore:0.58.0-beta'
build.gradle
group 'coinverse'
version '1.0-SNAPSHOT'
buildscript {
ext.kotlin_version = '1.2.61'
ext.junitJupiterVersion = '5.0.3'
ext.ktor_version = '0.9.4'
ext.appengine_version = '1.9.60'
ext.appengine_plugin_version = '1.3.4'
repositories {
mavenCentral()
jcenter()
}
dependencies {
classpath "org.jetbrains.kotlin:kotlin-gradle-plugin:$kotlin_version"
classpath 'org.junit.platform:junit-platform-gradle-plugin:1.0.3'
classpath "com.google.cloud.tools:appengine-gradle-plugin:$appengine_plugin_version"
}
}
apply plugin: 'java'
apply plugin: 'kotlin'
apply plugin: 'war'
apply plugin: 'com.google.cloud.tools.appengine'
sourceSets {
main.kotlin.srcDirs = [ 'src/main/kotlin' ]
}
sourceCompatibility = 1.8
repositories {
mavenCentral()
jcenter()
maven { url "https://kotlin.bintray.com/ktor" }
}
dependencies {
implementation "org.jetbrains.kotlin:kotlin-stdlib-jdk8:$kotlin_version"
testCompile group: 'junit', name: 'junit', version: '4.12'
testCompile("org.junit.jupiter:junit-jupiter-api:${junitJupiterVersion}")
testRuntime("org.junit.jupiter:junit-jupiter-engine:${junitJupiterVersion}")
testCompile("org.assertj:assertj-core:3.10.0")
testCompileOnly('org.apiguardian:apiguardian-api:1.0.0')
compile 'com.squareup.retrofit2:retrofit:2.3.0'
compile 'com.squareup.retrofit2:converter-gson:2.3.0'
compile 'com.squareup.retrofit2:adapter-rxjava:2.3.0'
compile 'io.reactivex.rxjava2:rxjava:2.2.0'
compile 'com.google.cloud:google-cloud-firestore:0.58.0-beta'
// Or compile 'com.google.cloud:google-cloud-firestore:0.58.0-beta'
compile 'com.google.firebase:firebase-admin:6.5.0'
compile "org.jetbrains.kotlin:kotlin-stdlib-jdk8:$kotlin_version"
compile "io.ktor:ktor-server-servlet:$ktor_version"
compile "io.ktor:ktor-html-builder:$ktor_version"
providedCompile "com.google.appengine:appengine:$appengine_version"
}
kotlin.experimental.coroutines = 'enable'
compileKotlin {
kotlinOptions.jvmTarget = "1.8"
}
compileTestKotlin {
kotlinOptions.jvmTarget = "1.8"
}
task run(dependsOn: appengineRun)
appengine {
deploy {
version = 'price-staging-1021653pm'
stopPreviousVersion = false
}
}
Initializing Firebase Strategies
1. Initialize on Google Cloud Platform
This method is promising as credentials are managed automatically.
// Use the application default credentials
GoogleCredentials credentials = GoogleCredentials.getApplicationDefault();
FirebaseOptions options = new FirebaseOptions.Builder()
.setCredentials(credentials)
.setProjectId(projectId)
.build();
FirebaseApp.initializeApp(options);
Firestore db = FirestoreClient.getFirestore();
2. Initialize on your own server
I've confirmed in GCPs IAM & admin > Service accounts that the key id's match with the Json object being used to authenticate.
I'm using this strategy successfully in another Firestore connected app deployed to AppEngine. The working app is built as a .Jar and deployed directly to AppEngine without using ktor, but rather with the steps outlined here.
// Use a service account
InputStream serviceAccount = new FileInputStream("path/to/serviceAccount.json");
GoogleCredentials credentials = GoogleCredentials.fromStream(serviceAccount);
FirebaseOptions options = new FirebaseOptions.Builder()
.setCredentials(credentials)
.build();
FirebaseApp.initializeApp(options);
Firestore db = FirestoreClient.getFirestore();
In my working .Jar built app I'm passing in the Json object programmatically to avoid issues with the file not being found. I tried the same programmatic implementation for this ktor application. It worked with gradle appengineRun but not when deployed.
val credentials = GoogleCredentials.fromStream(Gson().toJson(FirebaseCredentials(
"service_account",
"project-name",
"asdfghjkl",
"keyStringHere",
"firebase-adminsdk-dhr30#project-name.iam.gserviceaccount.com",
"1234567890",
"https://accounts.google.com/o/oauth2/auth",
"https://oauth2.googleapis.com/token",
"https://www.googleapis.com/oauth2/v1/certs",
"https://www.googleapis.com/robot/v1/metadata/x509/firebase-adminsdk-dhr30%40project-name-staging.iam.gserviceaccount.com"
)).byteInputStream())
val options = FirebaseOptions.Builder()
.setCredentials(credentials)
.setDatabaseUrl("https://project-name-staging.firebaseio.com")
.build()
FirebaseApp.initializeApp(options)
3. Initialize on your own server (Firebase console setup)
The only difference between #2 is this setup adds .setDatabaseUrl("https://yourProjectName.firebaseio.com").
4. Initialize cloud Firestore
FirestoreOptions firestoreOptions =
FirestoreOptions.getDefaultInstance().toBuilder()
.setProjectId(projectId)
.build();
Firestore db = firestoreOptions.getService();
Accessing Firestore Object
For #1-3 the Firebase app is initialized right away in the application's main() method. Then, the Firestore object is accessed from an object.
FirebaseClient.Kt
object FirebaseClient {
val firestore: Firestore
init {
firestore = FirestoreClient.getFirestore()
}
}
For #4 the Firestore object is created in the Kotlin object's init{...} and stored in the object as a value.
FirebaseClient.Kt
object FirebaseClient {
val firestore: Firestore
init {
val firestoreOptions = FirestoreOptions.getDefaultInstance().toBuilder()
.setTimestampsInSnapshotsEnabled(true)
.setProjectId("project-name")
.build()
firestore = firestoreOptions.service
}
}
Writing to Firestore
FirebaseClient.firestore.collection(someCollection).document(someDocument).collection(anotherCollection).add(someObject)
After utilizing Firebase authentication for a different project I discovered this is not an issue with Firebase authentication but rather with the application's main method. Therefore the various implementations above of Firebase authentication will work as expected when deployed to AppEngine.
Solution
I was expecting the application's main method to run once the app is deployed to AppEngine, similar to how an application's main method is called when ran in IntelliJ. However I realized main is only called once the app's hosted route is called.
ie: https://[yourProjectName].appspot.com
I've created a new StackOverflow post in order to determine how to run a Ktor app's main method automatically once deployed.
Related
I'm trying to deploy one of my functions from firebase CLI (version 8.12.1) and it keeps failing.
The function hasn't changed in weeks, so I am a bit confused as to why it's failing now.
Error from the CLI
functions[http-api-(europe-west1)]: Deployment error.
Build failed: Build error details not available. Please check the logs at https://console.cloud.google.com/logs/viewer?project=&advancedFilter=resource.type%3Dbuild%0Aresource.labels.build_id%3Dfeb2697d-29b4-4ab7-9b84-90d9f847be42%0AlogName%3Dprojects%2Fvestico-dev%2Flogs%2Fcloudbuild
Logs from the cloud console
Step #3 - "restorer": Restoring data for "google.nodejs.functions-framework:functions-framework" from cache
Step #3 - "restorer": \u001b[31;1mERROR: \u001b[0mfailed to restore: restoring data: GET https://storage.googleapis.com/eu.artifacts..appspot.com/containers/images/sha256:484d08dfc6a8f356c34a86fa4440fedf86f4fc398967eea66e4aab4e9ee81e3d?access_token=REDACTED: unsupported status code 404; body: NoSuchKeyThe specified key does not exist.No such object: eu.artifacts..appspot.com/containers/images/sha256:484d08dfc6a8f356c34a86fa4440fedf86f4fc398967eea66e4aab4e9ee81e3d
Finished Step #3 - "restorer"
ERROR: build step 3 "eu.gcr.io/fn-img/buildpacks/nodejs10/builder:nodejs10_20201005_20_RC00" failed: step exited with non-zero status: 46
The interesting piece is probably the error from above:
<Error>
<Code>NoSuchKey</Code>
<Message>The specified key does not exist.</Message>
<Details>No such object: eu.artifacts.<project-id>.appspot.com/containers/images/sha256:484d08dfc6a8f356c34a86fa4440fedf86f4fc398967eea66e4aab4e9ee81e3d</Details>
</Error>
What key is builder referring to? <project-id>#appspot.gserviceaccount.com has accesss to the cloud function with roles Cloud Functions Admin and Editor.
EDIT
Deploying through firebase deploy --only functions:<my-api>
That function uses #google-cloud/storage to get a public url for a storage resource.
I'm loading the service account configs like this:
const devServiceAccount = require("../../service-accounts/dev.json");
const prodServiceAccount = require("../../service-accounts/prod.json");
export const getAdminConfig = (): (AppOptions | undefined) => {
const baseConfigEnv = process.env.FIREBASE_CONFIG;
if (!baseConfigEnv) {
console.error("no firebase config environment");
return undefined;
}
const app = functions.config().app;
if (app === undefined) {
console.error("no firebase app config");
return undefined;
}
const serviceAccount = app.environment === 'dev' ? devServiceAccount : prodServiceAccount;
const adminConfig = JSON.parse(baseConfigEnv) as AppOptions;
adminConfig.credential = credential.cert(serviceAccount);
return adminConfig;
}
The cloud storage is used here.
const options = {
action: 'read',
expires: Date.now() + 1000 * 60 * 60 //1 hour
} as GetSignedUrlConfig;
const file = bucket.file(path);
filePathPromises.push(file.getSignedUrl(options))
});
My folder structure is as follows.
+ functions
+ lib
+ function.js
+ service-accounts
+ dev.json
+ prod.json
+ src
+ function.ts
I was ruling out that the service account files are the issue given that the files are loaded in getAdminConfig() for all functions in the project.
Update 10/13/20
I've verified the files uploaded to the GCF storage container. The JSON keys are there and in the right location. The paths match, so they should be found when the GCF is running.
Adding a hint for the next soul running into this problem. It seems to be caused by missing/inaccessible file in the restore/rollback process.
I was successfully removing the problem by simply:
Deleting my functions using the web firebase console.
Deploying normally again >firebase deploy
It seems there was an intermittent issue in Firebase Cloud Functions or GCF. I just ran firebase deploy --only functions again and it deployed successfully.
I am new to Spring cloud contract testing. I am to publish the stub jar to Jfrog artifactory from the producer side.
Following Gradle code:
publishing {
publications {
maven(MavenPublication) {
artifact("build/libs/provider-service-$version"+"-stubs.jar") {
extension 'jar'
}}}
repositories {
maven {
name 'libs-snapshot'
url "http://localhost:8081/artifactory/libs-snapshot/"
credentials {
username project.repoUser
password project.repoPassword
}
}
}
}
But on the consumer side I am not able to read the jar. I am getting the following error to read the jar.
Code:
#AutoConfigureStubRunner(ids = "com.test:provider-service:+:stubs:8082",
consumerName = "contracts",
properties = {"stubrunner.username=admin", "stubrunner.password=Cirrus123$"},
stubsPerConsumer = true,
stubsMode = StubRunnerProperties.StubsMode.REMOTE,
repositoryRoot = "http://localhost:8081/artifactory/libs-snapshot/")
Error:
java.lang.IllegalArgumentException: For groupId [com.test] artifactId [provider-service] and classifier [stubs] the version was not resolved! The following exceptions took place [org.eclipse.aether.transfer.MetadataNotFoundException: Could not find metadata com.test:provider-service/maven-metadata.xml in local (C:\Users\test\AppData\Local\Temp\aether-local7525112400154924089), org.eclipse.aether.transfer.MetadataTransferException: Could not transfer metadata com.test:provider-service/maven-metadata.xml from/to remote0 (http://localhost:8081/artifactory/libs-snapshot/): status code: 401, reason phrase: Unauthorized (401)]
But I am using the correct credentials to connect to Jfrog
After upgrading to com.crashlytics.sdk.android:crashlytics:2.7.1#aar (from 2.6.8), I can't disable Crashlytics anymore in my Firebase app.
Looks like there's some code in Crashlytics library itself that initializes Fabric with Crashlytics kit enabled whenever it detects that it's running inside a Firebase application. Indeed initializing with Crashlytics enabled and with ext.enableCrashlytics = false throws an UnmetDependencyException and crashes the app at startup (in fact, before my code in Application.onCreate runs).
Does anyone know a workaround for that? Sticking with 2.6.8 works for now.
This is what I have in my code that used to work until an upgrade:
app/build.gradle:
ext.enableCrashlytics = false
Application.java (onCreate, full method body as requested):
super.onCreate();
if (LeakCanary.isInAnalyzerProcess(this)) {
return;
}
LeakCanary.install(this);
// First Fabric invocation
Fabric.with(this, new Crashlytics.Builder().core(
new CrashlyticsCore.Builder().disabled(true).build()).build());
RxJavaPlugins.setErrorHandler(e -> LOGGER.error("Undeliverable RxJava error", e));
// First Firebase invocation
FirebaseDatabase db = FirebaseDatabase.getInstance();
if (BuildConfig.DEBUG) {
db.setLogLevel(com.google.firebase.database.Logger.Level.DEBUG);
}
db.setPersistenceEnabled(true);
according to mike answer, im add my code:
Gradle:
buildTypes {
release {
manifestPlaceholders = [crashlyticsEnabled: true]
}
debug {
manifestPlaceholders = [crashlyticsEnabled: false]
}
}
Manifest.xml:
<meta-data
android:name="firebase_crashlytics_collection_enabled"
android:value="${crashlyticsEnabled}" />
Mike from Fabric here. Use:
<meta-data android:name="firebase_crashlytics_collection_enabled" android:value="false" />
if you want to disable Crashlytics while using Firebase.
Along with mikes above answer,
If you are setting firebase crash properties somewhere in your code, make sure that you don't set them for debug code, otherwise you might notice strange behaviour for the app.
if (!BuildConfig.DEBUG) {
Crashlytics.setUserIdentifier(DataStore.storeId)
}
I just created a new deployment slot for my app, imported the publishing profile to Visual Studio, but after deployment I get this error message:
Error 8: An error occurred while creating the WebJob schedule: No website could be found which matches the WebSiteName [myapp__staging] and WebSiteUrl [http://myapp-staging.azurewebsites.net] supplied.
I have 2 webjobs, a continuous and a scheduled webjob.
I already signed in to the correct Azure account, as stated by this answer.
Will I need to set something else up in order to deploy my app to a staging Deployment Slot with webjobs?
My app is using ASP.NET, if it makes a difference?
There are a few quirks when using the Azure Scheduler. The recommendation is to use the new CRON support instead. You can learn more about it here and here.
Jeff,
As David suggested, you can/should migrate to the new CRON support. Here's an example. The WebJob will be deployed as a continuous WebJob.
Keep in mind that in order to use this you need to install the WebJobs package and extensions that are currently a prerelease. You can get them on Nuget.
Install-Package Microsoft.Azure.WebJobs -Pre
Install-Package Microsoft.Azure.WebJobs.Extensions -Pre
Also, as David suggested if you're not using the WebJobs SDK, you can also run this using a settings.job file. He provided an example here.
Program.cs
static void Main()
{
//Set up DI (In case you're using an IOC container)
var module = new CustomModule();
var kernel = new StandardKernel(module);
//Configure JobHost
var storageConnectionString = "your_connection_string";
var config = new JobHostConfiguration(storageConnectionString) { JobActivator = new JobActivator(kernel) };
config.UseTimers(); //Use this to use the CRON expression.
//Pass configuration to JobJost
var host = new JobHost(config);
// The following code ensures that the WebJob will be running continuously
host.RunAndBlock();
}
Function.cs
public class Functions
{
public void YourMethodName([TimerTrigger("00:05:00")] TimerInfo timerInfo, TextWriter log)
{
//This Job runs every 5 minutes.
//Do work here.
}
}
You can change the schedule in the TimerTrigger attribute.
UPDATE Added the webjob-publish-settings.json file
Here's an example of the webjob-publiss-settings.json
{
"$schema": "http://schemastore.org/schemas/json/webjob-publish-settings.json",
"webJobName": "YourWebJobName",
"startTime": null,
"endTime": null,
"jobRecurrenceFrequency": null,
"interval": null,
"runMode": "Continuous"
}
I am able to launch a local DynamoDB server from bash through this command:
java -Djava.library.path=./DynamoDBLocal_lib -jar DynamoDBLocal.jar -sharedDb &
Is there not a pure-java way to start the server in one's code? I don't mean a java callout to the shell through the Process object but a way such that when I run my app, the server starts, and when my app is killed, the server is killed.
I can live with an embedded database if such a mode exists, though something that reflects server consistency semantics would be ideal.
EDIT: September 23rd 2015
There was an announcement on Aug 3, 2015 that now adds the ability to have an embedded DynamoDB local running in the same process. You can add a Maven test dependency and use one of the ways below to run it.
<!--Dependency:-->
<dependencies>
<dependency>
<groupId>com.amazonaws</groupId>
<artifactId>DynamoDBLocal</artifactId>
<version>[1.11,2.0)</version>
</dependency>
</dependencies>
<!--Custom repository:-->
<repositories>
<repository>
<id>dynamodb-local-oregon</id>
<name>DynamoDB Local Release Repository</name>
<url>https://s3-us-west-2.amazonaws.com/dynamodb-local/release</url>
</repository>
</repositories>
And here is an example taken from the awslabs/aws-dynamodb-examples Github repository:
AmazonDynamoDB dynamodb = null;
try {
// Create an in-memory and in-process instance of DynamoDB Local that skips HTTP
dynamodb = DynamoDBEmbedded.create().amazonDynamoDB();
// use the DynamoDB API with DynamoDBEmbedded
listTables(dynamodb.listTables(), "DynamoDB Embedded");
} finally {
// Shutdown the thread pools in DynamoDB Local / Embedded
if(dynamodb != null) {
dynamodb.shutdown();
}
}
// Create an in-memory and in-process instance of DynamoDB Local that runs over HTTP
final String[] localArgs = { "-inMemory" };
DynamoDBProxyServer server = null;
try {
server = ServerRunner.createServerFromCommandLineArgs(localArgs);
server.start();
dynamodb = AmazonDynamoDBClientBuilder.standard().withEndpointConfiguration(
// we can use any region here
new AwsClientBuilder.EndpointConfiguration("http://localhost:8000", "us-west-2"))
.build();
// use the DynamoDB API over HTTP
listTables(dynamodb.listTables(), "DynamoDB Local over HTTP");
} finally {
// Stop the DynamoDB Local endpoint
if(server != null) {
server.stop();
}
}
Old answer
Like you said, there is currently no built-in way from DynamoDBLocal or the SDK to do this right now. It would be nice if there was an embedded DynamoDBLocal that you could start up in the same process.
Here is a simple workaround/solution using java.lang.Process to start it up and shut it down programmatically in case others are interested.
Documentation for DynamoDBLocal can be found here and here are the current definition of the arguments:
-inMemory — Run in memory, no file dump
-port 4000 — Communicate using port 4000.
-sharedDb — Use a single database file, instead of separate files for each credential and region
Note that this is using the most recent version of DynamoDBLocal as of August 5th, 2015.
final ProcessBuilder processBuilder = new ProcessBuilder("java",
"-Djava.library.path=./DynamoDBLocal_lib",
"-jar",
"DynamoDBLocal.jar",
"-sharedDb",
"-inMemory",
"-port",
"4000")
.inheritIO()
.directory(new File("/path/to/dynamo/db/local"));
final Process process = processBuilder.start();
Runtime.getRuntime().addShutdownHook(new Thread() {
#Override
public void run() {
System.out.println("Shutdown DynamoDBLocal");
process.destroy();
try {
process.waitFor(3, TimeUnit.SECONDS);
} catch (InterruptedException e) {
System.out.println("Process did not terminate after 3 seconds.");
}
System.out.println("DynamoDBLocal isAlive=" + process.isAlive());
}
});
// Do some stuff
Write a gradle task to extract the Dynamodb-Local zip and now you can use https://github.com/marcoVermeulen/gradle-spawn-plugin gradle plugin to launch the dynamodb local. It is very easy to use and no need to do any process builder magic.
Sample code -
// to start dynamodb-local
task launch(type: SpawnProcessTask) {
println("Launching....")
command "java -Djava.library.path=/location/to/dynamodb-local/DynamoDBLocal_lib -jar /location/to/dynamodb-local/DynamoDBLocal.jar -inMemory -delayTransientStatuses"
ready "Initializing DynamoDB Local"
}
// to stop dynamodb-local process
task stop(type: KillProcessTask)