Fly.io deployment - secret env variables undefined - next.js

I'm pretty new to coding (started in August) and will try to provide all the necessary info needed!
I am working on a blog for a friend in Next.js that has a ghost backend for her to work in (deployment on fly.io worked perfectly fine with a ready-made docker image) and an extra frontend page. We need the latter because she wants the blog posts to appear in a specific way. So we cannot use the provided templates by ghost.
The frontend website is the one, I could not deploy yet. It fails during the build phase and shows the following error:
#10 46.29 Error occurred prerendering page "/". Read more: https://nextjs.org/docs/messages/prerender-error
#10 46.29 TypeError: Failed to parse URL from undefined/ghost/api/v3/content/posts/?key=undefined&fields=title,slug,html&include=tags
#10 46.29 at Object.fetch (node:internal/deps/undici/undici:11118:11)
#10 46.29 at async getPosts (/app/.next/server/pages/index.js:33:17)
#10 46.29 at async getStaticProps (/app/.next/server/pages/index.js:38:19)
#10 46.29 at async renderToHTML (/app/node_modules/next/dist/server/render.js:385:20)
#10 46.29 at async /app/node_modules/next/dist/export/worker.js:277:36
#10 46.29 at async Span.traceAsyncFn (/app/node_modules/next/dist/trace/trace.js:79:20)
#10 46.29 info - Generating static pages (1/6)
#10 46.29 (node:114) ExperimentalWarning: The Fetch API is an experimental feature. This feature could change at any time
#10 46.29 (Use `node --trace-warnings ...` to show where the warning was created)
#10 46.36 info - Generating static pages (2/6)
#10 46.38 info - Generating static pages (4/6)
#10 46.42 info - Generating static pages (6/6)
#10 46.42
#10 46.43 > Build error occurred
So, the variables somehow don’t "reach" my fetch request: undefined/ghost/api/v3/content/posts/?key=undefined&fields=title,slug,html&include=tags. The associated function looks like this:
async function getPosts() {
const res = await fetch(
`${process.env.BLOG_URL}/ghost/api/v3/content/posts/?key=${process.env.CONTENT_API_KEY}&fields=title,slug,html&include=tags`,
).then((resp) => resp.json());
const posts = res.posts;
return posts;
}
My Dockerfile looks like this:
# Initialize builder layer
FROM node:18-alpine AS builder
ENV NODE_ENV production
# Install necessary tools
RUN apk add --no-cache libc6-compat yq --repository=https://dl-cdn.alpinelinux.org/alpine/edge/community
WORKDIR /app
# Copy the content of the project to the machine
COPY . .
RUN yq --inplace --output-format=json '.dependencies = .dependencies * (.devDependencies | to_entries | map(select(.key | test("^(typescript|#types/*|#upleveled/)"))) | from_entries)' package.json
RUN --mount=type=secret,id=BLOG_URL \
BLOG_URL="$(cat /run/secrets/BLOG_URL)"
RUN --mount=type=secret,id=CONTENT_API_KEY \
CONTENT_API_KEY="$(cat /run/secrets/CONTENT_API_KEY)"
RUN yarn install --frozen-lockfile
RUN yarn build
# Initialize runner layer
FROM node:18-alpine AS runner
ENV NODE_ENV production
# Copy built app
COPY --from=builder /app/.next ./.next
# Copy only necessary files to run the app (minimize production app size, improve performance)
COPY --from=builder /app/node_modules ./node_modules
COPY --from=builder /app/public ./public
COPY --from=builder /app/package.json ./
COPY --from=builder /app/.env.production ./
CMD ["yarn", "start"]
The build problem seems to have something to do with the environment variables, which I have tried to 1) set via flyctl secrets set BLOG_URL=xyz for runtime and 2) tried to set via flyctl deploy --build-secret BLOG_URL=xyz and tell docker to use it in my Dockerfile like so (also see above):
RUN --mount=type=secret,id=BLOG_URL \
BLOG_URL="$(cat /run/secrets/BLOG_URL)"
RUN --mount=type=secret,id=CONTENT_API_KEY \
CONTENT_API_KEY="$(cat /run/secrets/CONTENT_API_KEY)"
The (runtime) secrets exist when I run flyctl secrets list.
I don’t know what is going on and why neither the build secrets nor the runtime secrets change anything about my ‘undefined’ URL.
I appreciate any hints and help!

I could find a solution/a workaround with the help of a friend. The issue could be solved in a part of the code, that I did not post yet:
export const getServerSideProps = async () => {
const posts = await getPosts();
if (typeof posts === 'undefined') {
return {
props: {
error: 'Nothing to see here',
},
};
}
return {
props: { posts },
};
};
In this piece of code, I had used getStaticProps which needs the variables during build. By using getServerSideProps, the runtime env variables are enough.
So, now I do not use any docker secrets, neither during deployment nor in my Dockerfile, only runtime fly secrets.
🆒

Related

firebase emulator cannot refresh dynamic import Functions

Background
I use dynamic import for Firebase Functions to reduce cold start time based on this post: https://medium.com/firebase-developers/organize-cloud-functions-for-max-cold-start-performance-and-readability-with-typescript-and-9261ee8450f0
index.ts under functions folder has only one line
exports.consent = require("./myFunctionGroup");
index.ts under functions/myFunctionGroup
import * as functions from "firebase-functions";
export const helloWorld = functions.https.onCall(async (data, context) => {
return await (await import("./helloWorld")).default(data, context);
});
Content of helloWorld.ts under functions/myFunctionGroup
import * as functions from "firebase-functions";
export default async (
data: {},
context: functions.https.CallableContext
) => {
return "hello";
}
There is no problem when the emulator first launched as it return "hello" correctly. However, when I change to return to 'return "world";', even though the emulator seems to be aware of the change and prints ✔ functions: Loaded functions definitions from source:..., it still return the old value of hello. Maybe it has some kind of cache for the original imported module. When I restart process again, then it will then print "world".
Question:
Is there any setting to make the emulator aware my change and force to clear the Functions cache? I tried to use debugger and it realized my code has changed so that I can step through the new code. It's just not working for the Firebase emulator.
It was expected behavior,any changes/modification in the code after running the emulator will not be reflected. You can also refer to this documentation:
Note:Code changes you make during an active session are automatically
reloaded by the emulator. If your code needs to be transpiled
(TypeScript, React) make sure to do so before running the emulator.
You can run your transpiler in watch mode with commands like tsc -w to
transpile and reload code automatically as you save.
You can follow the below steps,As mentioned in this stackoverflow thread
After every change to the ts files, run "npm run build" to compile the code again.
Change "build": "tsc" to "build": "tsc -w" in package.json if you want to auto-compile after every change.
There is a bug raised for this at github which is still open you can track the updates there.

Next.js env file depending deployment stage

Let's say I am deploying Next.js in different env, for example.
local development
staging deployment
production deployment
Previously I hand used .env file with one of the framework where I could easily name the file like .env.local, .env.stage & .env.prod and when I run node app locally it would load .env.local, with export STAGE=stageframework would use.env.stage`, like wise fro productoin.
Is that such support in Next js where I can have different .env file for different stage. If it is supported then how would I specify which stage Next.js is running.
You can have different environments, just need to follow this
For production
// .env.production.local
// This file is for production and run with next start or npm run start
// NODE_ENV=production
For development
// .env.development.local
// This file is for development and run when next dev or npm run dev
// NODE_ENV=development
For tests
// .env.test.local
// This file is for tests with jest or cypress for example
// NODE_ENV=test
If you want know more about this, here is the complete info about environments in next js
Update:
If you want to run more environments without next standard, you can do this manually:
// package.json
"scripts": {
...
"dev:staging": "APP_ENV=staging next dev",
}
...
// next.config.js
require('dotenv-flow').config({
node_env: process.env.APP_ENV || process.env.NODE_ENV ||
'development',
});
const env = {};
Object.keys(process.env).forEach((key) => {
if (key.startsWith('NEXT_PUBLIC_')) {
env[key] = process.env[key];
}
});
module.exports = {
env,
};
And when you run npm run/yarn dev:staging your staging environment from .env.staging.local will be loaded, my reference is from flybayer and you can read more here

Importing "gitlab" package from MeteorJS returns empty object

I've installed gitlab in my Meteor project with meteor npm install --save gitlab and imported the package in the imports/api/foo.js file with all the following variations (the comment on the front is the log of the Gitlab object):
import Gitlab from 'gitlab'; // {}
import * as Gitlab from 'gitlab'; // { default: {}, [Symbol(__esModule)]: true }
import { Gitlab } from 'gitlab'; // undefined
const Gitlab = require('gitlab'); // {}
const Gitlab = require('gitlab/dist/es5'); // {}
const Gitlab = require('gitlab/dist/latest'); // {}
If I run just console.log(require('gitlab')) with NodeJS, I get the correct result.
How am I supposed to import 'gitlab' from a meteor application?
I tried to reproduce the issue with a clean Meteor 1.8.0.2 project and it is working fine for me:
/server/main.js:
import Gitlab from 'gitlab'
Meteor.startup(() => {
console.log(Gitlab) // [Function: Bundle]
const api = new Gitlab({
url: 'http://example.com', // Defaults to https://gitlab.com
token: 'abcdefghij123456' // Can be created in your profile.
})
console.log(api) // full API as in documentation
})
So what options do you have here?
Make sure you use gitlab on the server
Check the node_modules folder, whether it is really installed there.
Try to reset your project using meteor reset and then start again so all node_modules a rebuild and all Meteor packages are rebuild and the local dev build is rebuild. This will often fix things.
Create a fresh project and start to reproduce the issue step by step, starting from my working example and change the file structure stepwise to the structure of your project.

TestCafe integration with cucumber - test cases in github project time out

I have been experimenting with javascript frameworks for test automation and one of them is testCafe. I have been able to set up a simple TestCafe project and run some test cases for my application. However, now, the requirement is to have some kinda BDD support built in it. I looked up a few testCafe-cucumber integration projects on GitHub but I can't get them to run. here are a few that I tried:-
1) https://github.com/rquellh/testcafe-cucumber
- I cloned the repo,
- did npm install,
- run the test cases using "npm test",
- blank browser launches but test doesn't run. I see this error in VS code console:
× Before # features\support\hooks.js:46
Error: function timed out, ensure the promise resolves within 20000 milliseconds
at Timeout._onTimeout (C:\Users\Mo\Desktop\TestCafe\github\testCafeBDD\testcafe-cucumber\node_modules\cucumber\src\user_code_runner.js:61:18)
at ontimeout (timers.js:482:11)
at tryOnTimeout (timers.js:317:5)
at Timer.listOnTimeout (timers.js:277:5)
× After # features\support\hooks.js:60
ReferenceError: testController is not defined
Then I tried another gitHub project namely this one : https://github.com/kiwigrid/gherkin-testcafe
the run command in readme doesn't work for me, it doesn't even recognize the "gherkin-testcafe".
When I run my TestCafe test cases without the cucumber I have this line in my package.json
"scripts": {
"test": "testcafe chrome Tests/ -e --proxy https.proxy.mycompany.com:8000"
},
the proxy is mentioned because I am behind a proxy and without this the browser launches but does not run any test cases. I found this fix on testCafe site
I am guessing(not sure yet) this could be the issue with cucumber integration as well. None of these frameworks work as they don't set up the proxy anywhere. Can someone point me in the right direction? if the proxy needs to be set up then where in the framework does it need to go- an example would be helpful?
TestCafe/Cucumber integrations rely on starting TestCafe runner programmatically.
In the repo, search for this sequence:
const runner = tc.createRunner();
return runner
.src('./test.js')
.screenshots('reports/screenshots/', true)
.browsers(browser)
.run()
.catch(function(error) {
console.error(error);
});
or search for this sequence:
await runner
.browsers(browsers)
.specs(specs)
.steps(steps)
.concurrency(concurrency)
.startApp(app, appInitDelay)
.tags(tags)
.run(...)
Chain the useProxy method on runner object (do it before the run()method):
const runner = tc.createRunner();
return runner
.src('./test.js')
.screenshots('reports/screenshots/', true)
.browsers(browser)
.useProxy('username:password#proxy.mycorp.com')
.run()
.catch(function(error) {
console.error(error);
});

How to call a Meteor method from the command line

I have a Meteor app and want to call a server method from the command line, so that I can write a bash script to perform scheduled operations.
Is there any way to either call a method directly, or submit a form which will then trigger server-side code?
I've tried using curl to call a method, but either it's not possible or I'm missing something basic. This doesn't work:
curl "http://localhost:3000/Meteor.call('myMethod')"
nor does:
curl -s -d "http://localhost:3000/imports/api/test.js" > out.html
where test.js:
var test = function(){
console.log('hello');
}
I thought of using a form but I can't think how to create a submit event because the Meteor client uses template events that then call server methods.
I'll be very grateful for any help! This feels like it should be a simple thing but has me stumped.
Edit: I've also tried phantomjs and slimerjs as run through casperjs.
phantomjs is no longer maintained and generates an error:
TypeError: Attempting to change the setter of an unconfigurable property.
https://github.com/casperjs/casperjs/issues/1935
slimerjs errors with Firefox 60 and I can't figure out how to 'downgrade' back to the supported 59, and the option to disable automatic updates of Firefox no longer seems to exist. The error is:
c is undefined
https://github.com/laurentj/slimerjs/issues/694
You could make use of the node ddp package to call the Meteor method in an own js file that you created with a specific script. From there you can pipe all outs to wherever you want.
Let's assume the following Meteor method:
Meteor.methods({
'myMethod'() {
console.log("hello console")
return "hello result"
}
})
The upcoming steps will let you call this method from another shell, assuming your Meteor application is running.
1. Install ddp in your global npm directory
$ meteor npm install -g ddp
2. Create the script to call your method in your test directory
$ mkdir -p ddptest
$ cd ddptest
$ touch ddptest.js
Place the ddp script code into the file with the editor or command of your choice.
(The follwing code is freely taken from the package's readme. Feel free to configure to your needs.)
ddptest/ddptest.js
var DDPClient = require(process.env.DDP_PATH);
var ddpclient = new DDPClient({
// All properties optional, defaults shown
host : "localhost",
port : 3000,
ssl : false,
autoReconnect : true,
autoReconnectTimer : 500,
maintainCollections : true,
ddpVersion : '1', // ['1', 'pre2', 'pre1'] available
// uses the SockJs protocol to create the connection
// this still uses websockets, but allows to get the benefits
// from projects like meteorhacks:cluster
// (for load balancing and service discovery)
// do not use `path` option when you are using useSockJs
useSockJs: true,
// Use a full url instead of a set of `host`, `port` and `ssl`
// do not set `useSockJs` option if `url` is used
url: 'wss://example.com/websocket'
});
ddpclient.connect(function(error, wasReconnect) {
// If autoReconnect is true, this callback will be invoked each time
// a server connection is re-established
if (error) {
console.log('DDP connection error!');
console.error(error)
return;
}
if (wasReconnect) {
console.log('Reestablishment of a connection.');
}
console.log('connected!');
setTimeout(function () {
/*
* Call a Meteor Method
*/
ddpclient.call(
'myMethod', // namyMethodme of Meteor Method being called
['foo', 'bar'], // parameters to send to Meteor Method
function (err, result) { // callback which returns the method call results
console.log('called function, result: ' + result);
ddpclient.close();
},
function () { // callback which fires when server has finished
console.log('updated'); // sending any updated documents as a result of
console.log(ddpclient.collections.posts); // calling this method
}
);
}, 3000);
});
The code assumes that your app runs on localhost:3000, note that there is no conncection close on errors or undesired behavior.
As you can see at the top, the file imports your globally installed ddp package. Now in order to get it's path without using additional tools, just pass an environment variable (process.env.DDP_PATH) and let your shell handle the path resolving.
In order to get the installation path you can use npm root with the global flag.
Finally call your script via:
$ DDP_PATH=$(meteor npm root -g)/ddp meteor node ddptest.js
Which will give you the following output:
connected!
updated
undefined
called function, result: hello result
And logs hello console to the open session that is running your meteor app.
Edit: A note on using this in production
If you want to use this script in production you have to use the shell commands without the meteor command but using your installation of node and npm.
If you get in trouble with paths use process.execPath to find your node binary and npm root -g to find your global npm modules.
You can check out this documentation: Command Line | meteor shell.
While your meteor app is running, you can execute meteor shell to start an interactive console. In the console, you can do Meteor.call(...).
So if you want to write a script with using meteor shell, you might need to pipe the script file for meteor shell. Like,
$ meteor shell < script_file
See also the answer of "How can I pipe a command into the meteor shell?"

Resources