AWS CodeDeploy No space left on device With PNPM Project - aws-code-deploy

Recently, I start NestJS project with PNPM.
every steps are very okay but when I deploy project to EC2 I got this AWS CodeDeploy Error
as you can see my EC2 storage got full.
this problem happened every CodeDeploy install event hook...
this is a CI/CD strategy.
CI with GitAction -> build -> compress tar.gz -> upload S3 -> call CodeDeploy -> PM2 Deploy
this image is CodeDeployAgent auto decompression tar file form deployment-root
when code deploy execute install event hook the node_modules file size are way bigger than before like this..
inside of node_modules every dependency reinstall they are dependencies.
ex) node_modules 1.5G
- .pnpm 1.5G
- lib 1
- node_modules 500M ..
- lib 2
- node_modules 500M ..
- lib 3
- node_modules 500M ..
...
this is part of GitAction workflow
...
build:
name: 🪜 Build & S3 Upload
needs: dependency_install
runs-on: ubuntu-latest
strategy:
matrix:
pnpm-version: [ 7.x ]
node-version: [ 18.12.x ]
steps:
- uses: actions/checkout#v3
- name: Setup Environments
uses: ./.github/actions/setup
- name: Use pnpm ${{ matrix.pnpm-version }}
uses: pnpm/action-setup#v2
with:
version: ${{ matrix.pnpm-version }}
- name: Use Node ${{ matrix.node-version }}
uses: actions/setup-node#v3
with:
node-version: ${{ matrix.node-version }}
cache: pnpm
- name: Check dependency cache
uses: actions/cache#v3
with:
path: |
${{ github.workspace }}/node_modules
${{ github.workspace }}/common/*/node_modules
${{ github.workspace }}/apps/${{env.SERVICE}}/node_modules
key: ${{ needs.dependency_install.outputs.dependency_cache_key }}
- name: Build
run: pnpm build:${{ env.SERVICE }}
- name: Remove DevDependencies
run: pnpm install --production
- name: AWS access registration
uses: aws-actions/configure-aws-credentials#v1-node16
with:
aws-access-key-id: ${{ secrets.AWS_IAM_MANAGE_KEY_ID }}
aws-secret-access-key: ${{ secrets.AWS_IAM_MANAGE_SECRET_ACCESS_KEY }}
aws-region: ap-northeast-2
- name: add .env
run: aws s3 cp s3://${{env.BUCKET}}/env/.env.${{ env.TARGET_ENVIRONMENT }} ./
- name: tar compress
run: tar -cpzf ./build.tgz --files-from='tar_include.txt' apps/${{ env.SERVICE }}/ecosystem.config.js apps/${{ env.SERVICE }}/dist apps/${{ env.SERVICE }}/node_modules apps/${{ env.SERVICE }}/package.json .env.*
shell: bash
- name: S3 Upload
run: aws s3 cp --region ap-northeast-2 ./build.tgz s3://${{env.BUCKET}}/${{env.TARGET_ENVIRONMENT}}/build.tgz
deploy:
name: 📤 Deploy
needs: [test, build]
runs-on: ubuntu-latest
steps:
- uses: actions/checkout#v3
- name: Setup Environments
uses: ./.github/actions/setup
- name: AWS access registration
uses: aws-actions/configure-aws-credentials#v1-node16
with:
aws-access-key-id: ${{ secrets.AWS_IAM_MANAGE_KEY_ID }}
aws-secret-access-key: ${{ secrets.AWS_IAM_MANAGE_SECRET_ACCESS_KEY }}
aws-region: ap-northeast-2
- name: Code Deploy
run: aws deploy create-deployment --application-name $CODE_DEPLOY_APP_NAME --file-exists-behavior OVERWRITE --deployment-config-name CodeDeployDefault.OneAtATime --deployment-group-name ${{env.BUCKET}}-${{env.TARGET_ENVIRONMENT}} --s3-location bucket=$BUCKET,bundleType=tgz,key=${{env.TARGET_ENVIRONMENT}}/build.tgz
...
appspec.yml
version: 0.0
os: linux
files:
- source: /
destination: /home/ubuntu/app/my-project
overwrite: yes
permissions:
- object: /home/ubuntu/app/my-project
owner: ubuntu
group: ubuntu
mode: 755
hooks:
BeforeInstall:
- location: scripts/beforeInstall.sh
timeout: 120
runas: ubuntu
AfterInstall:
- location: scripts/deploy.sh
timeout: 180
runas: ubuntu
beforeInstall.sh
#!/bin/bash
REPOSITORY=/home/ubuntu/app/my-project
if [ -d $REPOSITORY ]; then
rm -rf $REPOSITORY
fi
mkdir -vp $REPOSITORY
how come node_modules storage size bigger and is there any solution to deploy project?
thank you!

Related

NextJs firebase hosting build on github actions not reading environment variables

I am deploying a NextJs application on Firebase hosting using Github Actions.
I have the following workflow file:
name: Deploy to Firebase Hosting on merge
on:
push:
branches:
- main
jobs:
build-and-deploy-hosting:
runs-on: ubuntu-latest
defaults:
run:
working-directory: hosting
steps:
- name: Check out latest version of the code
uses: actions/checkout#v2
- name: Install Node.js and NPM
uses: actions/setup-node#v2
with:
node-version: "14"
- name: Make envfile
uses: SpicyPizza/create-envfile#v1
with:
envkey_NEXT_PUBLIC_FIREBASE_API_KEY: '${{ secrets.NEXT_PUBLIC_FIREBASE_API_KEY }}'
envkey_NEXT_PUBLIC_FIREBASE_PROJECT_ID: '${{ secrets.NEXT_PUBLIC_FIREBASE_PROJECT_ID }}'
envkey_NEXT_PUBLIC_FIREBASE_AUTH_DOMAIN: '${{ secrets.NEXT_PUBLIC_FIREBASE_AUTH_DOMAIN }}'
envkey_NEXT_PUBLIC_FIREBASE_STORAGE_BUCKET: '${{ secrets.NEXT_PUBLIC_FIREBASE_STORAGE_BUCKET }}'
envkey_NEXT_PUBLIC_FIREBASE_MESSAGING_SENDER_ID: '${{ secrets.NEXT_PUBLIC_FIREBASE_MESSAGING_SENDER_ID }}'
envkey_NEXT_PUBLIC_DEVELOPMENT: false
file_name: .env.local
directory: ./
- run: npm ci
- run: npm run build
- run: npm run export
- uses: FirebaseExtended/action-hosting-deploy#v0
with:
repoToken: '${{ secrets.GITHUB_TOKEN }}'
firebaseServiceAccount: '${{ secrets.FIREBASE_SERVICE_ACCOUNTxxx }}'
channelId: live
projectId: xxxx
env:
FIREBASE_CLI_PREVIEWS: hostingchannels
Typically, NextJs looks for an .env.local file with the environment variables placed within it. So I added these as secrets in the Github repo and then reference them in the build file by making an .env.local file.
But it's never recognized because when the Action runs, I get the following error:
> Build error occurred
[Error: Your API key is invalid, please check you have copied it correctly.] ***
type: 't',
code: 'auth/invalid-api-key',
a: null
***
npm ERR! code ELIFECYCLE
npm ERR! errno 1
npm ERR! xxx#0.1.0 build: `next build`
npm ERR! Exit status 1
npm ERR!
I used the ENV context referenced in the GitHub docs, which seems more straightforward.
This Stack Overflow answer solved my final issue, the ENV context needs to be added at the top level above jobs.
for example:
name: Deploy to Firebase Hosting on merge
'on':
push:
branches:
- main
env:
NEXT_PUBLIC_FIREBASE_PUBLIC_API_KEY: ${{secrets.NEXT_PUBLIC_FIREBASE_PUBLIC_API_KEY}}
jobs:
build_and_deploy:
runs-on: ubuntu-latest
Pretty sure this is triggered by using quotes around the mustache templates (the ${{ secrets.SOME_NAME }} bits) which causes them to be interpreted literally rather than get replaced.
name: Deploy to Firebase Hosting on merge
on:
push:
branches:
- main
jobs:
build-and-deploy-hosting:
runs-on: ubuntu-latest
defaults:
run:
working-directory: hosting
steps:
- name: Check out latest version of the code
uses: actions/checkout#v2
- name: Install Node.js and NPM
uses: actions/setup-node#v2
with:
node-version: "14"
- name: Make envfile
uses: SpicyPizza/create-envfile#v1
with:
envkey_NEXT_PUBLIC_FIREBASE_API_KEY: ${{ secrets.NEXT_PUBLIC_FIREBASE_API_KEY }}
envkey_NEXT_PUBLIC_FIREBASE_PROJECT_ID: ${{ secrets.NEXT_PUBLIC_FIREBASE_PROJECT_ID }}
envkey_NEXT_PUBLIC_FIREBASE_AUTH_DOMAIN: ${{ secrets.NEXT_PUBLIC_FIREBASE_AUTH_DOMAIN }}
envkey_NEXT_PUBLIC_FIREBASE_STORAGE_BUCKET: ${{ secrets.NEXT_PUBLIC_FIREBASE_STORAGE_BUCKET }}
envkey_NEXT_PUBLIC_FIREBASE_MESSAGING_SENDER_ID: ${{ secrets.NEXT_PUBLIC_FIREBASE_MESSAGING_SENDER_ID }}
envkey_NEXT_PUBLIC_DEVELOPMENT: false
file_name: .env.local
directory: ./
- run: npm ci
- run: npm run build
- run: npm run export
- uses: FirebaseExtended/action-hosting-deploy#v0
with:
repoToken: ${{ secrets.GITHUB_TOKEN }}
firebaseServiceAccount: ${{ secrets.FIREBASE_SERVICE_ACCOUNTxxx }}
channelId: live
projectId: xxxx
env:
FIREBASE_CLI_PREVIEWS: hostingchannels

Firebase deployment on a subfolder in GitHub Workflows not triggering

I'm trying to setup workflow for my github repo and it seems that it is not triggering.
GitHub repo structure -
I want to only trigger the workflow on any commit to master if there is a change done in client folder. Below is the yml of the workflow. Please tell me where should I look into to fix this.
name: Firebase Deploy
on:
push:
branches:
- master
paths: -"client/**"
defaults:
run:
working-directory: client
jobs:
build:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout#v2
- uses: actions/setup-node#v2
with:
node-version: "12"
- run: npm ci
- run: CI=false npm run build
- name: GitHub Action for Firebase
uses: w9jds/firebase-action#v1.5.0
with:
args: deploy --only hosting
env:
FIREBASE_TOKEN: ${{ secrets.FIREBASE_TOKEN }}
Note - I've setup the FIREBASE_TOKEN in repo also.

Need help for CI and CD in Github Workflow for Firebase Web App using HUGO Framework

I'm using HUGO to build a website and connected it in Forestry.io for headless cms. Everything is in my GitHub repository.
I have this 2 files in workflows folder under .github folder: deploy.yml and integrate.yml
I always get this error:
13 Insufficient number of arguments or no entry found.
14 Alternatively, run 'webpack(-cli) --help' for usage info.
------deploy.yml and integrate.yml files------
name: Firebase Continuous Deployment
on:
push:
branches: [master]
jobs:
deploy:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout#master
- uses: actions/setup-node#master
with:
node-version: 10.16.0
- run: npm ci
- run: npm run build
- uses: w9jds/firebase-action#master
with:
args: deploy --only hosting
env:
FIREBASE_TOKEN: ${{ secrets.FIREBASE_TOKEN }}
name: Node Continuous Integration
on:
pull_request:
branches: [master]
jobs:
test_pull_request:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout#v2
- uses: actions/setup-node#v1
with:
node-version: 12
- run: npm ci
- run: npm test
- run: npm run build
Double-check with the w9jds/firebase-action GitHub Action instructions:
the build job is suppose to execute the npm commands, not the deploy job
the deploy job is supposed to call Firebase (w9jds/firebase-action#master) with paths parameters
That is:
steps:
- name: Checkout Repo
uses: actions/checkout#master
- name: Download Artifact
uses: actions/download-artifact#master
with:
name: dist
path: dist
Try with this:
jobs:
build:
name: Build
runs-on: ubuntu-latest
steps:
- name: Checkout Repo
uses: actions/checkout#master
- name: Install Dependencies
run: npm ci
- name: Build
run: npm run build
- name: Archive Production Artifact
uses: actions/upload-artifact#master
with:
name: dist
path: dist
deploy:
name: Deploy
needs: build
runs-on: ubuntu-latest
steps:
- name: Checkout Repo
uses: actions/checkout#master
- name: Download Artifact
uses: actions/download-artifact#master
with:
name: dist
path: dist
- name: Deploy to Firebase
uses: w9jds/firebase-action#master
with:
args: deploy --only hosting
env:
FIREBASE_TOKEN: ${{ secrets.FIREBASE_TOKEN }}

Deploy environment variables to Firebase hosting

I use Firebase in React and when initializing Firebase, I use environment variables fetched from my .env file with dotenv. I want to build and deploy my React app to Firebase hosting, I use GitHub Actions with the following .yml workflow file:
name: Deploy
on:
push:
branches:
- master
jobs:
build:
name: Build
runs-on: ubuntu-latest
steps:
- name: Checkout Repo
uses: actions/checkout#master
- name: Install Dependencies
run: npm install
- name: Build
run: npm run build
- name: Archive Production Artifact
uses: actions/upload-artifact#master
with:
name: public
path: public
deploy:
name: Deploy
needs: build
runs-on: ubuntu-latest
steps:
- name: Checkout Repo
uses: actions/checkout#master
- name: Download Artifact
uses: actions/download-artifact#master
with:
name: public
- name: Deploy
uses: w9jds/firebase-action#master
with:
args: deploy --only hosting
env:
FIREBASE_TOKEN: ${{ secrets.FIREBASE_TOKEN }}
However, my build step fails (i.e. npm run build) because the environment variables are not found. If I hardcode the values instead of using my environment variables, the workflow is successful.
Do I need to do add my environment variables to GitHub secrets similar to how I added FIREBASE_TOKEN in my deploy step and add them to the workflow to be something like:
name: Deploy
on:
push:
branches:
- master
jobs:
build:
name: Build
runs-on: ubuntu-latest
steps:
- name: Checkout Repo
uses: actions/checkout#master
- name: Install Dependencies
run: npm install
- name: Build
run: npm run build
env:
FIREBASE_API_KEY: ${{ secrets.FIREBASE_API_KEY }}
FIREBASE_AUTH_DOMAIN: ${{ secrets.FIREBASE_AUTH_DOMAIN }}
FIREBASE_DATABASE_URL: ${{ secrets.FIREBASE_DATABASE_URL }}
FIREBASE_PROJECT_ID: ${{ secrets.FIREBASE_PROJECT_ID }}
FIREBASE_STORAGE_BUCKET: ${{ secrets.FIREBASE_STORAGE_BUCKET }}
FIREBASE_MESSAGING_SENDER_ID: ${{ secrets.FIREBASE_MESSAGING_SENDER_ID }}
FIREBASE_APP_ID: ${{ secrets.FIREBASE_APP_ID }}
- name: Archive Production Artifact
uses: actions/upload-artifact#master
with:
name: public
path: public
deploy:
name: Deploy
needs: build
runs-on: ubuntu-latest
steps:
- name: Checkout Repo
uses: actions/checkout#master
- name: Download Artifact
uses: actions/download-artifact#master
with:
name: public
- name: Deploy
uses: w9jds/firebase-action#master
with:
args: deploy --only hosting
env:
FIREBASE_TOKEN: ${{ secrets.FIREBASE_TOKEN }}
I receive the same error, saying that it can't find the API token.
When I browse workflow files in other firebase-based React apps, I can't see anyone adding these environment variables like I have suggested. What is the proper way of handling environment variables in apps deployed to Firebase? Do they belong in the build or deploy step?
Your custom env variables must start with REACT_APP_. Any other variables except NODE_ENV will be ignored to avoid accidentally exposing a private key on the machine that could have the same name.
Please take a look at this article Adding Custom Environment Variables.
As an additional note you can also run the npm run build along with your env variables passed as parameters like this:
REACT_APP_FIREBASE_APIKEY=${{ secrets.REACT_APP_FIREBASE_APIKEY }} REACT_APP_FIREBASE_AUTHDOMAIN=${{ secrets.REACT_APP_FIREBASE_AUTHDOMAIN }} REACT_APP_FIREBASE_DBURL=${{ secrets.REACT_APP_FIREBASE_DBURL }} npm run build
Instead of adding a .env file you can easily add the secrets to github secrets.
Once the secrets are added to github secrets you can add them to env at the build process as follow in your workflow.
name: Firebase Deployment
on:
push:
branches:
- master
- feature/secrets_fix
env:
API_KEY: ${{ secrets.FIREBASE_API_KEY }}
APP_ID: ${{ secrets.FIREBASE_APP_ID }}
AUTH_DOMAIN: ${{ secrets.FIREBASE_AUTH_DOMAIN }}
MESSAGE_SENDER_ID: ${{ secrets.FIREBASE_MESSAGE_SENDER_ID }}
PROJECT_ID: ${{ secrets.FIREBASE_PROJECT_ID }}
STORAGE_BUCKET: ${{ secrets.FIREBASE_STORAGE_BUCKET }}
TOKEN: ${{ secrets.FIREBASE_TOKEN }}
jobs:
build:
name: Build
runs-on: ubuntu-latest
steps:
- name: Checkout Repo
uses: actions/checkout#v2.3.2
- name: Install Dependencies
run: npm install
- name: Build
env:
CI: false
run: npm run build
- name: Archive Production Artifact
uses: actions/upload-artifact#v2
with:
name: build
path: build
deploy:
name: Deploy
needs: build
runs-on: ubuntu-latest
steps:
- name: Checkout Repo
uses: actions/checkout#v2.3.2
- name: Download Artifact
uses: actions/download-artifact#v2
with:
name: build
path: build
- name: Deploy to Firebase
uses: w9jds/firebase-action#master
with:
args: deploy --only hosting
env:
FIREBASE_TOKEN: ${{ secrets.FIREBASE_TOKEN }}
secrets.${YOUR_GITHUB_SECRET_NAME}
Once they are mapped as above under env, you can easily retrieve them from react application using
process.env.YOUR_ENV_VARIABLE_NAME_HERE
Following is a example on firebaseConfig
const firebaseConfig = {
apiKey: process.env.API_KEY,
authDomain: process.env.AUTH_DOMAIN,
projectId: process.env.PROJECT_ID,
storageBucket: process.env.STORAGE_BUCKET,
messagingSenderId: process.env.MESSAGE_SENDER_ID,
appId: process.env.APP_ID
};
Hope this will help someone.

Firebase Deployment using Github Action

I'm trying to deploy my code using Github's actions to firebase.
I am receiving this error
success Installed "firebase-tools#7.11.0" with binaries:
- firebase
Done in 12.99s.
/home/runner/work/_temp/30a2b6cb-a097-4d73-ac92-5379d0cc6ccf.sh: line 2: firebase: command not found
my code for deploy is as following
deploy:
name: Deploy
needs: build
runs-on: ubuntu-latest
steps:
- name: Checkout Repo
uses: actions/checkout#master
- name: Download Artifact
uses: actions/download-artifact#v1
with:
name: dist
- name: yarn add firebase-tools
run: |
yarn global add firebase-tools
firebase deploy ${{ secrets.firebase_token }} --only hosting:**** --non-interactive
env:
PROJECT_ID: ****
how can I add firebase-tools globally?
I tried https://github.com/marketplace/actions/github-action-for-firebase
But getting Error
setting firebase project to ***
Now using project ***
=== Deploying to '***'...
i deploying hosting
✔ Deploy complete!
Project Console: *******************
Error: An unexpected error has occurred.
##[error]Docker run failed with exit code 2
Here is the code for this.
deploy:
name: Deploy
needs: build
runs-on: ubuntu-latest
steps:
- name: Checkout Repo
uses: actions/checkout#master
- name: Download Artifact
uses: actions/download-artifact#v1
with:
name: dist
- name: Deploy to Firebase
uses: w9jds/firebase-action#master
with:
args: deploy --only hosting:****
env:
FIREBASE_TOKEN: ${{ secrets.firebase_token }}
PROJECT_ID: ****
Docker run failed with exit code 2
That was reported in w9jds/firebase-action issue 17
For me it works when I remove "--only hosting:prod".
Probably it depends on the plan.
You can't keep --only hosting:prod unless you have your site setup for multi-domain hosting inside of your firebase.json file and one of the sites that you can deploy is called prod.
I would recommend to follow this way!
name: Build and deploy
on:
push:
branches: [ main ]
# Run workflow manually
workflow_dispatch:
jobs:
build:
name: Build & Deploy
runs-on: ubuntu-latest
steps:
- name: Checkout Repository
uses: actions/checkout#main
- name: Install dependencies
run: npm ci
- name: Build dependencies
run: npm run build
- name: Deploy to Firebase
uses: w9jds/firebase-action#master
with:
args: deploy --only hosting
env:
FIREBASE_TOKEN: ${{ secrets.FIREBASE_TOKEN }}
You can replace npm commands with your yarn commands, and it should work fine.

Resources