Slow Next.js builds on Gitlab CI - next.js

Question

We have a JS-based stack in our application - React with vast majority being a React Admin frontend, built on Next.js server, with Postgres, Prisma and Nexus on backend. I realize it's not a great use case for Next.js (React Admin basically puts entire application in a single "component" (root), so basically I have a giant index.tsx page instead of lots of smaller pages), but we've had quite terrible build times in Gitlab CI and I'd like to know if there's anything I can do about it.
We utilize custom gitlab-runners deployed on the company Kubernetes cluster. Our build job essentially looks like:
- docker login
- CACHE_IMAGE_NAME="$CI_REGISTRY_IMAGE/$CI_COMMIT_REF_SLUG:latest"
- SHA_IMAGE_NAME="$CI_REGISTRY_IMAGE/$CI_COMMIT_SHORT_SHA"
- docker pull $CACHE_IMAGE_NAME || true
- docker build
-t $CACHE_IMAGE_NAME
-t $SHA_IMAGE_NAME
--cache-from=$CACHE_IMAGE_NAME
--build-arg BUILDKIT_INLINE_CACHE=1
- docker push # both tags
And the Dockerfile for that is
FROM node:14-alpine
WORKDIR /app
RUN chown -R node:node /app
USER node
COPY package.json yarn.lock ./
ENV NODE_ENV=production
RUN yarn install --frozen-lockfile --production
COPY . .
# Prisma client generate
RUN npm run generate
ENV NEXT_TELEMETRY_DISABLED 1
RUN npm run build
ARG NODE_OPTIONS=--max-old-space-size=4096
ENV NODE_OPTIONS $NODE_OPTIONS
EXPOSE 3000
CMD ["yarn", "start"]
This built image is then deployed with Helm into our K8s with the premise that initial build is slower, but subsequent builds in the pipeline will be faster as they can utilize docker cache. This works fine for npm install (first run takes around 10 minutes to install, subsequent are cached), but next build is where hell breaks loose. The build times are around 10-20 minutes. I recently updated to Next.js 12.0.2 which ships with new Rust-based SWC compiler which is supposed to be up to 5 times faster, and it's actually even slower (16 minutes).
I must be doing something wrong, but can anyone point me in some direction? Unfortunately, React Admin cannot be split across several Next.js pages AFAIK, and rewriting it to not use the framework is not an option either. I've tried doing npm install and next build in the CI and copy that into the image, and store in the Gitlab cache, but that seems to just shift the time spent from installing/building into copying the massive directories in/out cache and into the image. I'd like to try caching the .next directory in between builds, maybe there is some kind of incremental build possible but I'm skeptical to say the least.

Answer 1

Well, there are different things that we can approach for making it faster.
You're using Prisma, but you're generating the client every time that you have a modification in any of the files, preventing the Docker cache to actually take care of that layer. If we take a look into the Prisma documentation, we need to generate the Prisma Client each time that there's a change on the Prisma schema, not in the TS/JS Code.
I will suppose you have your Prisma schematics under the prisma directory, but feel free to adapt my suppositions to the reality of your project:
ENV NODE_ENV=production
COPY package.json yarn.lock ./
RUN yarn install --frozen-lockfile --production
COPY prisma prisma
RUN npm run generate
You're using a huge image for your final container, which maybe doesn't have a significant impact on the build time, but it definitely has on the final size and the time required to load/download the image. I would recommend to migrate into a multi-stage solution like the following one:
ARG NODE_OPTIONS=--max-old-space-size=4096
FROM node:alpine AS dependencies
COPY package.json yarn.lock ./
RUN yarn install --frozen-lockfile --production
COPY prisma prisma
RUN npm run generate
FROM node:alpine AS build
WORKDIR /app
COPY . .
COPY --from=dependencies /app/node_modules ./node_modules
ENV NEXT_TELEMETRY_DISABLED 1
RUN npm run build
FROM node:alpine
ARG NODE_OPTIONS
ENV NODE_ENV production
ENV NODE_OPTIONS $NODE_OPTIONS
RUN addgroup -g 1001 -S nodejs
RUN adduser -S nextjs -u 1001
WORKDIR /app
COPY --from=build /app/public ./public
COPY --from=build /app/node_modules ./node_modules
COPY --from=build /app/package.json ./package.json
COPY --chown=nextjs:nodejs --from=build /app/.next ./.next
USER 1001
EXPOSE 3000
CMD ["yarn", "start"]
From another point of view, probably you could improve also the NextJS build by changing some tools and modifying the NextJS configuration. You should use locally the tool https://github.com/stephencookdev/speed-measure-webpack-plugin to analyze which is the culprit of that humongous build-time (which probably is something related to sass) and also take a look into the TerserPlugin and the IgnorePlugin.