Cloud Function returning 403 response - firebase

Question

I changed the connection setting of one of my cloud functions to 'Allow Internal Traffic Only' setting.
I have my nodejs app running in the same project, same region as my cloud function. I removed 'allUser' access from my cloud function and added My-PROJECT-ID#appspot.gserviceaccount.com as Invoker of my cloud function.
But I am getting 403 error now when I am calling the function from my nodejs app. What can I do to fix this?
I followed this as guidance: here
------------------UPDATE----------------
Many thanks for explanation below. It has started making sense now. So My setup is as follows as of now:
Cloud function side:
I have added My-PROJECT-ID#appspot.gserviceaccount.com as function invoker and removed 'allUsers' as an invoker.
Under variables, networking and advanced settings I have clicked on 'Allow internal traffic only' and then under Egress settings I have added the connector which I created earlier with an IP 10.8.0.0. I have added my connector in the format : projects/PROJECT_ID/locations/REGION/connectors/CONNECTOR_NAME and selected Route all traffic through the VPC connector
App Engine (NODE js) side:
When I make a call to the function when it was publicly available, I was using the given hostname. Now my POST request looks like the following:
const optionsCFS = {
hostname: "10.8.0.0",//process.env.CLOUD_URL,
port: 443, //(tried 28 as well)
timeout: 5000,
path: process.env.CLOUD_ORDER_SAVE_PATH, // remaining path
method: 'POST',
headers: {
'Content-Type': 'application/application-json',
'Content-Length': CFSdata.length,
//'charset': 'utf-8'
}
}
console.log('Going to call CF ')
const orderReq = https.request(optionsCFS, resCFServer =>
{ // Do something })
I get Error 502 - Bad Gateway.

Answer 1

When you set the traffic to internal only, you say to the Cloud Functions (or cloud run, it's the same behavior):
Hey, accept only the traffic that comes from the VPC.
However, you don't say:
Hey make my service only reachable through private IP and no longer through public IP
The difference is important, because even if you set your Cloud Functions (or your Cloud Run) with an ingress mode Allow internal traffic only, the service is still exposed on the internet, still reachable publicly, but the gateway in front of your service (GFE I guess, Google Front End), perform an additional check: "Do you come from the VPC?"
This check is based on the traffic metadata only present in the internal Google Network (that's also means that the traffic stay in the Google Cloud backbone, to keep these metadata).
So, I continue my explanation.... When you set a serverless VPC connector to App Engine, you can only route the private traffic to the VPC connector, compliant with the RFC 1918.
However, as explained, the Cloud Functions, and the Cloud Run, service are reachable on the internet, not on a private IP (compliant with the RFC 1918). And thus, your App Engine traffic don't go through the serverless VPC connector, and can't be accepted as "internal" traffic during the ingress check.
With Cloud Functions and Cloud Run, you can set up the vpc-egress value to private-ranges-only (similar to the default behavior of App Engine, route only the IPs in the RFC 1918 ranges) or all. It's this latest mode that you need to use to call a internal only service from Cloud Functions or Cloud Run.