I have two GCP projects. A testing environment, built from scratch and a production environment with an already existing Cloud SQL for MySQL instance. My goal is to set a replication pipeline with Data Fusion to replicate some MySQL table.
On the testing environment I'm able to connect data fusion to MySQL. It is not working on the production environment. I have the following error:
Communications link failure The last packet sent successfully to the server was 0 milliseconds ago. The driver has not received any packets from the server.
This error message is pretty mysterious to me. The two environment are setup exactly the same way as far as I can see so I don't understand where is this error coming from. What can I do to better understand what's behind this error?
my setup is one Cloud SQL for MySQL instance on Private IP, on VM with proxy SQL, one Private Cloud fusion instance.
Related
I've been going around in circles trying to look for information on this, and all I find is connecting to Google Cloud SQL from Google Cloud Run - this is not what I want.
I have a PostgreSQL database installed on a server, outside of GCP. I have a R Shiny app, being deployed as a docker container thru Google Cloud Run, that needs access to the PostgreSQL database that lives outside of GCP.
When I run my R Shiny docker container on my local machine, using docker desktop, it works fine. The connection is made and I see no errors. I logged into my postgresql db directly with the username/password and I also could log in, no problem.
When I run my R Shiny docker container in Google Cloud Run, I get the following errors:
Warning: Error in postgresqlNewConnection: RS-DBI driver: (could not connect xxxx#xxpostgresxx.client.ext:5432 on dbname "db_name": could not translate host name "xxpostgresxx.client.ext" to address: Name or service not known
In my *.R file, the connection is written like this:
drv <- dbDriver("PostgreSQL")
con <- dbConnect(drv, host = "xxx.xxx.ext", port = "5432", dbname = "db_name", user = "xxxx", password = "xxxx")
Is this an issue with my external PostgreSQL db not allowing the Google Cloud Run url to access it's database? Is it something with the "RS-DBI driver" itself? Or is there something I need to setup somewhere in GCP to get this connection to work?
I've tried pulling in the DBI library before the RPostgreSQL library (someone mentioned it) but that didn't solve anything.
I am a newbie with Google Cloud Platform, so you'll probably have to give more detail in your explanations. Thank you so much in advance.
=====================================
UPDATE:
So in order to make this work, I had to set up a "Serverless VPC access" without networking people to allow my Cloud Run service to connect to resources within our private VPN network.
Cloud Run needs a public IP or DNS name by which to locate your Postgres database; it appears to be having problems using xxx.xxx.ext.
I'm unfamiliar with .ext. Is that a legitimate public Internet DNS name? If not, that's your (first) problem.
Corrollary: could any other entity on the Internet, use a Postgres client and suitable credentials connect to that Postgres database?
I have created an R Shiny app that connects to a Cloud SQL instance. It runs fine on my local server, but when I upload to either shinyapps.io or to Cloud Run via Dockerfile, it is unable to connect.
Here is the code I am using to connect using RPostgres package:
conn <- dbConnect(
drv=RPostgres::Postgres(),
dbname='postgres',
sslrootcert=path to 'server-ca.pem',
sslcert=path to 'client-cert.pem',
sslkey=path to 'client-key.pem',
host='xxxxxxxxxxxxxxxxxxx',
port=5432,
user='username',
password='password_string',
sslmode='verify-ca')
I've checked the logs in Cloud Run, the error message I am seeing is the following:
Warning: Error in : unable to find an inherited method for function 'dbGetQuery' for signature '"character", "character"'
The dbGetQuery() function is called after the dbConnect function, and since it runs fine on my local server, I am fairly confident that what I am seeing is a connection issue, rather than a package namespace issue. But could be wrong.
I have opened up to all IPs by adding 0.0.0.0/0 as an allowed network. The weird thing is that OCCASIONALLY I CAN CONNECT from shinyapps.io, but most of the time it fails. I have not yet got it to work once from Cloud Run. This is leading me to think that it could be a problem with a dynamic IP address or something similar?
Do I need to go through the Cloud Auth proxy to connect directly between Cloud Run and Cloud SQL? Or can I just connect via the dbConnect method above? I figured that 0.0.0.0/0 would also include Cloud Run IPs but I probably don't understand how it works well enough. Any explanations would be greatly appreciated.
Thanks very much!
I have opened up to all IPs by adding 0.0.0.0/0 as an allowed network.
From a security standpoint, this is a terrible, horrible, no good idea. It essentially means the entire world can attempt to connect to your database.
As #john-hanley stated in the comment, the Connecting Cloud Run to Cloud SQL documentation details how to connect. There are two options:
via Public IP (the internet) using the Unix domain socket on /cloudsql/CLOUD_SQL_CONNECTION_NAME
via Private IP, which connects through a VPC using the Serverless VPC Access
If a Unix domain socket is not supported by your library, you'll have to use a different library or choose Option 2 and connect over TCP. Note that Serverless VPC Access connector has additional costs associated with using it.
I am facing trouble when my app written using NextJS with PrismaIO as database ORM deployed to Vercel tries to connect with MYSQL database which is hosted on AWS RDS fails due to time out. It shows below error after a while:
502: BAD_GATEWAY
Code: NO_RESPONSE_FROM_FUNCTION
ID: bom1::zrmv2-1609789429213-86b5142a230c
I have added my app hosted at Vercel's IP address whitelisted in security group of AWS RDS too but the app still timesout and fails with 502 error page. Please help.
Thank you everyone this has turned frustrating for me and this way I can't push it to production with my RDS db exposed to all IPs so I am dropping this use-case and converting my NextJS app to CRA based UI which will be deployed to S3 so that RDS and S3 can have common security protocol. I have tested the RDS with EC2 sharing common security group and they connect really well and works out of the box. Thank you everyone once again.
I've tried two methods to connect my Shiny app to a BigQuery table as its source data:
Hadley's bigrquery, and
Mark Edmondson's BigQueryR
They're both failing the same way, so it's clearly a DFU error.
In each case, when I execute the appropriate command to establish the authorized connection (gar_auth_service(json_file = /path/,scope = 'https://www.googleapis.com/auth/bigquery' and bq_auth(path = /path/, respectively), I get this:
This site can’t be reached localhost refused to connect. Try:
Checking the connection Checking the proxy and the firewall
ERR_CONNECTION_REFUSED
This error comes after what appears to be a normal Google login process in the browser. The error page is hosted at localhost:1410, if that's any help.
In the Console, I have:
Created a VM instance (Ubuntu 19)
Successfully installed R, RStudio, and Shiny
Successfully logged in to RStudio in my GCP instance (from the browser, obviously, using the Externa IP I reserved in GCP)
I've also already created a BigQuery table in the same project, and successfully connected to it from an R script on my local machine.
I'm trying to get that same R script to run grom my Google Compute Engine instance.
Have I provided enough details to ask for help? If not, let me know what else I should provide. I'm walking through teaching myself GCP right now, and I'm quite the novice.
Thanks!
To bypass this issue, try connecting to your Ubuntu 19 instance using Chrome Remote Desktop on your Compute Engine instance as documented here.
Chrome Remote Desktop allows you to remotely access applications with a graphical user interface from a local computer instead of using the External IP. For this approach, you don't need to open firewall ports, and you use your Google Account for authentication and authorization. I've tried and I was able to connect both Shiny Server and to the RStudio.
I am struggling to find a way to register multiple gateways. I have a local instance of my SQL server and have created a gateway to access to it from the AML Studio workspace. It works fine but now I would like to access to the same SQL server instance from another workspace. So the question is: how to register a new gateway without removing the previous one?
I followed this documentation.
Does the following explanation mean that there is no way to do that?
You can create and set up multiple gateways in Studio for each workspace. For example, you may have a gateway that you want to connect to your test data sources during development, and a different gateway for your production data sources. Azure Machine Learning gives you the flexibility to set up multiple gateways depending upon your corporate environment. Currently you can’t share a gateway between workspaces and only one gateway can be installed on a single computer.
It is quite limiting as connecting to the same server from multiple workspaces may be sometimes crucial.
Well, finally I have found a way to bypass this limitation. From this documentation I have found that:
The IR does not need to be on the same machine as the data source. But staying closer to the data source reduces the time for the gateway to connect to the data source. We recommend that you install the IR on a machine that's different from the one that hosts the on-premises data source so that the gateway and data source don't compete for resources.
So the logic is pretty simple. You provide access to your local server to another machine on vpn and install your gateway there. Important: I have set up the firewall rules on the server before, to be able to establish the connection remotely.