Remote Jupyter Notebook/ Jupyter Lab cannot connect to Kernel without ssh tunnelling - jupyter-notebook

Question

I am getting a problem in opening remote Jupyter Lab/ Jupyter Notebook, it is not connecting to the kernel, I've mentioned the setup steps, problem coming and attempted fixed below. Any solutions, suggestions, or directions are greatly welcomed. Thanks in advance.
Setup Steps Followed:
I am using a fresh Anaconda 3 install in Ubuntu 20.04 LTS. I set up the Jupyter Notebook/ Jupyter Lab to be accessed remotely using the method provided in this answer (IP of that computer is 172.17.25.75), I've quoted the steps below:
Step 1. jupyter notebook --generate-config
Step 2. gedit /home/koushik/.jupyter/jupyter_notebook_config.py and edit the values c.NotebookApp.allow_origin = '*' and
c.NotebookApp.ip = '0.0.0.0
Step 3. Allow port sudo ufw allow 8888
Step 4. Set Password jupyter notebook password
Step 5. Start Jupyter Lab using jupyter lab --no-browser or Start Jupyter Notebook using jupyter notebook --no-browser
And access the Jupyter Lab/ Jupyter Notebook from my computer using
http://172.17.25.75:8888 in the browser.
I am behind my Institue proxy server. When I connect to Jupyter Lab or Jupyter Notebook using ssh tunneling ssh -N -f -L localhost:8889:localhost:8888 zuddler#172.17.25.75 and access localhost:8889 everything works fine. But when I am not tunneling and accessing them using IP, the following errors occur.
Problem with Jupyter Lab:
Upon running the command jupyter lab --no-browser below is the terminal output:
(base) zuddler#silplab:~$ jupyter lab --no-browser
[I 11:42:09.625 LabApp] JupyterLab extension loaded from /home/zuddler/anaconda3/lib/python3.8/site-packages/jupyterlab
[I 11:42:09.625 LabApp] JupyterLab application directory is /home/zuddler/anaconda3/share/jupyter/lab
[I 11:42:09.627 LabApp] Serving notebooks from local directory: /home/zuddler
[I 11:42:09.627 LabApp] Jupyter Notebook 6.1.4 is running at:
[I 11:42:09.627 LabApp] http://silplab:8888/
[I 11:42:09.627 LabApp] Use Control-C to stop this server and shut down all kernels (twice to skip confirmation).
[I 11:42:15.449 LabApp] Build is up to date
Upon opening http://172.17.25.75:8888 in the browser and starting a python 3 notebook, it asked to select a kernel
screenshot 1
upon selecting the kernel Python 3, it says
**Error Starting Kernel**
Invalid Response: 405 Method Not Allowed
screenshot 2
but a kernel was started and the terminal showed the below line
[I 11:46:47.547 LabApp] Kernel started: 572fa891-918c-486c-997a-96dd19651968, name: python3
which is also evident from the below screenshot
screenshot 3
upon selecting the marked kernel, it gives the same error mentioned in screenshot 2.
Upon executing jupyter kernelspec list, one kernel detail was returned
(base) zuddler#silplab:~$ jupyter kernelspec list
Available kernels:
python3 /home/zuddler/anaconda3/share/jupyter/kernels/python3
Problem with Jupyter Notebook:
Upon running the command jupyter notebook --no-browser below is the terminal output:
(base) zuddler#silplab:~$ jupyter notebook --no-browser
[I 11:55:20.472 NotebookApp] JupyterLab extension loaded from /home/zuddler/anaconda3/lib/python3.8/site-packages/jupyterlab
[I 11:55:20.472 NotebookApp] JupyterLab application directory is /home/zuddler/anaconda3/share/jupyter/lab
[I 11:55:20.474 NotebookApp] Serving notebooks from local directory: /home/zuddler
[I 11:55:20.474 NotebookApp] Jupyter Notebook 6.1.4 is running at:
[I 11:55:20.474 NotebookApp] http://silplab:8888/
[I 11:55:20.474 NotebookApp] Use Control-C to stop this server and shut down all kernels (twice to skip confirmation).
Upon opening http://172.17.25.75:8888 in the browser and starting the previously created python 3 notebook, it gave the error Kernel not found asked to select a kernel
screenshot 4
on clicking Set Kernel, it gave a connection failed error as below
screenshot 5
but again a kernel was started and the terminal showed the below line (similar to Jupyter Lab)
[I 11:55:42.447 NotebookApp] Kernel started: d6af7843-cf02-43ce-adc2-14a7659ee4ff, name: python3
Again, upon executing jupyter kernelspec list, one kernel detail was returned
(base) zuddler#silplab:~$ jupyter kernelspec list
Available kernels:
python3 /home/zuddler/anaconda3/share/jupyter/kernels/python3
Attempted Fixed
Try 1. Tried updating jupyter using conda update jupyter, it returned
(base) zuddler#silplab:~$ conda update jupyter
Collecting package metadata (current_repodata.json): done
Solving environment: done
# All requested packages already installed.
So, I tried to check if I have the lastest versions of conda and jupyter and it returned:
(base) zuddler#silplab:~$ jupyter --version
jupyter core : 4.6.3
jupyter-notebook : 6.1.4
qtconsole : 4.7.7
ipython : 7.19.0
ipykernel : 5.3.4
jupyter client : 6.1.7
jupyter lab : 2.2.6
nbconvert : 6.0.7
ipywidgets : 7.5.1
nbformat : 5.0.8
traitlets : 5.0.5
(base) zuddler#silplab:~$ conda --version
conda 4.9.2
To check if there is some path issue, I checked using which as mentioned here and it returned
(base) zuddler#silplab:~$ which jupyter-notebook
/home/zuddler/anaconda3/bin/jupyter-notebook
(base) zuddler#silplab:~$ which jupyter-lab
/home/zuddler/anaconda3/bin/jupyter-lab
(base) zuddler#silplab:~$ which python
/home/zuddler/anaconda3/bin/python
so, the paths were correct.
I also set c.NotebookApp.allow_remote_access = True in .jupyter/jupyter_notebook_config.py file (as mentioned here), but it did not fix the issue.
Try 2. To be sure that the kernel had the correct python path, I checked the kernel.json file, the path to python 3 was correct
(base) zuddler#silplab:~$ cat /home/zuddler/anaconda3/share/jupyter/kernels/python3/kernel.json
{
"argv": [
"/home/zuddler/anaconda3/bin/python",
"-m",
"ipykernel_launcher",
"-f",
"{connection_file}"
],
"display_name": "Python 3",
"language": "python"
}
I further tried to rename the kernel folder as mentioned in this jupyter-notebook doc
A path ending in ipykernel/resources, like in the example above, is this default kernel. The default often does what you want, so if the python3 kernelspec points somewhere else and you can’t start a Python kernel, try deleting or renaming that kernelspec folder to expose the default.
the executed commands and outputs are below, it revealed the default kernel:
(base) zuddler#silplab:~$ cd /home/zuddler/anaconda3/share/jupyter/kernels/
(base) zuddler#silplab:~/anaconda3/share/jupyter/kernels$ ls
python3
(base) zuddler#silplab:~/anaconda3/share/jupyter/kernels$ mv python3 python3_bak
(base) zuddler#silplab:~/anaconda3/share/jupyter/kernels$ cd ~
(base) zuddler#silplab:~$ jupyter kernelspec list
Available kernels:
python3 /home/zuddler/anaconda3/lib/python3.8/site-packages/ipykernel/resources
python3_bak /home/zuddler/anaconda3/share/jupyter/kernels/python3_bak
but I am getting the same errors.
Try 3. I also tried to follow the suggestions referenced in this question, but to no benefit.