Sequentially running jupyter notebooks with different conda environments - jupyter-notebook

I've got 10 jupyter notebooks, each with many unique package dependencies (that conflict), so I've created a different anaconda environment for each notebook. Each notebook relies on the output of the previous one, which I store and read from local csv files.
Right now I am running each jupyter notebook manually (with their own anaconda environment) to get the final result. Is there a way to run a single script that runs the code of all the jupyter notebooks sequentially (with the correct anaconda environment for each one)?


You could do it in python and use runipy. You just have to install it with:
pip install runipy
An example on how to use it from the docs:
from runipy.notebook_runner import NotebookRunner
from IPython.nbformat.current import read
notebook = read(open("MyNotebook.ipynb"), 'json')
r = NotebookRunner(notebook)
r.run_notebook()
If you want to run each notebook in a different environment, you can activate each conda environmentfrom a python script. There are multiple ways to do so, one of them is this:
subprocess.run('source activate environment-name && "enter command here" && source deactivate', shell=True)
Replace the "enter command here" with the command you want to run. You
don't need the "source deactivate" at the end of the command but it's
included just to be safe.
This will temporarily activate the Anaconda environment for the
duration of the subprocess call, after which the environment will
revert back to your original environment. This is useful for running
any commands you want in a temporary environment.

Related

How to change Jupyter start-up folder for different conda environments

After seeing this post on how to set the start-up folder for Jupyter Notebooks, I looked for how to do so for specific conda environments and haven't found an answer.
Is there a way to open up a Jupyter notebook in a location that is different depending on which conda environment within which you're activating it? I'm looking for a solution like the one above, where I could change c.NotebookApp.notebook_dir = '/the/path/to/home/folder/', but in some environment-specific config file.
I guess an alternative would be to set some macro to activate the environment, cd to the desired folder location for this environment, then run jupyter notebook from that location.

I was able to generate a DOSKEY macro to do the job. I combined this answer which shows how to set persistent aliases (macros) in command prompt, with this answer which shows how to use multiple separate commands in a DOSKEY macro. As a summary here (mostly from Argyll's answer in the above persistent macro/DOSKEY post):
Create a file called something like alias.cmd
Insert the macro to automatically activate a conda environment, change file locations, and run a jupyter notebook from that location:
doskey start_myEnv = conda activate myEnv $T cd C:\Users\user\path\to\my\notebooks\ $T jupyter notebook
Run regedit and go to HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Command Processor
or HKEY_CURRENT_USER\Software\Microsoft\Command Processor if not on Windows 10.
Add a String entry with the name AutoRun with the value set as the full path to the alias.cmd file.
Anytime you open the command prompt, executing start_myEnv will now activate myEnv, change to the folder that relates to that environment, and start a jupyter notebook.

JupyterLab not loading environment variables

I have some passwords and such set in my bash_profile that I want to be able to access from within Jupyter notebooks. They are successfully loaded whenever I use just a Jupyter notebook, but not when I use JupyterLab.
I am using Anaconda, and I double checked that the location of my JupyterLab and JupyterNotebook's are the same using which jupyter/jupyter-lab/jupyter-notebook and they do all point to my Anaconda bin directory. I also made sure the anaconda3/bin directory is in my bash profile and that the conda environment was activated.
Whenever I run os.environ in a notebook using Lab versus just a plain notebook I expect the same output, however the Lab instance does not load anything I have manually added to my profile while a plain notebook does.

Specifying default env for jupyter notebook

Is there some way to tell jupyter notebook what the default conda env should be when creating new notebooks? Launching it on AWS Deep Learning AMI's gives me a pretty long list, but I really only care about one specific env.

If you go to your terminal first and activate the virtual environment:
$ source venv/bin/activate
or
$ conda activate venv
for conda environment.
And after that step, do the following:
$ jupyter notebook
And when you make a new script it should give you the option for chosing python3/python2, chose the one that solves your purpose. And this script will be using the activated environment. You can check it by importing a libraray specific to that environment.

How to remove IPython kernel from Hydrogen for Atom

I have created multiple Conda envs which I use as IPython kernels in Hydrogen for Atom using the following suggestion:
source activate thisenv
python -m ipykernel install --user --name thisenv
After deleting such a Conda env, however, Atom-Hydrogen still gives me that kernel as an option to select from when compiling the code.
How does one unlink or remove a Conda env after it is linked as a kernel to Atom-Hydrogen?

The original command you ran was to register your env as a kernel, which on OS X results in creating a folder in a common area, like so
/Users/<user>/Library/Jupyter/kernels/thisenv
If you only want to deregister the environment (but not delete it), you can simply delete the thisenv folder from this directory (or wherever the equivalent folder is on other systems). It is not necessary to remove the environment from Conda.
If you're having trouble finding where the env is registered, you can use the kernelspecs package to locate all the available kernels. This is the package that Atom uses to find kernels.

jupyter notebook not running through command line [duplicate]

I have installed Jupyter on Windows 10, Python 3.x via
$ pip install jupyter
The installation works fine, even though I did restart the terminal.
But trying to run
$ jupyter notebook
gives the following error
'jupyter' is not recognized as an internal or external command, operable program or batch file.
How and where do I find the executable for Jupyter?

If you are absolutely sure that your Python library path is in your system variables (and you can find that path when you pip install Jupyter, you just have to read a bit) and you still experience "command not found or recognized" errors in Windows, you can try:
python -m notebook
For my Windows at least (Windows 10 Pro), having the python -m is the only way I can run my Python packages from command line without running into some sort of error
Fatal error in launcher: Unable to create process using ' "
or
Errno 'THIS_PROGRAM' not found

Please try either of these commands first;
$ py -m notebook
$ python -m notebook
for jupyterlab users
py -m jupyterlab
Otherwise
$ python -m pip install jupyter --user
$ jupyter notebook
If this does not work.
pip does not add jupyter directly to path for local.
The output from
$ which python
/c/Users/<username>/AppData/Local/Programs/Python/Python35-32/python
After some digging I found a executable for jupyter in the folder:
C:\Users\<username>\AppData\Roaming\Python\Python35\Scripts\jupyter.exe
Difference between local and roaming folder
So if you want to be able to execute a program via command line, you need to add it into the %PATH variable. Here is a powershell script to do it. BE SURE TO ADD THE ";" before adding the new path.
$ [Environment]::SetEnvironmentVariable("Path", $env:Path + ";C:\Users\<username>\AppData\Roaming\Python\Python35\Scripts", [EnvironmentVariableTarget]::User)
Update:
if you are using python3, switch out python with python3
but I encourage you to use pyenv instead :)

I had the exact same problem and it was driving me crazy. Other answers provide a solution, but they don't explain why you and I are having this problem.
I will try to explain why this is happening and then provide some solutions.
You can go to the end to see the TL;DR.
1)What's going on? Why is this error happening?
I'll try to make a step-by-step answer so everything is explained clearly.
If you think it's too basic at the beginning, go to the end of this "article".
I'll first start with common things like running the python shell from the terminal or running pip. You'll see why you can do that from the terminal and we'll end up on why and how you can run the jupyter notebook from the terminal as well.
Ready? Let's start!
Have you ever wondered why you can type python in the terminal (command prompt) and suddenly start the Python interpreter?
Microsoft Windows [Version 10.0.18363.1440]
(c) 2019 Microsoft Corporation. All rights reserved.
C:\Users\YOUR-USERNAME>python
Python 3.9.1 (tags/v3.9.1:1e5d33e, Dec 7 2020, 17:08:21) [MSC v.1927 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>>
You probably already know (but maybe don't) that this is because Python was added to the Windows PATH environment variable. You probably did it at installation time or afterwards.
But, what is this PATH environment variable?
It basically allows you to run any executables, that are located inside
the paths specified in the variable, at the command prompt without
having to give the full path to the executable.
You can check the content of that PATH variable with:
>>> import sys
>>> for path in sys.path:
print(path)
C:\Users\YOUR-USERNAME\AppData\Local\Programs\Python\Python39\python39.zip
C:\Users\YOUR-USERNAME\AppData\Local\Programs\Python\Python39\DLLs
C:\Users\YOUR-USERNAME\AppData\Local\Programs\Python\Python39\lib
C:\Users\YOUR-USERNAME\AppData\Local\Programs\Python\Python39
C:\Users\YOUR-USERNAME\AppData\Local\Programs\Python\Python39\lib\site-packages
... (some other paths were taken out for clarity)
You can see this folder: C:\Users\YOUR-USERNAME\AppData\Local\Programs\Python\Python39. This is the place where Python version 3.9 is installed. Let's check its content:
<DIR> DLLs
<DIR> Doc
<DIR> etc
<DIR> include
<DIR> Lib
<DIR> libs
<DIR> Scripts
<DIR> share
<DIR> tcl
<DIR> Tools
LICENSE.txt
NEWS.txt
python.exe
python3.dll
python39.dll
pythonw.exe
vcruntime140.dll
vcruntime140_1.dll
Voilà! We have the python.exe file (an executable). We have a Python executable file in the PATH, that's why you can start the Python interpreter from the terminal with just typing python. If this wasn't the case you would have to type the full path to the executable file in the terminal:
C:\Users\YOUR-USERNAME> C:\Users\YOUR-USERNAME\AppData\Local\Programs\Python\Python39\python)
Instead of just:
C:\Users\YOUR-USERNAME> python
And what about when you use pip?
It's the same principle. You can run pip from the terminal because there is a pip executable file in the PATH variable.
If you go to C:\Users\YOUR-USERNAME\AppData\Local\Programs\Python\Python39\Scripts\ (which is in the PATH showed above) you'll see many executables files. One of them is pip. Actually I have three versions: pip, pip3.9 and pip3.
The Scripts folder allows exectuable files to be run from the terminal. Like pip or other libraries that you intend to run directly from the terminal. The Scripts folder:
...is not intended for you, it's for scripts that are installed as
components of modules that you install. For example, pip is a module,
but it also has a wrapper script by the same name, pip, which will be
installed in that directory.
If you put something there and it is properly in your PATH, then it
should be executable
That wrapper script would be the pip executable file. When this executable file is run, it locates the pip folder in the Python installation folder and runs pip.
But you could also run pip directly from the installation folder (C:\Users\YOUR-USERNAME\AppData\Local\Programs\Python\Python39\Lib\site-packages), without needing the executable pip file.
But, how can you do it?
I'm glad you ask. There is a Python way to run modules as the main module (without the need to import it).
python -m pip
When you run a module directly its name becomes __main__. What -m does is:
Search sys.path for the named module and execute its contents as the __main__ module.
What is __main__?
'__main__' is the name of the scope in which top-level code executes.
A module’s __name__ is set equal to '__main__' when read from standard
input, a script, or from an interactive prompt.
...
I guess that the pip executable does something similar, or at least, has the same effect: to start pip.
2)What does this have to do with the Jupyter Notebook?!
Think of the Jupyter Notebook as the same as pip. If you want to run jupyter in the terminal, you need an executable that it's on the PATH.
We have already seen that the executables of modules like pip or jupyter are located here C:\Users\YOUR-USERNAME\AppData\Local\Programs\Python\Python39\Scripts\.
If I check the content of the folder I see this:
easy_install-3.9.exe
easy_install.exe
f2py.exe
jsonschema.exe
jupyter-bundlerextension.exe
jupyter-console.exe
jupyter-nbconvert.exe
jupyter-nbextension.exe
jupyter-notebook.exe
jupyter-qtconsole.exe
jupyter-serverextension.exe
jupyter-trust.exe
pip.exe
pip3.9.exe
pip3.exe
I see the already mentioned pip, pip3.9 and pip3. But I don't see jupyter (the word "jupyter" alone).
If I type jupyter in the terminal I get the error that started all:
'jupyter' is not recognized as an internal or external command, operable program or batch file.
Finally we've reached an answer to your question!!!
'jupyter' is not recognized as a command because there is no executable file in the Scripts folder called jupyter.
So, let's try a different executable. What about jupyter-notebook?
BINGO! The notebook is running!
Serving notebooks from local directory:
C:\Users\YOUR-USERNAME\AppData\Local\Programs\Python\Python39\Scripts
Jupyter Notebook 6.3.0 is running at:
http://localhost:8888/?token=... (edited)
or http://127.0.0.1:8888/?token=... (edited)
Use Control-C to stop this server and shut down all kernels (twice to skip confirmation).
I don't know why I don't have a jupyter executable called 'jupyter'. The official documentation says to use jupyter notebook on the terminal, but it seems that in some cases it doesn't work. And I think it has to do with what I mentioned above: there is no jupyter exectuable in the Scripts folder.
If you remember, I told you that you can run pip as the main module using python -m pip.
It happens that you can do the same with jupyter.We just need to know how to call it. As in with pip, we have to check the folder where 3rd party libraries are installed: C:\Users\YOUR-USERNAME\AppData\Local\Programs\Python\Python39\Lib\site-packages.
You'll see jupyter_console, but this just creates an interactive notebook in the terminal, not exactly what you were looking for. You're also going to find folders ending with .dist.info, like jupyter_console-6.4.0.dist-info. This is just metadata of the Wheel Binary Package builder. You'll also see a folder like jupyterlab_pygments, but that's for JupyterLab. We want to use the classic Jupyter notebook.
What we want is to run notebook. How do we know this?
You'll see in the folder site-packages the folder (package) notebook. Inside there is a file called __main__.py:
#__main__.py
if __name__ == '__main__':
from notebook import notebookapp as app
app.launch_new_instance()
It's calling notebookapp.py which is a "A tornado based Jupyter notebook server." Yes, this is what we need.
We can see that launch_new_instance in the notebookapp calls launch_instance(), which "launches an instance of a Jupyter Application".
Perfect! We are in the correct folder. To run the jupyter notebook from the Python interactive shell we have to run the notebook package with:
python -m notebook
3)*** SUMMARY: SOLUTION ***
tl;dr:
I have explained and showed why this error is happening.
Now let's summarize the solutions:
To know the name of the jupyter executable (in the Scripts folder), so you can run directly from the terminal (Command Prompt) as:
jupyter notebook
or as:
jupyter-notebook
Or whatever name you have.
Run the notebook as the main module from Python:
python -m notebook
I hope this helps you as much as it helped me. I'm open to your comments and suggestions.

I had the same problem, but
py -m notebook
worked for me.

I got Jupyter notebook running in Windows 10. I found the easiest way to accomplish this task without relying upon a distro like Anaconda was to use Cygwin.
In Cygwin install python2, python2-devel, python2-numpy, python2-pip, tcl, tcl-devel, (I have included a image below of all packages I installed) and any other python packages you want that are available. This is by far the easiest option.
Then run this command to just install jupyter notebook:
python -m pip install jupyter
Below is the actual commands I ran to add more libraries just in case others need this list too:
python -m pip install scipy
python -m pip install scikit-learn
python -m pip install sklearn
python -m pip install pandas
python -m pip install matplotlib
python -m pip install jupyter
If any of the above commands fail do not worry the solution is pretty simple most of the time. What you do is look at the build failure for whatever missing package / library.
Say it is showing a missing pyzmq then close Cygwin, re-open the installer, get to the package list screen, show "full" for all, then search for the name like zmq and install those libraries and re-try the above commands.
Using this approach it was fairly simple to eventually work through all the missing dependencies successfully.
Cygwin package list
Once everything is installed then run in Cygwin goto the folder you want to be the "root" for the notebook ui tree and type:
jupyter notebook
This will start up the notebook and show some output like below:
$ jupyter notebook
[I 19:05:30.459 NotebookApp] Serving notebooks from local directory:
[I 19:05:30.459 NotebookApp] 0 active kernels
[I 19:05:30.459 NotebookApp] The Jupyter Notebook is running at:
[I 19:05:30.459 NotebookApp] Use Control-C to stop this server and shut down all kernels (twice to skip confirmation).
Copy/paste this URL into your browser when you connect for the first time, to login with a token:
http://localhost:8888/?token=xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx

In Python 3.7.6 for Windows 10. After installation, I use these commands.
1. pip install notebook
2. python -m notebook
OR
C:\Users\Hamza\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.7_qbz5n2kfra8p0\LocalCache\local-packages\Python37\Scripts
.
For my pc python-scripts are located in the above path. You can add this path in environment variables. Then run command.
1. jupyter notebook

You can add the following to your path
C:\[Python Installation path]\Scripts
e.g. C:\Python27\Scripts
It will start working for jupyter and every other pip install you will do here on.

Check whether you have given python PATH in environmental variables properly.
If not, then set python path. Then use:
$ python -m notebook

If you are using the Anaconda distribution, make sure when installing it that you check the "Change PATH" option.

first you should make sure that you are put your python path in your system variables ..
Then try run this
python -m pip install jupyter --user
and then
run this
py -m notebook or jupyter notebook

In Windows 10 you can use ipython notebook. It works for me.

I added
c:\users\[user]\appdata\roaming\python\python37\site-packages
to the path and it worked.

## windows CMD
for default install (just check "add path" and "next" when installing)
python -m notebook
for custom install in C:\
jupyter notebook

Problem for me was that I was running the jupyter command from the wrong directory.
Once I navigated to the path containing the script, everything worked.
Path-
C:\Program Files (x86)\Microsoft Visual Studio\Shared\Python37_64\Scripts

I have two python version installed:
1. Python 3.8.2: This was installed independently
2. Python 3.7.6: This was installed along with Anaconda 3
Multiple versions caused conflict even after setting the path variables correctly.
I have uninstalled the Python 3.8.2 and after restart, the command
jupyter notebook
Worked perfectly :)

you can create a batch file and search for Jupiter in your windows search and ooen its properties and you will get this string.
D:\anaconda3\python.exe D:\anaconda3\cwp.py D:\anaconda3 D:\anaconda3\python.exe D:\anaconda3\Scripts\jupyter-notebook-script.py "%USERPROFILE%/"
after getting this you can create a jupitor.bat file with this content it that
and you can save that file in a script folder in d or any drive and add the path of your script file in your environmental path
and then you can easly call this by typing jupitor in cmd.

To install I used
"pip install notebook" in windows command line
To run python -m notebook did not work for me, but python3 -m notebook worked

First run this command
pip install jupyter
then add system variable path , this path is where jupyter and other scripts are located
PATH = C:\Users<userName>\AppData\Roaming\Python\Python38\Scripts
e.g PATH=C:\Users\HP\AppData\Roaming\Python\Python38\Scripts
After that we can run jupyter from any folder/directory
jupyter notebook

I just found that error when I first intalled and run the jupyter notebook. Then I found the executable (.exe) file from
C:\Users\<user-name>\AppData\Local\Programs\Python\Python39\Scripts.
The actual file name was "jupyter-notebook".
The installation guide says it as "jupyter notebook" to run the server. You have to run the command "jupyter-notebook" in the command line and it will be run. Thanks!

My problem was my user's folder had a space in folder name.
After creating a new user and switching to that windows user, windows shortcuts and links from within' Anaconda worked fine.
Windows 8.1 64 Bit. Latest Anaconda.
Note: I ended up uninstalling an reinstalling Anaconda but my sense is the problem was really just the space in the windows user username/user folder.

Using python 3.6.3. Here after installing Jupyter through command 'python -m pip install jupyter', 'jupyter notebook' command didn't work for me using windows command prompt.
But, finally 'python -m notebook' did work and made jupyter notebook to run on local.
http://localhost:8888/tree

Here is how I resolved stated issue, hope it helps:
install python 3.7 using official website for python, while installing include installing PATH by checking it's box
after that open cmd (be sure to open it after step 1) and write: pip install jupyter ENTER
now you should be able to open jupyter notebook by using command: jupyter notebook
Seems simple, but it may as well help.

In windows 10:
If you used anaconda3 for Jupyter notebook installation and forgot to check the box to add the environment variables to the system during installation, you need to add the following environment variables to the "Path" variable manually:
(search windows settings for Edit environment variables")
Environment variables for Anaconda3

If you have installed jupyter with
"python -m pip install jupyter" command
instead of
"$ pip install jupyter" command
then follow these steps:
Create a notepad
Change its extension from ".txt" to ".ipynb"
Right click it and click "open with"
In the pop up, go to - C:\Users\<"windows_user_name">\AppData\Roaming\Python\Python38\Scripts
Click on "jupyter-lab.exe"

100% working solution:
Follow these steps:
Open the folder where you downloaded "python-3.8.2-amd64.exe" setup or any other version of python package
Double click on "python-3.8.2-amd64.exe'
Click "Modify"
You will see "Optional features"
Click "next"
Select "Add python to environment variables"
Click "install"
Then u can run jupyter in any desired folder u desire
E.g open "cmd" command prompt
Type :
E:
E:\>jupyter notebook
It will get started without showing
'Jupyter' is not recognized
Thanks

I was facing the same issue in windows7, as i just recoverd my computer with the help of recovery point and after that notebook just stopped working. I tried to change the path setting but nothing was working so I just simply uninstalled the python with the application from which it was installed and after that I installed it again. After that I installed jupyter notebook again and then it worked fine.
Thanks

I just installed JupyterLab on top of my Python 3.8/pip enabled Win10 machine, so I ran into these startup problem with windows.
If everything is ok (check that you have PATH for Python, eg. C:\Users[Username]\AppData\Local\Programs\Python\Python38-32\Scripts) you simply run:
jupyter-lab.exe
and that's it.
Cheers

Add system variable path, this path is where jupyter and other scripts are located
PATH -->
`C:\Users\<userName>\AppData\Roaming\Python\Python39\Scripts`
Like in my laptop PATH is:
"C:\Users\developer\AppData\Roaming\Python\Python39\Scripts"
After that, You will be able to run jupyter from any folder & any directory by running the below command
jupyter notebook

Had the same issue. Finally searched where jupyter.exe directory was located on my computer. For some reason it was under
C:\VTRoot\HarddiskVolume4\Users[user]\AppData\Local\Programs\Python
Whereas the Python is
C:\Users[user]\AppData\Local\Programs\Python
So I copied full Python39 folder from VTRoot to main Python39 python folder in AppData. And the issue is solved.

First check where your python is installed by heading over to
C:\Users\**username**\AppData\Local\Programs\Python\Python39\Scripts
this is the path usually you will find python installed. After this type in
cd.. to get one step out of the Scripts folder
After this install jupyter in here using:
$ pip install jupyter
The installation was fine and when I tried to open the jupyter notebook, it gave me the following error.
'jupiter is not recognized as an internal or external command, operable program or batch file.
I checked a lot of resources but still, I faced the same problem. After doing a lot of stuff from the internet, I myself figured out that maybe due to some updates(for some users) it won't open by typing the command
jupyter notebook
Instead, you have to type
jupyter-notebook
The only thing I was missing was this Hyphen-minus. Hope this works for you too.

Resources