How can I run Jupyter notebooks disconnecting/reconnecting to its kernels without losing all the output from completed cells?
I know this has been asked several times and there are plenty of Q&A and web pages speaking about this, but I could not understand whether or not what I would like to do is possible and, if it is possible, what I am doing wrong. Many web pages and questions date back to several years ago, so, may be, such a basic functionality has been finally implemented.
I set up a remote jupyter using screen/tmux and ssh port forwarding. So I am able to connect to the remote machine using a web browser.
This is the notebook I run:
cell 1
import time
i = 0
while i < 10:
i += 1
print(i, flush=True)
time.sleep(5)
print("done")
cell 2:
print(i)
When I run the first cell, close the browser, reconnect to the kernel running the notebook above (Jupyter -> Running -> notebook) it seems that nothing is running. However, when I run the second cell, I realize the kernel is still running the first cell: there is no output for a while, then 10 is printed.
But:
When I reopen the notebook, there is no indication about which cell is currently running;
The output is lost. Not only the output produced after disconnecting from the kernel (closing the browser tab), but also all the output produced by the running cell between the save action and the disconnection.
This makes Jupyter completely useless when running (long) notebooks whose output needs to be monitored every once in a while.
I have this question. Without using tricks (like capturing output with some magic, writing the output to external log files, running the notebook as a python script, ...), is it possible to configure jupyter in order to disconnect/reconnect from/to its kernels running ipynb files using only the web interface and keeping the output produced while no browsers were attached to it?
So, im playing around with jupyter notebook, and I have this code, that i wrote yesterday:
BUt the problem is, that when i try to acces my model object the next day. Jupyter does not allow me to do that
For some reason i am not able to acces the variables in cells that i wrote i a previous session. Why is this? and how do I circumvent?
If you shut down the Jupyter Notebook service this is to be expected. Your output and cell numbers are saved in the notebook, but objects actually do not stay in memory. Notice how you have In [4]: in both screenshots. This is because you restarted the service and the second screenshot shows your 4th run of that cell since the restart.
You need to rerun the entire notebook to load everything into memory. You can use Kernel> Restart & Run All from the top menu to do this automatically.
I am trying to run a R notebook on Microsoft's Azure notebooks cloud service.
When I am trying to run all cells, it displays a Loading required package: ggplot2 in the last cell and then the Kernel systematically crashes. I get:
The kernel appears to have died. It will restart automatically.
But the Kernel does not restart automatically.
How can I get a log describing the encountered issue? Is there a way to activate a debugger?
When you're running Jupyter usually you'll see messages about kernel issues in standard I/O of the console that you launch. In Azure Notebooks this gets redirected to a file at ~/.nb.log. You can open a new terminal by clicking on the Jupyter icon, and then doing New->Terminal, and doing cat ~/.nb.log. You could also start a new Python notebook for this purpose and do "!cat ~/.nb.log" - but unfortunately you can't just do that from an R notebook, they don't support the "magic" ! commands.
Usually that gives you a good starting point. If that doesn't help much you could try invoking R directly from the terminal and trying the repro steps there and see if that's more useful.
I have been using Jupyter Notebook for a while. Often when I try to stop a cell execution, interrupting the kernel does not work. In this case, what else can I do, other than just closing the notebook and relaunching it again? I guess this might be a common situation for many people.
Currently this is an issue in the github jupyter repository as well,
https://github.com/ipython/ipython/issues/3400
there seems to be no exact solution for that except killing the kernel
If you're ok with losing all currently defined variables, then going to Kernel > Restart will stop execution without closing the notebook.
This worked for me:
- Put the laptop to sleep (one of the power options)
- Wait 10 s
- Wake up computer (with power button)
Kernel then says reconnecting and its either interrupted or you can press interrupt.
Probably isn't fool proof but worth a try so you don't waste previous computation time.
(I had Windows 10 running a Jupyter Notebook that wouldn't stop running a piece of Selenium code)
There are a few options here:
Change the folder name of data:
Works if the cell is running already and pulling data from a particular folder. For example I had a for loop that when interrupted just moved to the next item in list it was processing.
Change the code in the cell to generate an error:
Works if the cell has not been run yet but is just in queue.
Restart Kernel:
If all else fails
Recently I also faced a similar issue.
Found out that there is an issue in Python https://github.com/ipython/ipython/issues/3400 and it was there for 6 some years and it has been resolved as of 1st March 2020.
One thing that might work is hitting interrupt a bunch of times. It's possible that a library you are using catches the interrupt signal and only stops after receiving the signal multiple times.
For example, when using sklearn's cross_val_score() I found that I have to interrupt once for each cross validation fold.
If you know in advance that you might want to stop without losing all your variables, the following solution might be useful:
In cells that take a while because of long loops, you may implement something like this in the loop:
if os.path.exists(os.path.join(os.getcwd(),'stop_true.txt')):
break
Then if you want to stop just create the file 'stop_true.txt'. And the loop stops before the next round.
Usually, the file is called 'stop_false.txt' until I rename it to stop the loop.
Additionally, the results of each loop are stored in a dictionary separately. Therefore I'm able to keep all results until the break happened and can restart the loop from this point onwards.
If the iPython kernel did not die, you might be able to inject Python code into it that saves important data using pyrasite. You need to install and run pyrasite as root, i.e. with sudo python -m pip install pyrasite or python3 as needed. Then you need to figure out the process id (PID) of the iPython kernel (e.g. via htop or ps aux | grep ipython), say 3873. Then, write a script that saves the state for example to a pickle in a file inject.py, say, it is a Pandas dataframe df in the global scope:
df.to_pickle("rescued_df.pkl")
Finally, and inject it into the process as follows:
sudo pyrasite 3873 inject.py
You may need to enable dtrace first like so:
echo 0 | sudo tee /proc/sys/kernel/yama/ptrace_scope
For me, setting up a time limit worked: https://github.com/scipopt/PySCIPOpt/issues/197. Specifically, I added "model.setRealParam("limits/time", 60)" piece of code and it automatically stops calculation after 60 seconds. You can set up any time instead of 60. But this is for pyscipopt package (solving optimization model). I am not sure how to set up the time limit for your specific problem.
Try this:
Close the browser tab in which Jupyter is running
Run jupyter server list
Kill each running server with jupyter server stop <PORT>
You can force the termination by deleting the cell. I copy the code, delete the cell, create a new cell, paste, and execute again. Works like a charm.
I suggest to restart the kernel (Kernel -> Restart Kernel) as suggested by #hamdog.
It will be ready to use after that. However, it will certainly delete all variables stored in memory.
When I start the tomcat server from console using the startup.bat script, a new command window opens which is filled with java logging statements.
I use Console2 which leverages tabs for each open console window. Is it possible to let the java system create a new tab within console2 instead of just opening a new command window?
This has nothing to do with java, its merely down to the way the catalina.bat is called from startup.bat
catalina.bat can be called with either a "start" argument or a "run" argument.
run Start Catalina in the current window
start Start Catalina in a separate window
So open startup.bat, scroll to the bottom you should see
"%EXECUTABLE%" start %CMD_LINE_ARGS%
change that to
"%EXECUTABLE%" run %CMD_LINE_ARGS%
exit
I add an exit after to close the calling window.
Unless Console2 hooks any APIs that create console windows – no. And that's also highly unlikely that they do or even can. Console2 does nothing more than hook up input and output of console windows. What the programs in those do is beyond what it's interested in.