jupyter notebook reindex with pandas error - jupyter-notebook

I'm having a strange issue. I have a problem with reindexing when i call my script with a jupyter notebook but it work fine when i call it directly using pycharm.
The first time i execute the notebook after i just started jupyter notebook it work but then it never work again. And it give me this error :
ValueError: cannot reindex from a duplicate axis
I suspect a problem between pandas and jupyter notebook. Because this error never appear when i use pycharm.
Do you have any idea on how i can fix this problem so that i can call my script from a jupyter notebook ?
I'm using the same conda env for both the jupyter notebook and pycharm.

I fond that Jupiter-notebook does not re-import the modules every time you execute it.
I had a problem with a variable that was not overwrite I changed the constructor of my script to be able to rewrite it and it work fine now.

Related

Prevent jupyter console from reading code from jupyter notebook

When on jupyter console and I hit the up arrow key, I'd like to go to the previously run line of code on that console. However it will read a previously run cell from a jupyter notebook instead. Is there a way to prevent this from happening?

IOPub data rate exceeded in Jupyter notebook (when viewing image)

I want to view an image in Jupyter notebook. It's a 9.9MB .png file.
from IPython.display import Image
Image(filename='path_to_image/image.png')
I get the below error:
IOPub data rate exceeded.
The notebook server will temporarily stop sending output
to the client in order to avoid crashing it.
A bit surprising and reported elsewhere.
Is this expected and is there a simple solution?
(Error msg suggests changing limit in --NotebookApp.iopub_data_rate_limit.)
Try this:
jupyter notebook --NotebookApp.iopub_data_rate_limit=1.0e10
Or this:
yourTerminal:prompt> jupyter notebook --NotebookApp.iopub_data_rate_limit=1.0e10
I ran into this using networkx and bokeh
This works for me in Windows 7 (taken from here):
To create a jupyter_notebook_config.py file, with all the defaults commented out, you can use the following command line:
$ jupyter notebook --generate-config
Open the file and search for c.NotebookApp.iopub_data_rate_limit
Comment out the line c.NotebookApp.iopub_data_rate_limit = 1000000 and change it to a higher default rate. l used c.NotebookApp.iopub_data_rate_limit = 10000000
This unforgiving default config is popping up in a lot of places. See git issues:
jupyter
IOPub data rate exceeded
It looks like it might get resolved with the 5.1 release
Update:
Jupyter notebook is now on release 5.2.2. This problem should have been resolved. Upgrade using conda or pip.
Removing print statements can also fix the problem.
Apart from loading images, this error also happens when your code is printing continuously at a high rate, which is causing the error "IOPub data rate exceeded". E.g. if you have a print statement in a for loop somewhere that is being called over 1000 times.
By typing 'jupyter notebook --NotebookApp.iopub_data_rate_limit=1.0e10' in Anaconda PowerShell or prompt, the Jupyter notebook will open with the new configuration. Try now to run your query.
Some additional advice for Windows(10) users:
If you are using Anaconda Prompt/PowerShell for the first time, type "Anaconda" in the search field of your Windows task bar and you will see the suggested software.
Make sure to open the Anaconda prompt as administrator.
Always navigate to your user directory or the directory with your Jupyter Notebook files first before running the command. Otherwise you might end up somewhere in your system files and be confused by an unfamiliar file tree.
The correct way to open Jupyter notebook with new data limit from the Anaconda Prompt on my own Windows 10 PC is:
(base) C:\Users\mobarget\Google Drive\Jupyter Notebook>jupyter notebook --NotebookApp.iopub_data_rate_limit=1.0e10
I have the same problem in my Jupyter NB on Win 10 when querying from a MySQL database.
Removing any print statements solved my problem.
For already running docker containers, try editing the file name - ~/.jupyter/jupyter_notebook_config.py
uncomment the line - NotebookApp.iopub_data_rate_limit =
and set high number like 1e10.
Restart the docker, it should fix the problem
I ran into this problem running version 6.3.0. When I tried the top rated solution by Merlin the powershell prompt notified me that iopub_data_rate_limit has moved from NotebookApp to ServerApp. The solution still worked but wanted to mention the variation, especially as internal handling of the config may become deprecated.
Easy workaround is to create a for loop and print. Then there wont be any issue. Printing directly wcc would cause if graph is huge. Hence any of below code will work as workaround.
wcc=list(nx.weakly_connected_components(train_graph))
for i in range(1,10):
print(wcc[i])
for i in wcc):
print(wcc)
Like others pointed out, print statement at a high rate can cause this. Resolve it by printing modulo a number using if statement. Example in python:
k = 10
if (i % k == 0):
print("Something")
Increase k if the warning persists.
Using Visual Studio Code, the Jupyter extension will be able to handle big data. launch from anaconda navigator
In general, trying to print something that is too long will trigger this error. I tried to print a string that was 9221593 characters long (too long), and that triggered the error.

Jupyter Notebooks Hang in Browser on Windows 10

I just installed Miniconda and the R Essentials bundle on my Windows 10 machine, following the instructions given here. Everything went swimmingly until I opened up an Anaconda command prompt and entered jupyter notebook and got an error. I then used ipython notebook which worked, so okay, no problem there.
However, after creating a new folder and trying to create a new R notebook within that folder, my Jupyter tabs started to hang. Whenever I try to do something, whether it is rename the notebook, run a block of code, basically anything, all of the Jupyter tabs sit there loading endlessly saying "Waiting for localhost..."
I try stopping the server and restarting it, but every time I try to do anything I get the same result. I also tried changing the port and running the command prompt as administrator--same result. I am using Chrome, which shouldn't be an issue.
Any ideas? I was really excited about using a Jupyter notebook to keep track of my analyses in R, but if I can't even get it to function out of the box I'll have to find a better solution.

Kernel loading forever in jupyter notebook

I'm using OSX Yosemite.
I've update ipython via conda and turns out notebook also has updated version, which I'm very excited to try this.The notebook has been converted into jupyter.
I'm using python 2.X, and already have existing .ipynb files. When I open it, new window appears but there's blank and nothing. I can create first cell there, but I already have my notebook. And the kernel also loading forever. There isn't any error log in the console. What do I have to do? Please help!
Nevermind, this solve the problem https://github.com/ipython/ipython/issues/5746
I'm using ccp notebook extension, and as Ian Hawke mentioned in the thread, remove the call to the extention at profile/static/custom/custom.js

How to develop with ipython notebook

I am new to IPython Notebook. I am using the Anaconda distribution on CrunchBang (Waldorf). My development cycle is as follows:
1. Open Spyder.
2. Open the .py file if not already loaded
3. Start IPython Notebook
4. Open the specific notebook from the main IPython screen
5. Select Cell/Run All
6. Note errors. If none goto step 11.
7. Save and close the notebook
8. Shutdown the notebook from main IPython screen
9. Correct errors in Spyder and save
10. go to step 4
11. Move on to the next part of the project and start the process over.
Is there a better approach for a noob? This really gets monotonous although I am learning quite a bit.
Thanks in advance
Forget Spyder for the time being just use the IPython notebook.
1, write code in notebook
2. test it
3. when done if needed make a py file...
You really will only need Spyder later for starting out it just complicates things for no gain
Use Spyder and .py files for writing big functions, classes, modules, tests, etc.
Use IPython notebooks for interactive work where you want to keep the output together with the code (e.g. data processing and analysis, demos, etc.).
To add to Ian's answer, another useful tool is the autoreload extension, which reloads modules automatically when they are changed.
To use, type into your IPython console or notebook:
%load_ext autoreload
%autoreload 2
For example:
This way you can work on a Python file and an IPython notebook at the same time, without having to reload the Python file after each change.
In addition to #dartdog's answer about developing directly in the notebook, if you must edit .py files used by the notebook then note the reload function which allows you to re-import already imported modules without having to shutdown and reopen the notebook.

Resources