I'm coding on a .ipynb file on a linux server.
The linux server I use has multiple GPUs on it, but I should only use idle GPU so as not to accidentally abort others' programme.
I've already known that for common .py file we can add some instructions at the command line to choose a common GPU(e.g. export CUDA_VISIBLE_DEVICES=#), but will it work for jupyter notebook? If not, how can I specify a GPU to work on.
You have to choose its name correctly. For instance there may be 3 GPU devices available namely "cuda:0","cuda:1","cuda:2". To choose the third one you need to run the following code:
if torch.cuda.is_available():
dev = "cuda:2"
else:
dev = "cpu"
device = torch.device(dev)
with tensorflow as below
gpus = tf.config.list_physical_devices('GPU')
if len(gpus) > 0:
tf.config.experimental.set_visible_devices(gpus[0], 'GPU')
Related
For python jupyter notebooks I am currently using VSCode python extension. However I cannot find any way to use alternative kernels. I am interested in jupyter R kernel in particular.
Is there any way to work with jupyter notebooks using R kernel in VSCode?
Yes, it is possible. It just requires an additional configuration to connect with the R kernel in VSCode.
It's worth noting that, if you prefer, you can use the notebook in VSCode Insiders where there is native support for notebooks in many languages, including R.
If you're using Jupyter in VSCode, firstly install IRkernel (R kernel).
According to the docs, run both lines to perform the installation:
install.packages('IRkernel')
IRkernel::installspec() # to register the kernel in the current R installation
Now, you should:
Reload Window Ctrl + R
Type Ctrl + Shift + P to search for "Jupyter: Create New Blank Notebook"
Click on the button right below ellipsis in upper right corner to choose kernel
Switch to the desired kernel, in this case, R's
That's it!
Agreed with #essicolo, if you are 100% stuck on using vscode this is a no-go.
[About kernels] Sorry, but as of right now this feature is only supported with Python. We are looking at supporting other languages in the future.
Yeah, that's the case for now, even if you start an external server. I hate having to say that, as we really want to support more of the various language kernels. But we started out with a Python focus and we still are pretty locked into that for the near future. Polyglot support is coming, but it won't be right away
per Microsoft Employee IanMatthewHuff
https://github.com/microsoft/vscode-python/issues/5109#issuecomment-480097310
preface - based on the phrasing of your question, I am making the assumption that you are trying to perform IRkernel in-line execution from your text ide without having to use a jupyter notebook / jupyterlab.
That said, if you're willing to go to the dark side, there might be some alternatives:
nteract's Hydrogen kernel for Atom IDE - the only text ide that I'm aware of that still supports execution against IRkernel. I know, I know - it's not vscode but it's as close as you'll probably get for now.
TwoSigma's Beaker notebook - it's been a lonngggg time for me but this a branch of jupyter that used to support polyglot editing, I'm not sure if that's still supported and it seems like you aren't that interested in notebooks anyway.
#testing_22 it works with me too
just add some note from my experience
It will failed If you run IRkernel::installspec() from RStudio or from Jupyter Conda environment failed way
Please run this syntax with VSCode terminal
install.packages('IRkernel')
IRkernel::installspec()
The rest is same, please restart VSCode and select "R" kernel from VSCode
I defined JUPYTER_PATH containing only a trusted set of kernels.
Is there an option to disable the search of kernels specs files in all other dirs?
I solved running
jupyter notebook '--KernelSpecManager.whitelist=["ir"]'
Source of solution in this comment: https://github.com/jupyter/jupyter_client/issues/144#issuecomment-242148826
I'm trying to run xv6 operating system on VirtualBox or VMWare in a Linux host. The official instructions said how to run the OS on qemu only. However, the official page (https://pdos.csail.mit.edu/6.828/2014/xv6.html) mentioned that xv6 can be booted directly on hardware also, but it's not clear how.
I want to boot xv6 on VirtualBox or VMware first. I extracted the following command from the Makefile, which runs xv6 from the command line after it's compiled using make command.
/usr/bin/qemu-system-i386 -serial mon:stdio -drive file=fs.img,index=1,media=disk,format=raw -drive file=xv6.img,index=0,media=disk,format=raw -smp 2 -m 512
Please help me how to proceed. If the procedure is already documented some reference will be helpful.
The instructions are here which is linked (via 6.828 tools page) from your link though they are a bit terse:
Using a Virtual Machine
Otherwise, the easiest way to get a compatible toolchain is to install
a modern Linux distribution on your computer. With platform
virtualization, Linux can cohabitate with your normal computing
environment. Installing a Linux virtual machine is a two step process.
First, you download the virtualization platform.
VirtualBox (free for Mac, Linux, Windows) — Download page
VMware Player (free for Linux and Windows, registration required)
VMware Fusion (Downloadable from IS&T for free).
VirtualBox is a little slower and less flexible, but free!
Once the virtualization platform is installed, download a boot disk
image for the Linux distribution of your choice.
Ubuntu Desktop is what we use.
This will download a file named something like
ubuntu-10.04.1-desktop-i386.iso. Start up your virtualization platform
and create a new (32-bit) virtual machine. Use the downloaded Ubuntu
image as a boot disk; the procedure differs among VMs but is pretty
simple. Type objdump -i, as above, to verify that your toolchain is
now set up. You will do your work inside the VM.
I can see how one could read that and not see the answer.
After the virtual machine is installed, download the Ubuntu Desktop .iso. Install that into the VM and fire it up. Presumably the Desktop will provide a clear mechanism for loading your OS. (Wait, I'm giving it a try. Will update with the result.)
Turns out that is simply a Ubuntu client desktop, and isn't anything special for running a sub-operating system.
Looking around some more, I found the commentary to be the best potential clue. It contains this (head scratcher) phrase:
To run xv6, install the QEMU PC simulators. To run in QEMU, run "make qemu".
If only it specified the context to get to that point! (Sorry I am not more help.)
I see that you want to boot it on VirtualBox or VMware, but another option would be to using docker to run xv6. A great guide for getting started with xv6 through docker is here.
The full guide is elaborate and can help you with getting started.
It is an alternative option, but one that can get you going fast hopefully.
It will only take 4 steps to get going with the xv6:
Step 1
Download and set up docker here
Step 2
- Run this command in PowerShell or bash to pull the ubuntu image with xv6 docker pull grantbot/xv6
Step 3
- To run the docker image and get going with xv6 run this command docker run -it grantbot/xv6
Step 4
- Now inside the shell in the ubuntu image run cd /home/a/xv6-public/ to enter the root folder of the xv6.
Done
- Now you can compile and run the xv6 with make qemu-nox
Step 1.Compile xv6
Download the code, unzip it and enter the directory, compile the operating system image and root file system, the command is as follows:
make xv6.img&&make fs.img
Step 2. Write image to disk
Create two disks in a existed vmware virtual machine（my vmware version is 15.2.2, linux version is Centos7.8）, the operation steps are: virtual machine settings -> add -> disk -> SCSI -> create a new virtual disk -> size 0.005 (allocate immediately, single file) -> name the disk "os", which means this disk is the operating system.
Create another disk named "fs" in the same way to put the root file system.
At this time, there should be "sdb" and "sdc" in the /dev/ directory (sda is the current operating system itself). If you do not see the "sdb" and "sdc", restart the guest operating system.
Write the operating system and root file system to the disk with the following command:
dd if=./xv6.img of=/dev/sdb bs=4k count=1000
dd if=./fs.img of=/dev/sdc bs=4k count=1000
shutdown the current virtual machine to ensure that the file has sync to the disk. At this time, the two images have been written to the disk, vmware saves the disk as a file, the location is in the directory of the current virtual machine, named os.vmdk, fs.vmdk, the next step will load these two files into the new virtual machine.
Step 3. Create xv6 virtual machine
To create an empty virtual machine, the operation steps are: customize (advanced) -> next -> install the operating system later -> choose other operating system type (choose other versions) -> take the virtual machine name as xv6 (name depend on you) ) -> Then use the default configuration all the way to "Next" to completion.
Right-click the created virtual machine and delete the disk created by default. Add the disk file created in the previous step to the current virtual machine. The operation steps are: add -> "disk" -> ide (note that this is an IDE instead of a SCSI disk, because xv6 reads an IDE format disk) -> use an existing virtual disk -> select the os.vmdk generate in the step 2->complete
Add fs.vmdk in the same way. Note that you must add os.vmdk first. Because os.vmdk is the operating system, it needs to be the first hard disk.
Now, you create a virtual machine which has two disk. one is os disk, another is root file system disk, all is ready.
Start the virtual machine, and the xv6 will start successfully.
How do I set a maximum memory limit for a jupyter notebook process?
If I use too much RAM the computer gets blocked and I have to press the power button to restart the computer manually.
Is there a way of automatically killing a jupyter notebook process as soon as a user-set memory limit is surpassed or to throw a memory error? Thanks
Under Linux you can use "cgroups" to limit resources for any software running on your computer.
Install cgroup-tools with apt-get install cgroup-tools
Edit its configuration /etc/cgconfig.conf to make a profile for the particular type of work (e.g. numerical scientific computations):
group app/numwork {
memory {
memory.limit_in_bytes = 500000000;
}
}
Apply that configuration to the process names you care about by listing them in /etc/cgrules.conf (in my case it is all julia executables, which I run through jupyter, but you can use it for any other software too):
*:julia memory app/numwork/
Finally, parse the config and set it as the current active config with the following commands:
~# cgconfigparser -l /etc/cgconfig.conf
~# cgrulesengd
I use this to set limits on processes running on a server that is used by my whole class of students.
This page has some more details and other ways to use cgroups https://wiki.archlinux.org/index.php/cgroups
I want to view an image in Jupyter notebook. It's a 9.9MB .png file.
from IPython.display import Image
Image(filename='path_to_image/image.png')
I get the below error:
IOPub data rate exceeded.
The notebook server will temporarily stop sending output
to the client in order to avoid crashing it.
A bit surprising and reported elsewhere.
Is this expected and is there a simple solution?
(Error msg suggests changing limit in --NotebookApp.iopub_data_rate_limit.)
Try this:
jupyter notebook --NotebookApp.iopub_data_rate_limit=1.0e10
Or this:
yourTerminal:prompt> jupyter notebook --NotebookApp.iopub_data_rate_limit=1.0e10
I ran into this using networkx and bokeh
This works for me in Windows 7 (taken from here):
To create a jupyter_notebook_config.py file, with all the defaults commented out, you can use the following command line:
$ jupyter notebook --generate-config
Open the file and search for c.NotebookApp.iopub_data_rate_limit
Comment out the line c.NotebookApp.iopub_data_rate_limit = 1000000 and change it to a higher default rate. l used c.NotebookApp.iopub_data_rate_limit = 10000000
This unforgiving default config is popping up in a lot of places. See git issues:
jupyter
IOPub data rate exceeded
It looks like it might get resolved with the 5.1 release
Update:
Jupyter notebook is now on release 5.2.2. This problem should have been resolved. Upgrade using conda or pip.
Removing print statements can also fix the problem.
Apart from loading images, this error also happens when your code is printing continuously at a high rate, which is causing the error "IOPub data rate exceeded". E.g. if you have a print statement in a for loop somewhere that is being called over 1000 times.
By typing 'jupyter notebook --NotebookApp.iopub_data_rate_limit=1.0e10' in Anaconda PowerShell or prompt, the Jupyter notebook will open with the new configuration. Try now to run your query.
Some additional advice for Windows(10) users:
If you are using Anaconda Prompt/PowerShell for the first time, type "Anaconda" in the search field of your Windows task bar and you will see the suggested software.
Make sure to open the Anaconda prompt as administrator.
Always navigate to your user directory or the directory with your Jupyter Notebook files first before running the command. Otherwise you might end up somewhere in your system files and be confused by an unfamiliar file tree.
The correct way to open Jupyter notebook with new data limit from the Anaconda Prompt on my own Windows 10 PC is:
(base) C:\Users\mobarget\Google Drive\Jupyter Notebook>jupyter notebook --NotebookApp.iopub_data_rate_limit=1.0e10
I have the same problem in my Jupyter NB on Win 10 when querying from a MySQL database.
Removing any print statements solved my problem.
For already running docker containers, try editing the file name - ~/.jupyter/jupyter_notebook_config.py
uncomment the line - NotebookApp.iopub_data_rate_limit =
and set high number like 1e10.
Restart the docker, it should fix the problem
I ran into this problem running version 6.3.0. When I tried the top rated solution by Merlin the powershell prompt notified me that iopub_data_rate_limit has moved from NotebookApp to ServerApp. The solution still worked but wanted to mention the variation, especially as internal handling of the config may become deprecated.
Easy workaround is to create a for loop and print. Then there wont be any issue. Printing directly wcc would cause if graph is huge. Hence any of below code will work as workaround.
wcc=list(nx.weakly_connected_components(train_graph))
for i in range(1,10):
print(wcc[i])
for i in wcc):
print(wcc)
Like others pointed out, print statement at a high rate can cause this. Resolve it by printing modulo a number using if statement. Example in python:
k = 10
if (i % k == 0):
print("Something")
Increase k if the warning persists.
Using Visual Studio Code, the Jupyter extension will be able to handle big data. launch from anaconda navigator
In general, trying to print something that is too long will trigger this error. I tried to print a string that was 9221593 characters long (too long), and that triggered the error.