I made a jupyter notebook that can read hdf5 files and use some functions to analyze the data. I would like to put this jupyter notebook on a server containing different hdf5 files and make it available for people who are working in other places. An exemple of functions would be to see the expression of some genes in a sample. Those people could open this jupyter notebook and add a list of specific genes to look at.
I am looking at JupyterLab, but I can see that people can read and modify the notebook. I would like that even if they modify it, when they finish, the notebook is as it was before they opened it. Do you think it is possible to do that ? I think I could do it locally with "read-only" but I don't know how to do it on jupyterLab.
I am a newbie for things related to servers. I will really appreciate your help and suggestions.
Thanks a lot :)
Some suggestions:
You could supply the notebook via a code sharing resource like GitHub and let them clone/copy/download use it as they see fit via their own resources. This insures they aren't changing your source notebook. And you don't have to worry about computational resources because they can run it where they prefer such as on their own cluster or at a cloud center.
Typically, I'd suggest the MyBinder project at https://mybinder.org/ for what you describe as it lets you share temporary active sessions that launch on remote servers. You set up a Github repository with notebooks and the data or a way to retrieve the data built in the notebook and when the temporary session launches, users can work through your notebook. This way they can extend and modify and run their own data and not change your source notebook. However, the resources are limiting as it is free, see here.
Examples:
solve for sediment transport
Resolving the measurement uncertainty paradox in ecological management
A quick introduction to RNAseq
bendit-binder
blast-binder
I've seen others use Code Ocean, see an example here.
There are other national/government-funded centers that allow similar hosting of services and resources that can be shared with others. CyVerse is one that is now running in the United States and several place, such as CyVerseUK in association with the Earlham Institute, and elsewhere. They offer notebooks to be served via their VICE apps in their Discover environment. Their resource allows more computational power and storage than the free, public MyBinder service.
You can use Mercury framework. It is an open-source tool that converts Python notebooks into interactive documents, like web apps, reports, dashboards, slides.
It requires an YAML header in the first cell in the notebook. In the header you define the presence of the notebook in the Mercury (title, description, show-code). Additionally, you can add widgets to the notebook. Widgets are directly connected to variables in the notebook. The variables which values are controlled by widgets should be in the separate cell.
The example notebook
The example notebook with YAML header
The YAML header
There are two widgets defined in the below YAML header. The name and points widgtes have the same name as variables in the code. User can change the widgets values, click Run and the notebook will be executed with new values.
---
title: My notebook
description: Notebook with plot
show-code: False
params:
name:
input: text
label: What is your name?
value: Piotr
points:
input: slider
label: How many points?
value: 200
min: 10
max: 250
---
The Web Application from Notebook
The final notebook after execution can be downloaded as standalone HTML or PDF file.
Deployment
The Mercury framework is built on top of Django. It can be deployed on any machine with Python available. The deployment for Heroku is very simple, and can be done with two commands (you need to have Heroku CLI tool):
heroku create my-amazing-web-app-name-here
git push heroku main
Please make sure that there is Procfile created for Heroku deployment with:
web: mercury run 0.0.0.0:$PORT
You can also deploy Mercury with docker-compose. Please check the docs for more details. You can also check the article on how to share Jupyter Notebook with non-programmers with Mercury tool.
You can install a local Gitlab server instead of publishing on Github. Then publish your Jupyter notebooks in Gitlab. Share your server URL with people. Our company is running all internally.
Related
This is basic educational question. I am confused about some fundamental concepts.
So I work on my local machine and I created R-markdown (Shiny doc) . Now, if I want to publish it, I just click in the top right corner (where I previously established account at shinyapps.io and connected to it). Well due to some reasons, I would like to publish it different place then shinyapps.io.
So my understanding is that I can accomplish it if I install and configure shiny server in docker. Is this accurate? Do I need Shiny proxy as well.
I also read about dockerizing shiny app, but this sound like to much work creating container instance for each app.
The publishing button works with shinyapps.io or rstudio connect. If you'd like to publish an app using free shiny-server, you won't be able to use the publishing button. You'll need to put a copy of the app in the folder tree you configure shiny-server to publish from. This is a bit more manual. You don't necessarily need Docker, but you do need a Linux computer to run shiny-server. If you want to put your apps in docker containers, you'll want to check out shinyproxy if you're trying to go the free route. RStudio has great paid options if you want to go that route.
I have a Virtual machine that I have customized with all the packages I need and it runs using virt-manager. I logged in and created a few directories and scripts in that image. I would like to "snapshot" it and run the snapshot on openstack. I don't care if the runtime state is preserved. All I want is for the packages and custom code that I have created on the virtual machine to be there.
I tried various things with the qcow2 image that virt-manager uses but that did not work. I don't quite understand - where are the changes I made to the file system?
I would like to generate an image with my customizations that I can subsequently run on an openstack cloud.
How can I proceed?
Thanks
Ranga
I'm looking into which service to chose when it comes to Jupyter Notebooks hosted in the cloud.
Google Colaboratory (GCL) seems to be able to share privately (with people with Google Apps accounts) and real-time collaborate with them.
However, for me, Azure Notebooks (AZNB) has some benefits that I prefer over GCL – But I cannot NOT have the feature of sharing and collaborating. It seems like such obvious features, that I'm posing the question here – am I missing something right in front of me?
Note: For AZNB, I've found the "share" option, which forces the notebook to be public, and the "clone" option, which lets others work on a copy of my library. This is not what I am looking for.
For now (29 sept 2018) - you can't get official way.
You can track this discussion here
https://github.com/Microsoft/AzureNotebooks/issues/329
I can suggest some option. Azure notebook service allow to use the terminal console, aznbsetup.sh file and have the preinstalled git.
You can put your notebooks in some private git repo, send the link and credential to your collaborators and tell some manual instructions
Create empty library
Open the terminal
Run the cd ~/library and git clone your_private_repo
All changes could be operated with standart git operations pull, push and etc
For autoupdates you can suggest the aznbsetup.sh with
#!/bin/bash
cd ~/library
git pull
It's worked for me.
Problem:
I would like to make julia available for our developers on our corporate network, which has no internet access at all (no proxy), due to sensitive data.
As far as I understand julia is designed to use github.
For instance julia> Pkg.init() tries to access:
git://github.com/JuliaLang/METADATA.jl
Example:
I solved this problem for R by creating a local CRAN repository (rsync) and setting up a local webserver.
I also solved this problem for python the same way by creating a local PyPi repository (bandersnatch) + webserver.
Question:
Is there a way to create a local repository for metadata and packages for julia?
Thank you in advance.
Roman
Yes, one of the benefits from using the Julia package manager is that you should be able to fork METADATA and host it anywhere you'd like (and keep a branch where you can actually check new packages before allowing your clients to update). You might be one of the first people to actually set up such a system, so expect that you will need to submit some issues (or better yet; pull requests) in order to get everything working smoothly.
See the extra arguments to Pkg.init() where you specify the METADATA repo URL.
If you want a simpler solution to manage I would also think about having a two tier setup where you install packages on one system (connected to the internet), and then copy the resulting ~/.julia directory to the restricted system. If the packages you use have binary dependencies, you might run into problems if you don't have similar systems on both sides, or if some of the dependencies is installed globally, but Pkg.build("Pkgname") might be helpful.
This is how I solved it (for now), using second suggestion by
ivarne.I use a two tier setup, two networks one connected to internet (office network), one air gapped network (development network).
System information: openSuSE-13.1 (both networks), julia-0.3.5 (both networks)
Tier one (office network)
installed julia on an NFS share, /sharename/local/julia.
soft linked /sharename/local/bin/julia to /sharename/local/julia/bin/julia
appended /sharename/local/bin/ to $PATH using a script in /etc/profile.d/scriptname.sh
created /etc/gitconfig on all office network machines: [url "https://"] insteadOf = git:// (to solve proxy server problems with github)
now every user on the office network can simply run # julia
Pkg.add("PackageName") is then used to install various packages.
The two networks are connected periodically (with certain security measures ssh, firewall, routing) for automated data exchange for a short period of time.
Tier two (development network)
installed julia on NFS share equal to tier one.
When the networks are connected I use a shell script with rsync -avz --delete to synchronize the .julia directory of tier one to tier two for every user.
Conclusion (so far):
It seems to work reasonably well.
As ivarne suggested there are problems if a package is installed AND something more than just file copying is done (compiled?) on tier one, the package wont run on tier two. But this can be resolved with Pkg.build("Pkgname").
PackageCompiler.jl seems like the best tool for using modern Julia (v1.8) on secure systems. The following approach requires a build server with the same architecture as the deployment server, something your institution probably already uses for developing containers, etc.
Build a sysimage with PackageCompiler's create_sysimage()
Upload the build (sysimage and depot) along with the Julia binaries to the secure system
Alias a script to julia, similar to the following example:
#!/bin/bash
set -Eeu -o pipefail
unset JULIA_LOAD_PATH
export JULIA_PROJECT=/Path/To/Project
export JULIA_DEPOT_PATH=/Path/To/Depot
export JULIA_PKG_OFFLINE=true
/Path/To/julia -J/Path/To/sysimage.so "$#"
I've been able to run a research pipeline on my institution's secure system, for which there is a public version of the approach.
Currently i am using postman rest client for checking API syntax for various products.
Is there any tool available like ipython notebook for testing REST API.
IPython notebook is a server which we can run python programs and store the programs and its outputs as a notebook. We can later open those notebooks and can view all the programs which we did.
I would like to store the API testing details with response and want to share those things.
Using postman, i can create a collection and share and i hope i cannot save the responses. Here in ipython notebook, i can save both the input and output.
You can either use python library to do your rest testing, or if you really want something more custom, I would suggest looking at writing a wrapper kernel or even maybe a full kernel for IPython. It allows to use your own custom languages in cells. See the a few of the existing kernels.
And don't be afraid to try to write one, it's quite easy,