I am playing with the RL colab that uses CartPole-v0 from gym. I opened the iPython notebook through jupyter connected to a remote centOS 7.3 server (not google drive). The following code renders the cartpole diagram correctly under google colab, but not with my jupyter setup:
!pip install gym[classic_control]
import gym
env = gym.make('CartPole-v0')
env.reset()
img = env.render('rgb_array')
plt.imshow(img)
Here is how I started my jupyter notebook:
xvfb-run -a -s "-screen 0 1400x900x24" jupyter notebook
Apparently the rendered image has either uninitialized dimensions or 0 dimensions. This is True whether using Jupyter or not.
Here is the full error:
I printed out the line just before the error in pyglet/image/__init__.py
def get_image_data(self):
# xxx = GLubyte * (len(self.format) * self.width * self.height)
print(GLubyte, len(self.format), self.width, self.height) # my code
buffer = (GLubyte * (len(self.format) * self.width * self.height))()
The results are either (4 0 0) or (4 1551512832 274094153), from which I infer the width and height are uninitialized. But I don't understand the internal well enough to debug further. In the former case, MemoryError becomes something error that says cannot render an image of size zero.
Any insight is appreciated. Here are some relevant threads:
How to run OpenAI Gym .render() over a server
https://blog.csdn.net/chestnutss/article/details/100734523
I managed to solve the environment issue with the following series of commands:
sudo yum install python-xvfbwrapper.noarch
sudo yum install mesa-dri-drivers
sudo yum install xorg-devel libglu1-mesa libgl1-mesa-devel libxinerama1 libxcursor1
sudo yum install glxinfo
sudo yum -y install freeglut-devel
Then the following rendering works in a jupyter browser notebook connected to the server
import gym
import matplotlib.pyplot as plt
env = gym.make('CartPole-v0')
env.reset()
img = env.render('rgb_array')
plt.imshow(img)
Related
How do I install a Linux image (https://cloud-images.ubuntu.com) in MicroStack (https://ubuntu.com/tutorials/microstack-get-started#2-install-microstack)? This question is close: How to install packages in cirros OS. But doesn't share the command that is needed. For example, I've tried the following:
microstack launch cirros --name test
The above works, but can't use it as it has no packages. So unusable.
So I tried the following, nothing is working
microstack launch ubuntu --name test
microstack launch debian --name test
microstack launch focal --name test
Second, I've download the a ubuntu ISO. And attempted to upload the image under Compute > Images. While it uploads in the GUI of OpenStack aka Horizon. The following error happens: "Flavor's disk is too small for requested image."
Is there a way to install and run RStudio on Google Colab?
I am aware that it is possible to run R code on Google Colab. Thus, i was wondering if there is also a work-around to install and run RStudio?
I am using Google Colab and RStudio from time to time. To make the set up easier, I usually copy & paste the following setup script (and use alternative 2).
After creating a new Google Colab Notebook, execute the following commands:
# Add new user called "rstudio" and define password (here "password123")
!sudo useradd -m -s /bin/bash rstudio
!echo rstudio:password123 | chpasswd
# Install R and RStudio Server (Don't forget to update version to latest version)
!apt-get update
!apt-get install r-base
!apt-get install gdebi-core
!wget https://download2.rstudio.org/server/bionic/amd64/rstudio-server-1.4.1103-amd64.deb
!gdebi -n rstudio-server-1.4.1103-amd64.deb
#===============================================================================
# ALTERNATIVE 1: Use ngrok
#===============================================================================
# Advantage: Runs in the background
# Disadvantage: Not so stable
# (often 429 errors during RStudio usage due to max 20 connections without account)
# Optionally register for a free accoount which gets this number up to 40 connections:
# https://ngrok.com/pricing
# Install ngrok (https://ngrok.com/)
!wget -c https://bin.equinox.io/c/4VmDzA7iaHb/ngrok-stable-linux-amd64.zip
!unzip -o ngrok-stable-linux-amd64.zip
# Run ngrok to tunnel RStudio app port 8787 to the outside world.
# This command runs in the background.
get_ipython().system_raw('./ngrok http 8787 &')
# Get the public URL where you can access the Dash app. Copy this URL.
! printf "\n\nClick on the following link: "
! curl -s http://localhost:4040/api/tunnels | python3 -c \
"import sys, json; print(json.load(sys.stdin)['tunnels'][0]['public_url'])"
# ==> To access to the RStudio server
# - click on this link and
# - use the username "rstudio" and
# - the password you defined at the first cell ("password123" in this example).
#===============================================================================
# ALTERNATIVE 2 (preferred): Use localtunnel
#===============================================================================
# (see also: https://github.com/naru-T/RstudioServer_on_Colab/blob/master/Rstudio_server.ipynb)
# Advantage: Stable usage of RStudio
# Disadvantage: Does not run in the background (i.e. Colab blocked)
# Install localtunnel
!npm install -g npm
!npm install -g localtunnel
# Run localtunnel to tunnel RStudio app port 8787 to the outside world.
# This command runs in the background.
!lt --port 8787
# ==> To access to the RStudio server
# - click on this link,
# - click button "Click to Continue" on the "friendly reminder" page,
# - use the username "rstudio" and
# - the password you defined at the first cell ("password123" in this example).
I have Airflow running using the postgres backend - all fine. Additionally I have running a Jupyter server on the same host where Airflow runs. Now I thought I can just access the airflow hooks from a notebook.
import pandas as pd
import numpy as np
import matplotlib as plt
from airflow.hooks.mysql_hook import MySqlHook
mysql = MySqlHook(mysql_conn_id = 'mysql-x')
sql = "select 1+1"
mysql.get_pandas_df(sql)
But I get this exception message:
OperationalError: (sqlite3.OperationalError) no such table: connection
[SQL: SELECT connection.password AS connection_password, connection.extra AS connection_extra, connection.id AS connection_id, connection.conn_id AS connection_conn_id, connection.conn_type AS connection_conn_type, connection.host AS connection_host, connection.schema AS connection_schema, connection.login AS connection_login, connection.port AS connection_port, connection.is_encrypted AS connection_is_encrypted, connection.is_extra_encrypted AS connection_is_extra_encrypted
FROM connection
WHERE connection.conn_id = ?]
[parameters: ('mysql-x',)]
(Background on this error at: http://sqlalche.me/e/e3q8)
But what makes me get suspicious is that it not only does not find the connection_id (which I clearly can see in the Airflow UI). It also says: sqlite3.OperationalError - it very much looks like it is not even connected to the same postgres database. I have checked os.environ["AIRFLOW_HOME"] which seems to be correct.
EDIT 1:
After I start the jupyter notebook server after airflow, such that all environment variables are set, I get a different error:
/usr/local/lib/python3.7/site-packages/sqlalchemy/orm/attributes.py in __get__(self, instance, owner)
351
352 def __get__(self, instance, owner):
--> 353 retval = self.descriptor.__get__(instance, owner)
354 # detect if this is a plain Python #property, which just returns
355 # itself for class level access. If so, then return us.
/usr/local/lib/python3.7/site-packages/airflow/models/connection.py in get_password(self)
188 "Can't decrypt encrypted password for login={}, \
189 FERNET_KEY configuration is missing".format(self.login))
--> 190 return fernet.decrypt(bytes(self._password, 'utf-8')).decode()
191 else:
192 return self._password
/usr/local/lib/python3.7/site-packages/cryptography/fernet.py in decrypt(self, msg, ttl)
169 except InvalidToken:
170 pass
--> 171 raise InvalidToken
InvalidToken:
You can use this dockerfile to reproduce it:
FROM apache/airflow
USER root
# install mysql client
RUN apt-get update && apt-get install -y mariadb-client-10.3 unzip
# install mssql client and tools
RUN apt-get install -y curl gnupg libicu-dev libicu63
RUN curl https://packages.microsoft.com/keys/microsoft.asc -o key
RUN apt-key add < key
RUN curl https://packages.microsoft.com/config/ubuntu/16.04/prod.list > /etc/apt/sources.list.d/msprod.list
ENV ACCEPT_EULA=Y
RUN apt-get update && apt-get install -y mssql-tools msodbcsql17 unixodbc unixodbc-dev unzip libunwind8
RUN curl -Lq 'https://go.microsoft.com/fwlink/?linkid=2108814' -o sqlpackage-linux-x64-latest.zip
RUN mkdir /opt/sqlpackage/ && unzip sqlpackage-linux-x64-latest.zip -d /opt/sqlpackage/
RUN chmod a+x /opt/sqlpackage/sqlpackage && ln -sfn /opt/sqlpackage/sqlpackage /usr/bin/sqlpackage
RUN ln -sfn /opt/mssql-tools/bin/sqlcmd /usr/bin/sqlcmd
# install notebooks
RUN pip install jupyterlab pandas numpy scikit-learn matplotlib pymssql
#RUN cat /entrypoint.sh
# start additional notebok server
# this is a dirty hack but for the sake of this prototype good enough
RUN sed -i -e's/\# Run the command/airflow scheduler \& \njupyter notebook --ip=0.0.0.0 --port=9000 --NotebookApp.token="" --NotebookApp.password="" \& \n/' /entrypoint
EXPOSE 9000
# switch back to airflow user
USER airflow
RUN airflow initdb
RUN alias ll='ls -al'
How to install Airflow on Windows 7? getting below error while installing it using pip install apache-airflow :
---------------------------------------- Command "c:\users\shrgupta5\appdata\local\programs\python\python36-32\python.exe
-u -c "import setuptools, tokenize;__file__='C:\\Users\\SHRGUP~1\\AppData\\Loca l\\Temp\\pip-build-_yptw7sa\\psutil\\setup.py';f=getattr(tokenize, 'open', open) (__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __fi le__, 'exec'))" install
--record C:\Users\SHRGUP~1\AppData\Local\Temp\pip-_cwm0n u7-record\install-record.txt --single-version-externally-managed
--compile" fail ed with error code 1 in C:\Users\SHRGUP~1\AppData\Local\Temp\pip-build-_yptw7sa\ psutil\
I wouldn't bother trying to install Airflow on windows, even after you install it successfully you cannot run the airflow script due to a dependency on the unix-only module pwd
You can run Airflow on Windows by using the Docker setup from puckel https://github.com/puckel/docker-airflow.
Use VirtualBox and Docker Toolbox(legacy) and setup a docker-machine on your Windows computer (docker-machine create -d virtualbox --virtualbox-cpu-count "2" --virtualbox-memory "2048" default)
make sure to fork puckels git repo underneath c:/Users/yourusername/documents otherwise mounting of DAGS won't work
You should now be able to spin up Airflow e.g. by using the celery-executor setup with docker compose -f docker-compose-CeleryExecutor.yml up -d
I have setup a environment where I develop the DAGs on Windows, test them within the dockercontainer and then push the Dockerimage to Linux to production. I have added a more detailed tutorial here.
Airflow cannot be installed on Windows within the standard command prompt.
You need to use bash and afterwards change the config:
How to run Airflow on Windows
Download the source of airflow from pypi:
https://pypi.org/project/airflow/#files
Unzip and edit setup.cfg, then go to the install_requires section and change the version of psutil with the following: 'psutil>=5.4.7',
Finally, run python setup.py install in the source directory
This Q&A is a response to this comment. The answer to the question in the comment is not trivial, is too big for a comment, and not suitable as an answer to the question in that thread (answering my own question is officially encouraged). If you have a better answer please post it!
The question is: How to install R on Solaris on a VirtualBox virtual machine?
A more up-to-date version is available from csw: r_base. To install, see the example in Getting started where you replace vim with r_base:
pkgadd -d http://get.opencsw.org/now
/opt/csw/bin/pkgutil -U
/opt/csw/bin/pkgutil -a r_base
/opt/csw/bin/pkgutil -y -i r_base
To install a development environment, you might also want:
/opt/csw/bin/pkgutil -y -i gcc4g++
/opt/csw/bin/pkgutil -y -i texlive
Start by downloading and installing Oracle VM VirtualBox.
Then download and unzip the Oracle Solaris 11.1 VirtualBox Template. After you unzip the Oracle template you should see a file called OracleSolaris11_1.ova, that's what you'll open in VirtualBox.
Start VirtualBox, click on File, then Import Appliance, then navigate to chose the ova file you just extracted. It will take some time to import.
Start the Solaris virtual machine by clicking on the start button on VirtualBox. It will take some time to start up and you'll be prompted to add a root password, user name and user password. You'll then use those details to log in, wait for the system to load, choose gnome to ensure you get a desktop environment, and choose your time zone, keyboard layout and language (mine seems to highlight Chinese as the default choice, so be careful not to click through that one too quickly).
Eventually you'll get a desktop, right-click on the desktop and click open terminal, then in the terminal type (or paste):
sudo wget https://oss.oracle.com/ORD/ord-3.0.1-sol10-x86-64-sunstudio12u3.tar.gz && sudo wget https://oss.oracle.com/ORD/ord-3.0.1-supporting-sol10-x86-64-sunstudio12u3.tar.gz
That will connect to the internet and download two files you need. The next line will unpack those two archives:
sudo tar -xzvf ord-3.0.1-sol10-x86-64-sunstudio12u3.tar.gz && sudo tar -xzvf ord-3.0.1-supporting-sol10-x86-64-sunstudio12u3.tar.gz
And then this next line installs R, watch for the prompts after you run the line:
sudo bash install.sh
A lot will flash by in the terminal, concluding with Installation of <ORD> was successful
Now the next bit is where I deviate from the instructions here because I didn't understand them. You'll move all files beginning with lib from the archives that you unpacked into another directory where they are needed by R:
sudo mv lib* /usr/lib/64/R/lib/
That will return nothing in the terminal. Then we can run R simply by typing in the terminal like so
R
And now you should have a regular R session running in the terminal.