I followed approach in this thread. I can easily set env variable in jupyter hub using the %env VAR = 5. However, when I try to print out this variable in the terminal I get only a blank line, as if the variable did not exist at all. Is it somehow possible to be able to print in terminal the env var defined in the notebook?
Setting environment variables from the notebook results in these variables being available only from that notebook.
%env VAR=TEST
import os
print(os.environ["VAR"])
...
>>> TEST
If you want to persist the variable, you need to put it either in the kernel.json file, or in systemd service file for jupyterhub, or in something like ~/.bashrc.
JupyterHub has a jupyterhub_config.py file (e.g. located in /etc/jupyterhub/jupyterhub_config.py) which can be used to control various aspects of the hub and the notebook environment.
In order to make a specific environment variable available to all notebooks on the hub, add the following lines to it:
c.Spawner.environment = {
'VAR': 'Test'
}
and restart the JupyterHub service (something like sudo service jupyterhub restart).
P.S. If you would just like to forward an environment variable from the user's environment, there is also
c.Spawner.env_keep = ['VAR']
Following from #leopold.talirz's answer, for those wanting to modify an environment variable without overwriting it (i.e. append a path to the PATH variable), I found you can do something like the following,
import os
original_path = os.environ['PATH']
c.Spawner.environment = {
'PATH': '/path/to/foo:{}'.format(original_path)
}
NOTE: In my case, I'm working with The Littlest JupyterHub, so I put the above in /opt/tljh/config/jupyterhub_config.d/environment.py.
Related
I'm using dagster 0.11.3 (the latest as of this writing)
I've created a Dagster pipeline (saved as pipeline.py) that looks like this:
#solid
def return_a(context):
return 12.34
#pipeline(
mode_defs=[
ModeDefinition(
executor_defs=[dask_executor] # Note: dask only!
)
]
)
def the_pipeline():
return_a()
I have the DAGSTER_HOME environment variable set to a directory that contains a file named dagster.yaml, which is an empty file. This should be ok because the defaults are reasonable based on these docs: https://docs.dagster.io/deployment/dagster-instance.
I have an existing Dask cluster running at "scheduler:8786". Based on these docs: https://docs.dagster.io/deployment/custom-infra/dask, I created a run config named config.yaml that looks like this:
execution:
dask:
config:
cluster:
existing:
address: "scheduler:8786"
I have SUCCESSFULLY used this run config with Dagster like so:
$ dagster pipeline execute -f pipeline.py -c config.yaml
(I checked the Dask logs and made sure that it did indeed run on my Dask cluster)
My question is: How can I get Dagit to use this Dask cluster?
The only thing I have found that seems related is this:
https://docs.dagster.io/_apidocs/execution#executors
...but it doesn't even mention Dask as an option (it has dagster.in_process_executor and dagster.multiprocess_executor, which don't seem at all related to dask).
Probably I need to configure dagster-dask, which is documented here: https://docs.dagster.io/_apidocs/libraries/dagster-dask#dask-dagster-dask
...but where do I put that run config when using Dagit? There's no way to feed config.yaml to Dagit, for example.
Some options:
you can manually plug in the values that are in config.yaml in to the dagit playground
you can bind the config directly to the executor if you do not need to change it ever https://docs.dagster.io/concepts/configuration/configured#configured-api
you can create a preset from that config yaml https://docs.dagster.io/tutorial/advanced-tutorial/pipelines#pipeline-config-presets
Given the context, I would recommend the configured API
I want to set some parameters as defined here(https://github.com/nteract/papermill#python-version-support). The catch is, I want to be able to do this via UI. I have a JHub installed on my cluster and while opening it, I want certain parameters to be set by default.
Also, when I pass the parameters via papermill(the above script gets saved somewhere and then I will run it via papermill), I want the latter to override the former.
I tried looking into several topics in pure JuPyter notebooks but in vain.
For the user to access some parameters as soon as her notebook starts, ipython needs to know the startup cells. This can be done via the following commands in case of JuPyterHub:
proxy:
secretToken: "yada yada"
singleuser:
image:
name: some_acc_id.dkr.ecr.ap-south-1.amazonaws.com/demo
tag: 12h
lifecycleHooks:
postStart:
exec:
command: ["/bin/sh", "-c", 'ipython profile create; cd ~/.ipython/profile_default/startup; echo ''run_id = "sample" ''> aviral.py']
imagePullSecret:
enabled: true
registry: some_acc_id.dkr.ecr.ap-south-1.amazonaws.com
username: aws
email: aviral#abc.com
Make sure you are escaping the quotes in the yaml correctly, or simply follow what I have done above.
Once this is done, papermill will override the params but for that, you have to make sure that the cell is tagged as "parameters". For instance, in my jupyterhub, every notebook that starts has run_id variable with the value "sample".
I'm trying to set an environment variable for an API key that I don't want in my code. My source javascript looks something like this :
.get(`http://api-url-and-parameters&api-key=${process.env.API_KEY}`)
I'm using webpack and the package dotenv-webpack https://www.npmjs.com/package/dotenv-webpack to set API_KEY in a gitignored .env file and it's all running fine on my local. I'd like to also be able to set that variable when deploying through Netlify, I've tried adding it through to GUI to the 'build environment variables', and also to set it directly in the build command, but without success.
Any idea what might be the issue ?
WARNING: If this is a secret key, you will not want to expose this environment variable value in any bundle that gets returned to the client. It should only be used by your build scripts to be used to create your content during build.
Issue
dotenv-webpack expects there to be a .env file to load in your variables during the webpack build of your bundle. When the repository is checked out by Netlify, the .env does not exist because for good reason it is in .gitignore.
Solution
Store your API_KEY in the Netlify build environment variables and build the .env using a script prior to running the build command.
scripts/create-env.js
const fs = require('fs')
fs.writeFileSync('./.env', `API_KEY=${process.env.API_KEY}\n`)
Run the script as part of your build
node ./scripts/create-env.js && <your_existing_webpack_build_command>
Caveats & Recommendations
Do not use this method with a public facing repository [open] because any PR or branch deploy could create a simple script into your code to expose the API_KEY
The example script above is for simplicity so, make any script you use be able to error out with a code other than 0 so if the script fails the deploy will fail.
You can set Dotenv-webpack to load system environment variables as well as those you have declared in your .env file by doing the following:
plugins: [
new Dotenv({
systemvars: true
})
]
I.e Setting the systemvars attribute of your webpack dotenv plugin to true.
Note that system environment variables with the same name will overwrite those defined in your .env file.
Source: https://www.npmjs.com/package/dotenv-webpack#properties
if you go to corresponding site's settings in Netlify, under build&deploy you can find a section called environment variables you can easily add your environment variables from there. if you add MY_API_KEY variable to environment variables you will be able to access it inside your project via process.env.MY_API_KEY.
If you're using Nuxt JS there is a more "straight forward" approach.
Just edit the nuxt.config.js like so:
module.exports = {
env: {
GOOGLE_API_KEY: process.env.GOOGLE_API_KEY
},
// ...
Then add the GOOGLE_API_KEY to Netlify through the build environment variables as usual.
Credit goes to yann-linn and his answer on github.
What you can also do is also to define a global constant in Webpack. Netlify environment variables defined in UI will work with it. You don't need dotenv or dotenv-webpack.
webpack.config.js
const webpack = require("webpack");
module.exports = {
plugins: [
new webpack.DefinePlugin({
"process.env.API_KEY": JSON.stringify(process.env.API_KEY)
}),
]
}
However again, of course you shouldn't do it just inputting enviornmental variables in the frontend if your API key is confidential and project public. The API key will appear in the source code of the website and will be easily accessible for everyone visiting it. Lambda function would be a better option.
You can use the Netlify's config file also ...
You can find documentation here.
Also i wanted to have the same ENV variables with with different values per branch/environment.
This workaround worked for me:
Create a netlify.toml file like:
[build]
NUXT_ENV_BASE_API = "/api"
NUXT_ENV_HOST_DOMAIN = "https://your-domain.gr"
[context.branch-deploy]
environment = { NUXT_ENV_BASE_API = "/dev-api", NUXT_ENV_HOST_DOMAIN = "https://dev.your-domain.gr" }
[context.production]
environment = { NUXT_ENV_BASE_API = "/api", NUXT_ENV_HOST_DOMAIN = "https://your-domain.gr" }
And deploy in Netlify ...
I have some Lua code, which I use in my openresty nginx.conf file. This Lua code contains such lines:
...
local secret = os.getenv("PATH")
assert(secret ~= nil, "Environment variable PATH not set")
...
Just for testing reasons I tried to check if PATH variable is set and for some reason the assert statement does not pass. I see in the console:
Environment variable PATH not set
However, when I run this
$ echo $PATH
I see, that this variable indeed has some value. So, what is wrong with that and how can I fix it?
You need to tell nginx to make environment variables available. From the docs for the env directive: "By default, nginx removes all environment variables inherited from its parent process except the TZ variable. This directive allows preserving some of the inherited variables, changing their values, or creating new environment variables."
So, in your case you'd need to specify env PATH; in nginx.conf.
I would like to set the configuration of my symfony2 project using environment variables.
In the server I have defined:
SYMFONY__DATABASE__USER
SYMFONY__DATABASE__PASSWORD
SYMFONY__DATABASE__NAME
SYMFONY__DATABASE__HOST
SYMFONY__DATABASE__DRIVER
My parameters.yml.dist looks like this:
#app/config/parameters.yml.dist
parameters:
database_host: "%database.host%"
database_port: ~
database_name: "%database.name%"
database_user: "%database.user%"
database_password: "%database.password%"
database_driver: "%database.driver%"
when I run composer I get an exception
composer install --dev --no-interaction --prefer-source
[Symfony\Component\DependencyInjection\Exception\ParameterNotFoundException]
You have requested a non-existent parameter "database.driver". Did you mean one of these: "database_user", "database_driver"?
These variables are defined in the server so I can modify the parameters.yml.dist to define these values. But this does not seams the right way, because wat I really want to use are the environment variables.
Note: I want to read this environment variables in travis, heroku and my vagrant machine. I only want to have in the repository the vagrant machine variables.
Which is the proper way to do this?
How should look my parameters.yml.dist?
Looks you are doing everything okay.
Here is the complete documentation for Setting Environment Variables which I believe you already read.
What is important to note is this:
Also, in order for your console to work (which does not use Apache),
you must export these as shell variables. On a Unix system, you can
run the following:
$ export SYMFONY__DATABASE__USER=user
$ export SYMFONY__DATABASE__PASSWORD=secret
I remember once I have a similar issue, I was setting everything on APACHE, but when running commands it wasn't working because I forgot to EXPORT the variables on the system.
Be aware that using export is a temp solution, if you reset your server those values will be lost, you will need to setup in a permanent way according to your OS.
I think you solved this long time ago, but the problem is actually that you have 2 _ between DATABASE and USER and the parser for this have a string replace function that replaces every __ with a . .
For your example to work you should have written like this:
SYMFONY__DATABASE_USER -> database_user
SYMFONY__DATABASE__USER -> database.user
You can try this bundle if your system version is >= 2.6.2:
This bundle provides a way to read parameters from environment
variables at runtime. The value defined in the container parameter is
used as fallback when the environment variable is not available.