A relative sqlalchemy path to a sqlite database can be written as:
sqlite:///folder/db_file.db
And an absolute one as:
sqlite:////home/user/folder/db_file.db
Is it possible to write a path relative to home? Like this:
sqlite:///~/folder/db_file.db
Or even better, can the path contain environment variables?
sqlite:////${MY_FOLDER}/db_file.db
This is the context of an alembic.ini file. So if the previous objectives are not possible directly, may I be able to cheat using variable substitution?
[alembic]
script_location = db_versions
sqlalchemy.url = sqlite:///%(MY_FOLDER)s.db
...
I have gone around this issue by modifying the values in the config object just after env.py imports it:
# this is the Alembic Config object, which provides
# access to the values within the .ini file in use.
config = context.config
# import my custom configuration
from my_app import MY_DB_URI
# overwrite the desired value
config.set_main_option("sqlalchemy.url", MY_DB_URI)
Now config.get_main_option("sqlalchemy.url") returns the MY_DB_URI you wanted.
As others have pointed out, one key is 3 slashes for relative, 4 for absolute.
But it took for me than just that...
Had trouble with just a string, I had to do this:
db_dir = "../../database/db.sqlite"
print(f'os.path.abspath(db_dir): {str(os.path.abspath(db_dir))}')
SQLALCHEMY_DATABASE_URI = "sqlite:///" + os.path.abspath(db_dir) # works
# SQLALCHEMY_DATABASE_URI = "sqlite:///" + db_dir # fails
From the alembic documentation (emphasis mine):
sqlalchemy.url - A URL to connect to the database via SQLAlchemy. This configuration value is only used if the env.py file calls upon them; in the “generic” template, the call to config.get_main_option("sqlalchemy.url") in the run_migrations_offline() function and the call to engine_from_config(prefix="sqlalchemy.") in the run_migrations_online() function are where this key is referenced. If the SQLAlchemy URL should come from some other source, such as from environment variables or a global registry, or if the migration environment makes use of multiple database URLs, the developer is encouraged to alter the env.py file to use whatever methods are appropriate in order to acquire the database URL or URLs.
So for this case, the sqlalchemy url format can be circunvented and generated by python itself.
Chef has a very elaborate (maybe too much so) scheme for cookbooks to provide default values of attributes. I think Puppet does something similar with class parameters where defaults usually go into params.pp. With Salt, I've seen:
specifying default value in dictionary/pillar lookups.
the grains.filter_by merging of default attribute values with user-provided pillar data (e.g., map.jinja in apache-formula)
in a call to file.managed state, specifying default attribute values as the defaults parameter and user-specified pillar data as context.
Option 1 seems to be the most common, but has the drawback that the template file becomes very hard to read. It also requires repeating the default value whenever the lookup is done, making it very easy to make a mistake.
Option 2 feels closest in spirit to Chef's approach, but seems to expect the defaults broken down into a dictionary of cases based on some filtering attribute (e.g., the OS type recorded in grains).
Option 3 is not bad, but puts attribute defaults into the state file, instead of separating them into their own file as they are with option 2.
Saltstack's best practices doc endorses Option 2, except that it doesn't address how to merge defaults with user-specified values without having to use grains.filter_by. Is there any way around it?
Note: The behavior of defaults.get changed in version 2015.8, and so the method described here no longer works. I am leaving this answer for users of older versions and will post a similar method for current versions.
defaults.get coupled with a defaults.yaml file should do what you want. Assume your formula tree looks like this:
my-formula/
files/
template.jinja
init.sls
defaults.yaml
# my-formula/init.sls
my-formula-conf-file:
file.managed:
- name: {{ salt['defaults.get']('conf_location') }}
- source: {{ salt['defaults.get']('conf_source') }}
... and so on.
# defaults.yaml
conf_location: /etc/my-formula.conf
conf_source: salt://my-formula/files/template.jinja
# pillar/my-formula.sls
my-formula:
conf_location: /etc/my-formula/something.conf
This will end with the configuration file placed at /etc/my-formula/something.conf (the pillar value) using salt://my-formula/files/template.jinja as the source (the default, for which no pillar override was supplied).
Note the unintuitive structure of the pillar and defaults files; defaults.get expects defaults.yaml to have its values at the root of the file, but expects the pillar overrides to be in a dictionary named after the formula, because consistency is for the weak.
The documentation for defaults.get gives its example using defaults.json instead of defaults.yaml. That works but I find yaml much more readable. And writable.
There is a bug using defaults.get from inside a managed template rather than within the state file, and as far as I know it's still open. It can still be made to work; the workaround is behind the link.
The behavior of defaults.get changed in 2015.8, possibly due to a bug. This answer describes a compatible method of getting the same results in (at least) 2015.8 and later.
Suppose your formula tree looks like this:
something/
files/
template.jinja
init.sls
defaults.yaml
# defaults.yaml
conf_location: /etc/something.conf
conf_source: salt://something/files/template.jinja
# pillar/something.sls
something:
conf_location: /etc/something/something.conf
The idea is that formula defaults are in defaults.yaml, but can be overridden in pillar. Anything not provided in pillar should use the value in defaults. You can accomplish this with a few lines at the top of any given .sls:
# something/init.sls
{%- set pget = salt['pillar.get'] %} # Convenience alias
{%- import_yaml slspath + "/defaults.yaml" as defaults %}
{%- set something = pget('something', defaults, merge=True) %}
something-conf-file:
file.managed:
- name: {{ something.conf_location }}
- source: {{ something.conf_source }}
- template: jinja
- context:
slspath: {{ slspath }}
... and so on.
What this does: The contents of defaults.yaml are loaded in as a nested dictionary. That nested dictionary is then merged with the contents of the something pillar key, with the pillar winning conflicts. The result is a nested dictionary containing both the defaults and any pillar overrides, which can then be used directly without concern to where a particular value came from.
slspath is not strictly required for this to work; it's a magic variable that contains the directory path to the currently-running sls. I like to use it because it decouples the formula from any particular location in the directory tree. It is not normally available from managed templates, which is why I pass it on as explicit context above. It may not work as expected in older versions, in which case you'll have to provide a path relative to the root of the salt tree.
The downside to this method is that, so far as I know, you can't access the final dictionary with salt's colon-based nested-keys syntax; you need to descend through it one level at a time. I have not had problems with that (dot syntax is easier to type anyway), but it is a downside. Another downside is the need for a few lines of boilerplate at the top of any .sls or template using the technique.
There are a few upsides. One is that you can loop over the final dictionary or its sub-dicts with .items() and the Right Thing will happen, which was not the case with defaults.get and which drove me insane. Another is that, if and when the salt team restores defaults.get's old functionality, the defaults/pillar structure suggested here is already compatible and they'll work fine side by side.
I created a parameter in my parameters file:
parameters:
category:
var: test
I can access this in PHP by fetching it like this:
$var = $this->container->getParameter('category');
$var = $var['var'];
But how can I access this parameter in my config.yml? For example, I want to pass this parameter to all my twig files as a global twig variable:
twig:
globals:
my_var: %category.var% # throws ParameterNotFoundException
(Sidequestion:
I assumed I could access it via getParamter('category.var'), but got an error there. Do you know a nicer way than my two-liner? $this->container->getParameter('category')['var'] works, but is a syntax error according to my IDE.)
$this->container->getParameter('category')['var']
..is actually a pretty good way to go. Which version of PHP is your IDE using for its syntax checking? Somebody please correct me, but I think this behavior was made valid in 5.3 or 5.4.
I am working on my own boilerplate using grunt's CLI possibilities like so grunt init:webdesign-project - for this I created a folder named webdesign-project within node_modules/grunt/init and a corresponding webdesign-project.js file. So far everything works great.
Now I wanted to insert my own "questions" with the grunt.helper function like this
grunt.helper('prompt_for', 'img_path', 'img')
however this gives me
TypeError: Cannot set property 'name' of undefined
at Object.module.exports.grunt.registerHelper.grunt.utils.spawn.cmd (/usr/lib/node_modules/grunt/tasks/init.js:573:17)
at Task.helper (/usr/lib/node_modules/grunt/lib/util/task.js:117:19)
at Object.exports.template (/usr/lib/node_modules/grunt/tasks/init/webdesign-project.js:30:11)
at Object.module.exports.grunt.registerHelper.done (/usr/lib/node_modules/grunt/tasks/init.js:240:27)
at Object.task.registerTask.thisTask.fn (/usr/lib/node_modules/grunt/lib/grunt/task.js:58:16)
at Task.<anonymous> (/usr/lib/node_modules/grunt/lib/util/task.js:341:36)
at Task.start (/usr/lib/node_modules/grunt/lib/util/task.js:357:5)
at Object.grunt.tasks (/usr/lib/node_modules/grunt/lib/grunt.js:143:8)
at Object.module.exports [as cli] (/usr/lib/node_modules/grunt/lib/grunt/cli.js:36:9)
at Object.<anonymous> (/usr/lib/node_modules/grunt/bin/grunt:19:14)
Isn't it possible to define your own variables using this function?
EDIT: Does anybody know if a documentation for this function exists? (Couldn't find one yet)
You shouldn't modify or add files within the node_modules/ folder as they will be overwritten upon updating with npm. Take a look at the init docs for creating custom init templates: https://github.com/gruntjs/grunt/blob/master/docs/task_init.md#creating-custom-templates
I recommend copying one of the existing init templates to: ~/.grunt/tasks/init/webdesign-project.js and modify from there.
I figured out how to achieve the "custom prompt" - in case anybody is interested:
Grunt's grunt.helper('prompt_for', '...') function apparently only takes a predefined set of values in lieu of '...'. Actually this should not be surprising, as there are some pretty unique features for some of those values (e.g. when you've entered testproject as your project's name, "(git://github.com/peter/testproject.git)" will automatically be proposed.
Solution: Take a look at the .js file of the gruntfile template (node_modules/grunt/tasks/init/gruntfile.js) - creating a custom prompt goes like this:
{
name: 'img_path',
message: 'Name the folder where all image files are located',
default: 'img',
// warning: '' couldn't find any use for this optional property
}
instead of
grunt.helper('prompt_for', 'img_path', 'img')
Closed. This question is opinion-based. It is not currently accepting answers.
Want to improve this question? Update the question so it can be answered with facts and citations by editing this post.
Closed 2 years ago.
The community reviewed whether to reopen this question last year and left it closed:
Original close reason(s) were not resolved
Improve this question
In my endless quest in over-complicating simple stuff, I am researching the most 'Pythonic' way to provide global configuration variables inside the typical 'config.py' found in Python egg packages.
The traditional way (aah, good ol' #define!) is as follows:
MYSQL_PORT = 3306
MYSQL_DATABASE = 'mydb'
MYSQL_DATABASE_TABLES = ['tb_users', 'tb_groups']
Therefore global variables are imported in one of the following ways:
from config import *
dbname = MYSQL_DATABASE
for table in MYSQL_DATABASE_TABLES:
print table
or:
import config
dbname = config.MYSQL_DATABASE
assert(isinstance(config.MYSQL_PORT, int))
It makes sense, but sometimes can be a little messy, especially when you're trying to remember the names of certain variables. Besides, providing a 'configuration' object, with variables as attributes, might be more flexible. So, taking a lead from bpython config.py file, I came up with:
class Struct(object):
def __init__(self, *args):
self.__header__ = str(args[0]) if args else None
def __repr__(self):
if self.__header__ is None:
return super(Struct, self).__repr__()
return self.__header__
def next(self):
""" Fake iteration functionality.
"""
raise StopIteration
def __iter__(self):
""" Fake iteration functionality.
We skip magic attribues and Structs, and return the rest.
"""
ks = self.__dict__.keys()
for k in ks:
if not k.startswith('__') and not isinstance(k, Struct):
yield getattr(self, k)
def __len__(self):
""" Don't count magic attributes or Structs.
"""
ks = self.__dict__.keys()
return len([k for k in ks if not k.startswith('__')\
and not isinstance(k, Struct)])
and a 'config.py' that imports the class and reads as follows:
from _config import Struct as Section
mysql = Section("MySQL specific configuration")
mysql.user = 'root'
mysql.pass = 'secret'
mysql.host = 'localhost'
mysql.port = 3306
mysql.database = 'mydb'
mysql.tables = Section("Tables for 'mydb'")
mysql.tables.users = 'tb_users'
mysql.tables.groups = 'tb_groups'
and is used in this way:
from sqlalchemy import MetaData, Table
import config as CONFIG
assert(isinstance(CONFIG.mysql.port, int))
mdata = MetaData(
"mysql://%s:%s#%s:%d/%s" % (
CONFIG.mysql.user,
CONFIG.mysql.pass,
CONFIG.mysql.host,
CONFIG.mysql.port,
CONFIG.mysql.database,
)
)
tables = []
for name in CONFIG.mysql.tables:
tables.append(Table(name, mdata, autoload=True))
Which seems a more readable, expressive and flexible way of storing and fetching global variables inside a package.
Lamest idea ever? What is the best practice for coping with these situations? What is your way of storing and fetching global names and variables inside your package?
How about just using the built-in types like this:
config = {
"mysql": {
"user": "root",
"pass": "secret",
"tables": {
"users": "tb_users"
}
# etc
}
}
You'd access the values as follows:
config["mysql"]["tables"]["users"]
If you are willing to sacrifice the potential to compute expressions inside your config tree, you could use YAML and end up with a more readable config file like this:
mysql:
- user: root
- pass: secret
- tables:
- users: tb_users
and use a library like PyYAML to conventiently parse and access the config file
I like this solution for small applications:
class App:
__conf = {
"username": "",
"password": "",
"MYSQL_PORT": 3306,
"MYSQL_DATABASE": 'mydb',
"MYSQL_DATABASE_TABLES": ['tb_users', 'tb_groups']
}
__setters = ["username", "password"]
#staticmethod
def config(name):
return App.__conf[name]
#staticmethod
def set(name, value):
if name in App.__setters:
App.__conf[name] = value
else:
raise NameError("Name not accepted in set() method")
And then usage is:
if __name__ == "__main__":
# from config import App
App.config("MYSQL_PORT") # return 3306
App.set("username", "hi") # set new username value
App.config("username") # return "hi"
App.set("MYSQL_PORT", "abc") # this raises NameError
.. you should like it because:
uses class variables (no object to pass around/ no singleton required),
uses encapsulated built-in types and looks like (is) a method call on App,
has control over individual config immutability, mutable globals are the worst kind of globals.
promotes conventional and well named access / readability in your source code
is a simple class but enforces structured access, an alternative is to use #property, but that requires more variable handling code per item and is object-based.
requires minimal changes to add new config items and set its mutability.
--Edit--:
For large applications, storing values in a YAML (i.e. properties) file and reading that in as immutable data is a better approach (i.e. blubb/ohaal's answer).
For small applications, this solution above is simpler.
How about using classes?
# config.py
class MYSQL:
PORT = 3306
DATABASE = 'mydb'
DATABASE_TABLES = ['tb_users', 'tb_groups']
# main.py
from config import MYSQL
print(MYSQL.PORT) # 3306
Let's be honest, we should probably consider using a Python Software Foundation maintained library:
https://docs.python.org/3/library/configparser.html
Config example: (ini format, but JSON available)
[DEFAULT]
ServerAliveInterval = 45
Compression = yes
CompressionLevel = 9
ForwardX11 = yes
[bitbucket.org]
User = hg
[topsecret.server.com]
Port = 50022
ForwardX11 = no
Code example:
>>> import configparser
>>> config = configparser.ConfigParser()
>>> config.read('example.ini')
>>> config['DEFAULT']['Compression']
'yes'
>>> config['DEFAULT'].getboolean('MyCompression', fallback=True) # get_or_else
Making it globally-accessible:
import configpaser
class App:
__conf = None
#staticmethod
def config():
if App.__conf is None: # Read only once, lazy.
App.__conf = configparser.ConfigParser()
App.__conf.read('example.ini')
return App.__conf
if __name__ == '__main__':
App.config()['DEFAULT']['MYSQL_PORT']
# or, better:
App.config().get(section='DEFAULT', option='MYSQL_PORT', fallback=3306)
....
Downsides:
Uncontrolled global mutable state.
A small variation on Husky's idea that I use. Make a file called 'globals' (or whatever you like) and then define multiple classes in it, as such:
#globals.py
class dbinfo : # for database globals
username = 'abcd'
password = 'xyz'
class runtime :
debug = False
output = 'stdio'
Then, if you have two code files c1.py and c2.py, both can have at the top
import globals as gl
Now all code can access and set values, as such:
gl.runtime.debug = False
print(gl.dbinfo.username)
People forget classes exist, even if no object is ever instantiated that is a member of that class. And variables in a class that aren't preceded by 'self.' are shared across all instances of the class, even if there are none. Once 'debug' is changed by any code, all other code sees the change.
By importing it as gl, you can have multiple such files and variables that lets you access and set values across code files, functions, etc., but with no danger of namespace collision.
This lacks some of the clever error checking of other approaches, but is simple and easy to follow.
Similar to blubb's answer. I suggest building them with lambda functions to reduce code. Like this:
User = lambda passwd, hair, name: {'password':passwd, 'hair':hair, 'name':name}
#Col Username Password Hair Color Real Name
config = {'st3v3' : User('password', 'blonde', 'Steve Booker'),
'blubb' : User('12345678', 'black', 'Bubb Ohaal'),
'suprM' : User('kryptonite', 'black', 'Clark Kent'),
#...
}
#...
config['st3v3']['password'] #> password
config['blubb']['hair'] #> black
This does smell like you may want to make a class, though.
Or, as MarkM noted, you could use namedtuple
from collections import namedtuple
#...
User = namedtuple('User', ['password', 'hair', 'name']}
#Col Username Password Hair Color Real Name
config = {'st3v3' : User('password', 'blonde', 'Steve Booker'),
'blubb' : User('12345678', 'black', 'Bubb Ohaal'),
'suprM' : User('kryptonite', 'black', 'Clark Kent'),
#...
}
#...
config['st3v3'].password #> passwd
config['blubb'].hair #> black
I did that once. Ultimately I found my simplified basicconfig.py adequate for my needs. You can pass in a namespace with other objects for it to reference if you need to. You can also pass in additional defaults from your code. It also maps attribute and mapping style syntax to the same configuration object.
please check out the IPython configuration system, implemented via traitlets for the type enforcement you are doing manually.
Cut and pasted here to comply with SO guidelines for not just dropping links as the content of links changes over time.
traitlets documentation
Here are the main requirements we wanted our configuration system to have:
Support for hierarchical configuration information.
Full integration with command line option parsers. Often, you want to read a configuration file, but then override some of the values with command line options. Our configuration system automates this process and allows each command line option to be linked to a particular attribute in the configuration hierarchy that it will override.
Configuration files that are themselves valid Python code. This accomplishes many things. First, it becomes possible to put logic in your configuration files that sets attributes based on your operating system, network setup, Python version, etc. Second, Python has a super simple syntax for accessing hierarchical data structures, namely regular attribute access (Foo.Bar.Bam.name). Third, using Python makes it easy for users to import configuration attributes from one configuration file to another.
Fourth, even though Python is dynamically typed, it does have types that can be checked at runtime. Thus, a 1 in a config file is the integer ‘1’, while a '1' is a string.
A fully automated method for getting the configuration information to the classes that need it at runtime. Writing code that walks a configuration hierarchy to extract a particular attribute is painful. When you have complex configuration information with hundreds of attributes, this makes you want to cry.
Type checking and validation that doesn’t require the entire configuration hierarchy to be specified statically before runtime. Python is a very dynamic language and you don’t always know everything that needs to be configured when a program starts.
To acheive this they basically define 3 object classes and their relations to each other:
1) Configuration - basically a ChainMap / basic dict with some enhancements for merging.
2) Configurable - base class to subclass all things you'd wish to configure.
3) Application - object that is instantiated to perform a specific application function, or your main application for single purpose software.
In their words:
Application: Application
An application is a process that does a specific job. The most obvious application is the ipython command line program. Each application reads one or more configuration files and a single set of command line options and then produces a master configuration object for the application. This configuration object is then passed to the configurable objects that the application creates. These configurable objects implement the actual logic of the application and know how to configure themselves given the configuration object.
Applications always have a log attribute that is a configured Logger. This allows centralized logging configuration per-application.
Configurable: Configurable
A configurable is a regular Python class that serves as a base class for all main classes in an application. The Configurable base class is lightweight and only does one things.
This Configurable is a subclass of HasTraits that knows how to configure itself. Class level traits with the metadata config=True become values that can be configured from the command line and configuration files.
Developers create Configurable subclasses that implement all of the logic in the application. Each of these subclasses has its own configuration information that controls how instances are created.