I am having some data loaded inside of my pillar file configuration.sls which is using Jinja templating. I would like to do some additional processing on part of that data (dictionary structure) before saving as pillar. The additional processing is a bit complicated to be written in Jinja so I would like to do such additional processing in some Python function and be able to call that function like salt module.
I know I can define my own module inside of _modules called example.py in which I can have defined function called get_processed_data(data) and in state file call it in Jinja like:
{% set data_to_save = salt.example.get_processed_data(data) %}
But I am unable to do such thing in pillar configurations.sls file. When I try, pillar renders with error that it does not know example. Is there any approach how to do such thing?
I am aware about ext_pillars, but as far as I know, these are used like pillar file written in Python. What I am interested in, is to be able to create just simple function, which could be called in pillar file. And also I would guess it should be doable somehow, because in pillar it is possible to call salt.file.find(), where file is also a module (built in, but still).
since pillar is rendered on the master you need to make sure the module is synced to the master.
when you sync using salt <minion> saltutil.sync_modules you are syncing to the minion. to sync to the master use salt-run saltutil.sync_modules
see https://docs.saltproject.io/en/latest/ref/runners/all/salt.runners.saltutil.html#salt.runners.saltutil.sync_modules
everything else will be the same.
Related
I am trying to compile SQLite into a library which another application is then going to link against. I am not able to compile SQLite directly into that application for reasons which are beyond the scope of this question.
However, I need a VFS to be available which by default is not. In trying to figure out how to get this working I am trying to get the vfstrace shim to be made available to the application which is linking to SQLite. This will easily prove that it is working as I can log SQLite VFS activity from the shim and see that it is actually being used when the application calls SQLite.
How do I do this? All the examples I have found show the case of when you have a source file (such as shell.c) and you compile it, sqlite3.c and test_vfstrace.c to produce an executable. However, I do not have this luxury. I could compile sqlite.c and test_vfstrace.c and generate the libsqlite3.so library file, but there is no "main" function in which to call vfstrace_register so that the VFS is actually available. Is there some other hook for the library case where I can set this up? If no, how do I make a new VFS available?
SQLite is initialized via the sqlite3_initialize() function, which is called automatically when various functions such as sqlite3_open() are called by the user, although it is a no-op on subsequent invocations. This function in turn calls the OS specific initialization function sqlite3_os_init(). This is the function that initializes all the VFSes that are built into SQLite.
For the example VFS given, append test_vfstrace.c to the amalgamation and then put a call like this in sqlite3_os_init() right before the return statement:
vfstrace_register("trace",0,(int(*)(const char*,void*))fputs,stderr,1);
Then build the library with these modifications.
The last parameter's value of 1 will make this VFS be the default, so if you don't want trace messages printed for every SQLite operation performed via code linked the library you just created change this to 0 and explicitly specify this VFS when opening a database to trace the calls for. For instance, specify the SQLITE_USE_URI compile time option, and then pass a filename of the form: "file:database.db?vfs=unix" where "database.db" is the actual name of the file to open.
I think I've a good understanding of Symfony and how bundle works.
However I've never found how to solve a simple problem: make a reusable bundle that provides data like tables/Doctrine entities pre-filled with (i.e.) all country names in the world, all provinces of Italy, tax rates history in England and so on.
Of course the purpose is to provide forms, services and controllers relying on this data source, without the need to copy and paste tables and entities across projects.
How would you do that?
Data fixtures IMHO are not an option because an obvious reason: you are going to purge your database while it's running.
A custom command reading from a static data-source (json, YAML) and performing inserts/updates?
First step is declaring a Doctrine entity in your Bundle. I think you should create DataFixtures to populate your datas into db.
You maybe should consider to use Seeds instead of Fixtures.
Fixtures are fake datas, used to test your application
Seeds are the minimal datas required for your application to work.
Technically, these are exactly the same thing, you declare it under the "DataFixtures/" folder and you import them with the "doctrine:fixtures:load" command.
You can create a folder "Fixtures/", and a folder "Seeds/" under the folder "DataFixtures", then load your seeds with the command
php app/console doctrine:fixtures:load --fixtures=/path/to/seeds/folder --append
It was suggested in the comments that it may be safer, especially in production environment, to create a custom Symfony2 command to force the "--append" mode. Without this mode, your database will be purged, and you could loose your production data.
This answer assumes you're using composer to install your bundles (and you really exclude fixtures as an option).
What you can do, is make an SQL export of the data you want, and make sure it uses INSERT IGNORE INTO, and get the correct unique constraints.
Then you save that file somewhere in your bundle, in a "data" or "fixtures" folder.
so your path to that file will be like:
"vendor/company/epicbundle/data/countries.sql"
What you then can do, is add post-insert and post-update commands in your composer.json, that looks like this:
"post-install-cmd": [
"php app/console doctrine:query:sql \"$(cat vendor/company/epicbundle/data/countries.sql)\""
]
If you only want it to run on install, you only add it there, if you sometimes update the sql file, you also add it to the post-update-cmd.
Please note that this solution only works if people don't temper with the table names, otherwise the queries will fail.
If you want a more save/stable solution, you can write your own post-install script in Symfony that uses the entity manager, and there you can use, for example, a csv file, and insert/update it row by row.
Basically, anything you could implement would surely rely on persistence mechanisms used in your ORM/ODM/whatever. So, you'll end up implementing a typical fixture loading mechanism, at least partially: you'd execute code that saves some provided data; if it's serialized you'd do XML/JSON/YAML parsing (but this is just a technicality) and persist the results into the database.
Thus, it's not bad to stick with Doctrine Fixtures. They are programmable and extensible (you can even fetch your data from the web upon loading).
As stated in #paul-andrieux's answer, if you are worried about data loss (e.g. your bundle's seeds are loaded when the end user's DB is already up), you should use doctrine:fixtures:load --append and let the constraints do their job (like, in a country names table you'd have a unique constraint on country name or even a 'slug') so that inserting duplicate rows silently fails inserting a single entity, in case if your bundle has updated the country list, and the end user had a previous version.
If you really worry about your end users' data you could write a wrapper for the doctrine:fixtures:load command that would have the --append flag always on and register it as a separate command. (You could run needed migrations there, too)
#lxg's hard-coded IDs problem is solvable, too. Try using natural keys where applicable (e.g. the countries table would have a slug primary key that would be great-britain for Grean Britain). This way your searches would be pretty easy: $em->find('\MyBundle\Country', 'great-britain');. If you cannot come up with a natural key, then maybe the entity is not really needed for the end user.
UPD. Here's an article that could be useful: http://www.craftitonline.com/2014/09/doctrine-migrations-with-schema-api-without-symfony-symfony-cmf-seobundle-sylius-example/
Generally speaking, the bundle embedded the entities that will be loaded via the ORM/ODM using their built-in commands (like doctrine:schema:update, doctrine:migration:diff, ...) and provides a custom command that load the required fixtures using the ODM/ORM
This command can read the fixtures in multiple way (parsing yaml, xml, raw sql, dql, ...), it is just a matter of taste. Tones of bundles, parser, ... exist for those tasks.
In your documentation, you just have to state in a clear way that the developer must run this command after your bundle installation and schema update.
I usually follow the unofficial Meteor FAQ on how to structure my codebase, but I can't figure out where I should put my global constants.
To give an example: I have some database entries with a constant GUID that I need to reference in many points of my app. So far I just attached the constants to the relevant collection, such that in collections/myCollectionWithGuids.coffee it would say:
#MyCollectionWithGuids = new Meteor.Collection "myCollectionWithGuids"
#MyCollectionWithGuids.CONSTANT_ID = "8e7c2fe3-6644-42ea-b114-df8c5211b842"
This approach worked fine, until I need to use it in the following snippet, located in client/views/myCollectionWithGuidsView.coffee, where it says:
Session.setDefault "selectedOption", MyCollectionWithGuids.CONSTANT_ID
...which is unavailable because the file is being loaded before the Collections are created.
So where should I put my constants then, such that they are always loaded first without hacking in a bunch of subdirectories?
You could rely on the fact that a directory names lib is always treated first when it comes to load order.
So I would probably advise you to organize your code as follow :
lib/collections/collection.js
client/views/view.js
In your particular use case this is going to be okay, but you might find cases when you have to use lib in your client directory as well and as the load order rules stack (subdirectories being loaded first), it will be loaded BEFORE the lib folder residing in your project root.
For the moment, the only way to have full control over the load order is to rely on the package API, so you would have to make your piece of code a local package of your app (living in the packages directory of your project root).
It makes sense because you seem to have a collection and a view somehow related, plus splicing your project into a bunch of collaborative local packages tends to be an elegant design pattern after all.
Creating a local package is really easy now that Meteor 0.9 provide documentation for the package.js API.
http://docs.meteor.com/#packagejs
I would put your collection definitions in a lib directory. File structure documentation explains that all files under the lib directory get loaded before any other files, which means your variable would be defined when you attempt to access it in your client-side code.
Generally speaking, you always want your collections to be defined before anything else in your application is loaded or executed, since your application will most likely heavily depend upon the use of the collection's cursor.
In Meteor is there a way to include a js file in another js file.
Specifically, server side and most importantly at start up.
The use case I am running into is for complicated Meteor.startups where I need to load quite a bit of data to the mongodb into a variety of collections.
In order to have different test scripts I have to have more than one file each with duplicate data.
So, is there anyway to have say a boostrap.js file that calls Meteor.startup and then is able to load different files in order to load up the test data?
Or can this be done in a different way through some kind of object?
By design Meteor will automatically include all the javascript files in the the entire project (except in the public folder) but only segregate them between the server and client.
You could create objects in separate files and just use the functions or objects whenever you please, they should all be available at startup.
Try using my module loader made for use with Meteor. It's very similar to AMD: https://github.com/matb33/meteor-smd
I would like to use a pre-commit hook in riak to validate the json schema of the data submitted
There are a number of implementations in javascript on json-schema.org
http://json-schema.org/implementations.html
is there a way to load an external library or libraries into riak so i can use them inside the pre- and post-commit hooks
and preferably also the external schema definition files (or at least load them from a bucket)
I use clients in multiple languages to submit data to riak, and it would be ideal if riak could do the validtion of all data submitted.
Ok got my awnser from the mailing list,
Yes. For JavaScript, this can be done in app.config in 'js_source_dir'
in the 'riak_kv' settings:
{js_source_dir, "/etc/riak/javascript"},