I have two .sls state files. One sets up a system service. The other formats and mounts storage volumes.
The system service doesn't need to know or care about its underlying storage; it just writes data to a given directory. Likewise, the state that does the volume setup doesn't need to know or care what's using said volume, and will be reused in other contexts. It seems inappropriate for either .sls to include the other. However, if the intended data directory happens to be a mounted volume, I need to make sure that the mount happens before the service state runs.
The problem: If neither file includes the other, I don't think I can use require or require_in. I can use order: 0 in the mounting state, and that will ensure the mount goes first, but that doesn't express a dependency; if the mount state fails, the service will still try to run.
What's the Right Way to handle this case? My aim is to create a requisite if and only if it is relevant, without either sls being dependent on the other's implementation details (or even existence).
I can imagine two possibilities:
use a wrapper state "data-with-mount" with include, in the correct order, the two needed states, which is only called when there is this double need
use a parameter (in pillar) like "data-with-mount: true" with you can then use in you state to include or not the require, something like
[...]
{% if salt['pillar.get']('data-with-mount', False) %}
require:
- sls: format-and-mount
{% endif %}
Not sure it completely suits your case but based on the information given I think it can be done that way.
Related
I have disabled template caching, via these settings:
when#dev:
twig:
auto_reload: true
cache: false
Here is the rub, I need my node visitors to run each time a template is extended.
I have a rare instance, where the same template is used several times, and the code is breaking because the assumption was made, they'd run every single time with the above configuration.
This appears to not be the case, which makes sense for a default setting. Why would you traverse a template which has already been traversed and visited?
In my case, I actually need this inefficient behaviour. Is there any way to tweak Twig service into always calling parse() on extended or included templates, which are invoked one or more times in a single request?
Again, I have several template includes which all extend from a simple base template, but the node visitors are only invoked on the first inclusion. The code is expecting to be run on every inclusion, regardless of how many times.
Hopefully I made sense :)
In apache airflow: What I want is to show a small web ui / form to a human to let him make a decision that we cannot implmement in code (yet). I haven't found anything in the docs or with google for that yet - so is there any architectural problem that would prevent that?
At this point in time, there is no such thing in Airflow.
(thinking out loud) You could set up a form yourself, which sets a True/False somewhere, such as a file or database. In Airflow, you'd then use a sensor to check whether True or False is set. A few examples; you could use the SqlSensor for a database query, the S3KeySensor for a file to be present on AWS S3, or the generic PythonSensor to provide your own condition.
Our team is transitioning to Robot Framework and we have some conceptual questions. Our biggest at the moment is how to set up a database insert/ delete dynamically, depending on the test we are trying to run. For example, we may have a test suite where we want to test an endpoint like so:
Record exists, endpoint is valid
Record exists with data flaw, error is handled correctly
Record does not exist, error is handled correctly
...etc
Each of these requires a different kind of document to be inserted into our database, then deleted, in set up/ tear down.
Currently, we're setting up each test as a separate suite and explicitly setting the insert statements (in this case, the JSON, since we're using MongoDB) in each Suite Setup keyword. We have some factory-method resources to help reduce redundancy but it's still very redundant. We are copying and pasting entire .robot files and changing a value or two in each one.
We have tried the data-driver library but we haven't been able to get it working with the variable scoping. And we can't set these set up/ tear down steps as simple test steps since we need to be sure that each document is created/ destroyed before and after each test.
In the docs, they say that you should avoid passing data between tasks:
This is a subtle but very important point: in general, if two operators need to share information, like a filename or small amount of data, you should consider combining them into a single operator. If it absolutely can’t be avoided, Airflow does have a feature for operator cross-communication called XCom that is described in the section XComs.
I fundamentally don't understand what they mean. If there's no data to pass between two tasks, why are they part of the same DAG?
I've got half a dozen different tasks that take turns editing one file in place, and each send an XML report to a final task that compiles a report of what was done. Airflow wants me to put all of that in one Operator? Then what am I gaining by doing it in Airflow? Or how can I restructure it in an Airflowy way?
fundamentally, each instance of an operator in a DAG is mapped to a different task.
This is a subtle but very important point: in general if two operators need to share
information, like a filename or small amount of data, you should consider combining them
into a single operator
the above sentence means that if you want any information that needs to be shared between two different tasks then it is best you could combine them into one task instead of using two different tasks, on the other hand, if you must use two different tasks and you need to pass some information from one task to another then you can do it using
Airflow's XCOM, which is similar to a key-value store.
In a Data Engineering use case, file schema before processing is important. imagine two tasks as follows :
Files_Exist_Check : the purpose of this task is to check whether particular files exist in a directory or not
before continuing.
Check_Files_Schema: the purpose of this task is to check whether the file schema matches the expected schema or not.
It would only make sense to start your processing if Files_Exist_Check task succeeds. i.e. you have some files to process.
In this case, you can "push" some key to xcom like "file_exists" with the value being the count of files present in that particular directory in Task Files_Exist_Check.
Now, you "pull" this value using the same key in Check_Files_Schema Task, if it returns 0 then there are no files for you to process hence you can raise exception and fail the task or handle gracefully.
hence sharing information across tasks using xcom does come in handy in this case.
you can refer following link for more info :
https://www.astronomer.io/guides/airflow-datastores/
Airflow - How to pass xcom variable into Python function
What you have to do for avoiding having everything in one operator is saving the data somewhere. I don't quite understand your flow, but if for instance, you want to extract data from an API and insert that in a database, you would need to have:
PythonOperator(or BashOperator, whatever) that takes the data from the API and saves it to S3/local file/Google Drive/Azure Storage...
SqlRelated operator that takes the data from the storage and insert it into the database
Anyway, if you know which files are you going to edit, you may also use jinja templates or reading info from a text file and make a loop or something in the DAG. I could help you more if you clarify a little bit your actual flow
I've decided that, as mentioned by #Anand Vidvat, they are making a distinction between Operators and Tasks here. What I think is that they don't want you to write two Operators that inherently need to be paired together and pass data to each other. On the other hand, it's fine to have one task use data from another, you just have to provide filenames etc in the DAG definition.
For example, many of the builtin Operators have constructor parameters for files, like the S3FileTransformOperator. Confusing documentation, but oh well!
I have a bitbake configuration that generates two partitions, the BOOT (FAT) partition containing uBoot, uEnv.txt, etc. and a root file system that gets mounted read-only. There may be instances where the filesystem isn't a separate partition, but rather a ramdisk, so I'm trying to enforce a design pattern that works in both instances:
What I'm trying to do is provide some of the files in the root filesystem as links to locations on the SD card. This way I can build a single SD card image and the minor edits for node IDs or names can be easily tweaked by end-users. So for example, if /etc/special_config.conf would be a useful one, then rather than store it on the read-only partition, a link would be created pointing back to the real file on the BOOT partition.
So far I've tried making a recipe that, for that case, does the following:
IMAGE_BOOT_FILES += "special_config.conf"
do_install () {
ln -s /media/BOOT/special_config.conf \
${D}${sysconfigdir}/special_config.conf
}
This doesn't seem to do anything. The IMAGE_BOOT_FILES doesn't collect the special_config.conf file into the BOOT partition, as if when the system image gets populated all of those changes get wiped out.
Has anyone seen a clever way to enforce this kind of behavior in BitBake?
If I understand correctly, you get your ${sysconfdir}/special_config.conf symlink in the image (via a package built from recipe mentioned), but you don't get the special_config.conf file on your BOOT partition using wic image fstype.
If that's the case, then the only problem is that you define IMAGE_BOOT_FILES in the package recipe, rather than defining it in the image recipe, because this variable is only evaluated at image build time. So drop that from your config file recipe and add it to the image recipe, it should work this way.