Airflow KubernetesPodOperator: pass securityContext parameter - airflow

Anyone could give me an example on passing some parameters as "runAsNonRoot" when creating a pod through KubernetesPodOperator?
I've tried to dig through the documentation but it is not clear.

At current this does not appear to be supported in the operator.
You can see that the KubePodOp has an init that makes a PodGenerator. It then adds all the volumes and mounts to it before generating. This does not at any point call the only method in which you could pass a SecurityContext add_init_containerwhose documentation appears to have been cut off mid sentence.

You can create pods through KubernetesPodOperator such as in python format. Here is the list of all parameters you can pass through KubernetesPodOperator.
I could not find a specific example on how to pass this "runAsNonRoot" parameters through KubernetesPodOperator. In YAML format, pod security parameters are defined as following:
spec:
containers:
# specification of the pod’s containers
# ...
securityContext:
readOnlyRootFilesystem: true
runAsNonRoot: true

You can pass a dictionary to the KubernetesPodOperator constructor with the following content:
security_context = {"runAsNonRoot": True}
You can look up the keys and value datatypes that you can pass via this dict in class "V1SecurityContext" and the linked classes (/python3.6/site-packages/kubernetes/client/models/v1_security_context.py).

Related

What is GRAPHDB_CONTEXT_TEST in Graphdb?

I am trying to set a new repository in GraphDB. The npm documentation say that
GRAPHDB_CONTEXT_TEST = 'http://ont.enapso.com/repo'; where is this "http://ont.enapso.com/repo" coming from?
Also, why do we need { prefix: 'entest', iri: 'http://ont.enapso.com/test#' } ?
In the test repository, it is:
But I don't understand if the inside the quotes is just a string, or a link?
GRAPHDB_CONTEXT_TEST = 'http://ont.enapso.com/repo';
This is the global variable in which we have 'http://ont.enapso.com/repo', which is to define which named graph would be used.
{ prefix: 'entest', iri: 'http://ont.enapso.com/test#' }
These are the prefixes that we could pass in along with their IRI. They are used to perform SPARQL queries which require your prefixes.
We pass the IRI inside the quotes referred to as an internationalized resource identifier. It is used to identify uniquely.
You can also check the updated documentation of the pack available at
ENAPSO GraphDB Client
ENAPSO GraphDB Admin
Hope that answers your questions.

Trying to export dynamodb table variables from a serverless stack fails with intrinsic function !Ref

I have a working stack built with servereless framework which includes a dynamodb table (stack was already deployed successfully). I am trying to export the dynamo table's variables (name and arn basically) so these could be used in another stack I have deployed.
To achieve this I have the following:
in serverless.yml:
resources:
Resources:
AqDataTable: ${file(resources/AqDataTable.yml):AqDataTable}
Outputs:
AqDataTableName: ${file(resources/AqDataTable.yml):Outputs.AqDataTableName}
AqDataTableArn: ${file(resources/AqDataTable.yml):Outputs.AqDataTableArn}
(...)
custom:
AqDataTable:
name: !Ref AqDataTable
arn: !GetAtt AqDataTable.Arn
stream_arn: !GetAtt AqDataTable.StreamArn
in resources/AqDataTable.yml:
Outputs:
AqDataTableName:
Value: ${self:custom.AqDataTable.name}
Export:
Name: ${self:custom.AqDataTable.name}-Name
AqDataTableArn:
Value: ${self:custom.AqDataTable.arn}
Export:
Name: ${self:custom.AqDataTable.name}-Arn
When trying to deploy I get the following error:
Serverless Error ---------------------------------------
Trying to populate non string value into a string for variable ${self:custom.AqDataTable.name}. Please make sure the value of the property is a string.
The way I worked around this is by replacing AqDataTable.name value in the serverless.yml custom section from !Ref AqDataTable to a "harder-coded" value: AqDataTable-${self:provider.stage} but obviously this is a bad practice which I would like to avoid.
I'd appreciate any inputs on why this stack format invalidates the !Ref intrinsic function, or better ways to achieve what I am after here.
Many thanks!
In case anyone ever faces this issue:
After going over the docs one more time, apparently what I initially tried to do is not possible. According to CF docs:
For outputs, the value of the Name property of an Export can't use Ref
or GetAtt functions that depend on a resource.
Similarly, the ImportValue function can't include Ref or GetAtt
functions that depend on a resource.

Reusing salt state snippets

In my salt state files I have several occurrences of a pattern which consists of defining a remote repository and importing a gpg key file definition, e.g.
import_packman_gpg_key:
cmd.run:
- name: rpm --import http://packman.inode.at/gpg-pubkey-1abd1afb.asc
- unless: rpm -q gpg-pubkey-1abd1afb-54176598
packman-essentials:
pkgrepo.managed:
- baseurl: http://ftp.gwdg.de/pub/linux/misc/packman/suse/openSUSE_Tumbleweed/Essentials/
- humanname: Packman (Essentials)
- refresh: 1
require:
- cmd: import_packman_gpg_keygpg-pubkey-1abd1afb-54176598
I would like to abstract these away as a different state, e.g.
packman-essentials:
repo_with_key.managed:
- gpg_key_id: 1abd1afb-54176598
- gpg_key_src: http://packman.inode.at/gpg-pubkey-1abd1afb.asc
- repo_url: http://ftp.gwdg.de/pub/linux/misc/packman/suse/openSUSE_Tumbleweed/Essentials/
- repo_name: Packman (Essentials)
which will in turn expand to the initial declarations above. I've looked into custom salt states ( see https://docs.saltstack.com/en/latest/ref/states/writing.html#example-state-module ) but I only found references on how to create one using Python. I'm looking for one which is based only on state definitions, as writing code for my specific problem looks overkill.
How can I create a custom state which reuses the template I've been using to manage package repositories?
This is what macros are for
Here is an example of simple macros for some heavily used by me constructs
However in your example, why do you cmd.run to import key?
pkgrepo.managed seems to support gpgkey option to download the key

Alice Bundle : dynamic entity loop parameter

I use Hautelook AliceBundle which use Faker to generate fixtures with real world data.
In Alice, we can use parameters in our YAML file like below:
parameters:
pwd_parameter: anything
My\UserEntity:
#generate 10 users with password equals to my parameter
user_{1..10}:
password: '<{pwd_parameter}>'
Is it possible to use parameters to generate dynamic numbers of fixtures?
The solution I am looking for is to do this:
parameters:
pwd_parameter: anything
nb_users: 10
My\UserEntity:
#generate nb_users users with password equals to my parameter
user_{1..nb_users}: #<---THIS IS THE PROBLEM
password: '<{pwd_parameter}>
I tried:
user_{1.. nb_users }
user_{1.. <nb_users> }
user_{1..<{nb_users}> }
user_{1..<{nb_users}> }
which throws:
Warning: array_merge(): Argument #1 is not an array
How can I generate my number of entities dynamically?
It is not possible to do something like that since fixtures yaml configuration files do not get merged with symfony parameters.
Nelmio\Alice\Fixtures\Fixture\RangeName is the class used to validate and parse configuration from your example.
Take a look at RangeName::canBuild() and you'll see regular expression that validates yaml key.
You could create your own Builder Method that would randomize number of fixture rows inserted.

What is job.get() and job.getBoolean() in mapreduce

I am working on pdf document clustering over hadoop so I am learning mapreduce by reading some examples on internet.In wordcount examples have lines
job.get("map.input.file")
job.getboolean()
What is function of these functions?what is exactly map.input.file where is it to set? or is it just a name given to input folder?
Please post answer if anyone know.
For code see the following link
wordcount 2.0 example=http://hadoop.apache.org/docs/r1.0.4/mapred_tutorial.html
These are job configurations. i.e. set of configurations which are passed on to each mapper and reducer. Now, these configurations consist of well defined mapreduce/hadoop related configurations as well as user-defined configurations.
In your case, map.input.file is a pre-defined configuration and yes it is set to a comma separated list of all the paths you have set as input path.
While wordcount.skip.patterns is a custom configuration which is set as per user's input, and you may see this configuration to be set in run() as follows:
conf.setBoolean("wordcount.skip.patterns", true);
As for when to use get and when to use getBoolean, it should be self-explanatory, as whenever you want to set a value of type boolean you will use getBoolean and setBoolean to get and set the specific config value respectively. Similarly you have specific methods for other data types as well. If it is string then you may use get().

Resources