There is a way to change the default ui_color for an airflow operator?
The only way I found is to write my own operator and inherit from the specific airflow operator, But it's too complicated for this simple change.
I dont know why, but the default colors are very similar, look at my own spark wrapper against the default python/kube/bash operators
Example image
thanks!
You can overwrite the value after importing the module, like below. You could then define you colorset once and apply import it to all your DAGs for instance:
from airflow.operators.oracle_operator import OracleOperator
OracleOperator.ui_color = '#0000FF'
### your DAG code below
Related
I wanted to use filemask in the GoogleCloudStoragePrefixSensor. I cant use the GoogleCloudStoragePrefixSensor because I also need to see the ending oif the file mask. BAsically my file is like "tv_link_input_*.xml". So, tried using GoogleCloudStorageObjectSensor, but its keep running without any OP. Using airflow 1.10.14. Code
check_file_gcs = GoogleCloudStorageObjectSensor(
task_id='check_file_gcs',
bucket=source_bucket,
object="processing/tv_link_input_*xml",
poke='10',
dag=dag
)
Thanks
I want to use an ipyvuetify widget called ProgressCircular to show the loading process. Therefore, I was trying to figure out how to show and hide the widget in my code.
progress=v.ProgressCircular(width=3,
color='red',
indeterminate=True,
)
Although I was able to see all the attributes with dir(), I still couldn't find the right one to use. How do people figure out how to use classes or functions in a package that lacks samples.
dir(v.ProgressCircular)
You can use display(progress) within an ipywidgets Output widget.
import ipyvuetify as v
import ipywidgets as ipyw
import time
progress=v.ProgressCircular(width=3,
color='red',
indeterminate=True,
)
output = ipyw.Output()
display(output)
with output:
display(progress)
time.sleep(2)
output.clear_output()
I will assume that you are working in a Jupyter environment :
after declaring your widget place it on the last line of your cell or use display as suggested by #ac24:
progress = v.ProgressCircular(
width = 3,
color = 'red',
indeterminate = True
)
progress
# alternatively
# display(progress)
once it's done you can play with it using some very basic html attributes
progress.class_ = 'd-none' # disapear
progress.class_ = None # shown
As you were complaining about the documentation, see here for the usage of HTML attributes https://ipyvuetify.readthedocs.io/en/latest/usage.html#setting-attributes, more examples would be useless as the possible combinations of html attributes are virtually infinite. Lucky for us vuetify.js is providing a very complete one that can be used in combination with the ipyvuetify one :
https://vuetifyjs.com/en/styles/display/
No need to use Output or styles for this, just make a container widget and change its children:
import ipyvuetify as v
import time
progress=v.ProgressCircular(width=3,
color='red',
indeterminate=True,)
container = v.Html(tag='div', children=[progress])
display(container)
time.sleep(2)
container.children=[v.Chip(children=['Done'])]
I am using a PythonOperator in my Airflow DAG and I need to print something inside the operator's Python function. I tried to print but apparently it didn't work out. Not so sure that's going to work. Next I tried to pass self.log in PythonOperator but I am not sure how to pass that reference.
task = PythonOperator(
task_id='task1',
python_callable=my_func,
params={ ... },
provide_context=True,
dag=dag
)
...
def my_func(**context):
...
print(some_message) # this didn't work.
The way you should be logging in Airflow (and in Python in general) is through the logging module, that is,
import logging
on top of the DAG definition and
logging.info(some_message)
in place of the print statement in your my_func function. Other functions you can use besides info() (with different logging level/criticality) can be found in the Python docs: https://docs.python.org/3/howto/logging.html#logging-basic-tutorial
How can I define a simple cell magic that just executes the cell as if the %%mymagic wasn't there?
The context is that we are using the wonderful IPython parallel framework. In some places, we also use its defined %%px magic. But sometimes we'd like to run the same notebook without a cluster (local only). In that case, %%px isn't defined and I would have to comment it out. Instead, in that case I'd like to redefine %%px so that:
%%px: would be a no-op.
%%px --local: just runs the cell, no other side-effect.
Alternatively, all %%px (with --local or not) could just run the cell, if that's simpler.
Another approach would be to create an ipyparallel Client that is a fake one, i.e. with 0 nodes (but would still operate correctly, e.g. with regard to %%px --local). But that would be for another question.
Things I've tried:
%alias_magic px time (after all, I don't care if the cell is timed). Unfortunately, %%time doesn't take arguments and chokes on --local.
Define my own "no-op" magic:
if USE_CLIENT:
pass
else:
# temporarily define %%px cell magic
from IPython import get_ipython
def px(line, cell):
"""Do nothing"""
pass
get_ipython().register_magic_function(px, 'cell')
but that succeeds a little too well at doing really nothing (i.e. the cells are not executed).
Look into IPython/core/magics/execution.py to see if there is any hook I could reuse (something that would just execute the cell). I haven't found, but perhaps I haven't looked hard enough.
Any other idea?
I think the relevant command is
self.shell.run_cell(cell)
We can define a magic command that just does not have effect as:
def f(line, cell):
print('==> line:[{}]'.format(line))
print('==> cell:\n # {}'.format('\n # '.join(cell.split())))
print('==================================================================')
res = get_ipython().run_cell(cell)
get_ipython().register_magic_function(f, 'cell', 'cache')
Here is an example run:
In your case, try:
if USE_CLIENT:
pass
else:
# temporarily define %%px cell magic
from IPython import get_ipython
def px(line, cell):
res = get_ipython().run_cell(cell)
get_ipython().register_magic_function(px,'cell','px')
I'm doing a content migration with collective.transmogrifier and I'm reading files off the file system with transmogrify.filesystem. Instead of importing the files "as is", I'd like to import them to a sub directory in Plone. What is the easiest way to modify the _path?
For example, if the following exists:
/var/www/html/bar/index.html
I'd like to import to:
/Plone/foo/bar/index.html
In other words, import the contents of "baz" to a subdirectory "foo". I see two options:
Use some blueprint in collective.transmogrifier to mangle _path.
Write some blueprint to mangle _path.
Am I missing anything easier?
Use the standard inserter blueprint to generate the paths; it accepts python expressions and can replace keys in-place:
[manglepath]
blueprint = collective.transmogrifier.sections.inserter
key = string:_path
value = python:item['_path'].replace('/var/www/html', '/Plone/foo')
This thus takes the output of the value python expression (which uses the item _path and stores it back under the same key.