AirFlow DatabricksSubmitRunOperator does not take in notebook parameters - airflow

I am trying to trigger a notebook from Airflow. The notebook has parameters defined as widgets and I am trying to pass values to it through the notebook_params parameter and though it triggers, when I look at the job submitted, parameters do not seem to be passed.
E.g. code
new_cluster = {'spark_version': '6.5.x-cpu-ml-scala2.11',
'node_type_id': 'Standard_DS3_v2',
'num_workers': 4
}
notebook_task = DatabricksSubmitRunOperator(task_id='notebook_task',
json={'new_cluster': new_cluster,
'notebook_task': {
'notebook_path': '/Users/abc#test.com/Demo',
'notebook_parameters':'{"fromdate":"20200420","todate":"20200420", "datalakename":"exampledatalake", "dbname": "default", "filesystem":"refined" , "tablename":"ntcsegmentprediction", "modeloutputpath":"curated"}'
},
})
however, DatabricksRunNowOperator supports it, and it works
notebook_run = DatabricksRunNowOperator(task_id='notebook_task',
job_id=24,
notebook_params={"fromdate":"20200420","todate":"20200420", "datalakename":"exampledatalake", "dbname": "default", "filesystem":"refined" , "tablename":"ntcsegmentprediction", "modeloutputpath":"curated"}
)
In the documentation and source code of DatabricksSubmitRunOperator in here
it says it can take in a notebook_task. If it can, not sure why it can't take in parameters
What am I missing?
If more information is required, I can provide that as well.

You should use base_parameters instead of notebook_params
https://docs.databricks.com/dev-tools/api/latest/jobs.html#jobsnotebooktask

To use it with the DatabricksSubmitRunOperator you need to add it as an extra argument in the json parameter: ParamPair
notebook_task_params = {
'new_cluster': cluster_def,
'notebook_task': {
'notebook_path': 'path',
'base_parameters':{
"param1": "**",
"param2": "**"}
}
}
notebook_task = DatabricksSubmitRunOperator(
task_id='id***',
dag=dag,
trigger_rule=TriggerRule.ALL_DONE,
json=notebook_task_params)
And then you can just use the dbutils.widgets.get to retrieve the value or set defaults.
param1 = getArgument("param1", "default")
param2 = getArgument("param2", "default")
getArgument > (DEPRECATED) Equivalent to get

Related

I try to fetch data from firbase but I get this error.The method '[]' can't be unconditionally invoked because the receiver can be 'null'

'''
QuestionModel getQuestionModelFromDatasnapshot(
DocumentSnapshot questionSnapshot) {
QuestionModel questionModel = new QuestionModel();
questionModel.question = questionSnapshot.data()'question'];
List<String> options = [
questionSnapshot.data()["option1"],
questionSnapshot.data()["option2"],
questionSnapshot.data()["option3"],
questionSnapshot.data()["option4"]
];
options.shuffle();
questionModel.option1 = options[0];
questionModel.option2 = options[1];
questionModel.option3 = options[2];
questionModel.option4 = options[3];
questionModel.correctOption = questionSnapshot.data()["option1"];
questionModel.answered = false;
print(questionModel.correctOption.toLowerCase());
return questionModel;
}
'''
It looks like you're using dart with null safety on and you are accessing a list or map that might be null. It's hard to tell from your code, but I'm guessing the offending line is
questionModel.correctOption = questionSnapshot.data()["option1"];
It might be one of the other [] calls earlier in your code (such as
questionSnapshot.data()["option1"]), but the solution is the same.
The problem is that questionSnapshot.data() might return null, and the compiler is flagging it.
If you are sure the data is not null, you can use the ! operator, like this:
questionModel.correctOption = questionSnapshot.data()!["option1"];
Another solution is a conditional null-safe operator ?, with a default value. In this case, the code would look like this:
questionModel.correctOption = questionSnapshot.data()?["option1"]??"DefaultValue;
But I would recommend checking if the returned data is null. It's easier for others to read and can be logged more easily:
var data = questionSnapshot.data();
if (data != null) {
questionModel.correctOption = data["option1"];
}
else {
//do something in that case
}

Terraform: pass variable list to container environment variables

I am trying to find a way how to pass/map the list variable to kubernetes_deployment container environment variables. I run an asp.net core application in the container and I trying to modify the app with setting env variables to override the app settings.json. It is not a problem to override single values from settings.json, the problem is if I need to define/override the whole array there.
I have variable list like this in terrafrom:
variable "allowed_cars" {
type = list(
object({
manufacturer = string
model = string
})
)
}
Then, I have resource definition of kubernetes_deployment with containers.
I believe it could work if I set the env variables for the container like this:
env {
name = "App__AllowedCars__0__Manufacturer"
value = "xxx"
}
env {
name = "App__AllowedCars__0__Model"
value = "xxx"
}
env {
name = "App__AllowedCars__1__Manufacturer"
value = "xxx"
}
env {
name = "App__AllowedCars__1__Model"
value = "xxx"
}
...
Is there a way how to pass these env variables to the container in a dynamic way based on allowed_cars terraform variable? I don't know how many items will be defined for each environments, etc...
Thank you very much.
Something of this definition (used example of azurerm_container_group in azure)
environment_variables = "${merge(var.env_vars,var.secure_env_vars,local.master_env)}"
and then variables can be passed in
variable "env_vars" {
type = "map"
description = "envvaars"
default = {
WEB_USER = "locust"
HATCH_RATE = 25
LOCUST_COUNT = 50
LOCUST_FILE = "/locust/locustfile.py"
ATTACKED_HOST = "https://api-perf.yrdy.com"
}
}
variable "secure_env_vars" {
type = "map"
description = "secure env vars"
default = {
WEB_PASSWORD = "dummy"
API_KEY = "test"
}
}
As #ydaetskcoR mentioned, the dynamic block was solution for me.
https://www.terraform.io/docs/language/expressions/dynamic-blocks.html
dynamic "env" {
for_each = var.allowed_cars
content {
name = "App__AllowedCars__0__${env.key}__Manufacturer"
value = env.value["manufacturer"]
}
}

Intersystems caché - relationale mapping (custom sql storage)

I have more globals in caché db with same data structure. For each global I defined class with SQL storage map, but I need to do it generically for all globals. Is it possible to define one class with sql storage map which will be used for mapping before every SQL query execution? I need to avoid class declaration for each global which I need to be accessible via SQL. I use ODBC for execute SQL statements.
If someone can help me, i will very appreciate it
My globals looks like this:
^glob1("x","y","SL",1) = "Name"
^glob1("x","y","SL",1,"Format") = "myFormat"
^glob1("x","y","SL",1,"Typ") = "my Type"
^glob1("x","y","SL",2) = "Name2"
^glob1("x","y","SL",2,"Format") = "myFormat2"
^glob1("x","y","SL",2,"Typ") = "Type2"
^nextGlob("x","y","SL",1) = "Next Name"
^nextGlob("x","y","SL",1,"Format") = "Next myFormat"
^nextGlob("x","y","SL",1,"Typ") = "my Type"
^another("x","y","SL",13) = "Another Name"
^another("x","y","SL",13,"Format") = "Another myFormat"
^another("x","y","SL",13,"Typ") = "Another Type"
I want to have sql access to globals using one ObjectScript class.
If you needed only read data from Caché by ODBC. So, in ODBC you can use CALL statement. And you can write some SqlProc, which can be called by ODBC.
As I can see, all of your globals with the same structure. If it so, it will be easy. You can put something like this, in your class.
Query Test() As %Query(ROWSPEC = "ID:%String,Global:%String,Name:%String,Typ:%String,Format:%String") [ SqlProc ]
{
}
ClassMethod TestExecute(ByRef qHandle As %Binary) As %Status
{
#; Initial settings
#; List of Globals
set $li(qHandle,1)=$lb("glob1","nextGlob","another")
#; Current Global index
set $li(qHandle,2)=1
#; Current ID in global
set $li(qHandle,3)=""
Quit $$$OK
}
ClassMethod TestClose(ByRef qHandle As %Binary) As %Status [ PlaceAfter = TestExecute ]
{
Quit $$$OK
}
ClassMethod TestFetch(ByRef qHandle As %Binary, ByRef Row As %List, ByRef AtEnd As %Integer = 0) As %Status [ PlaceAfter = TestExecute ]
{
set globals=$lg(qHandle,1)
set globalInd=$lg(qHandle,2)
set id=$lg(qHandle,3)
set AtEnd=1
for {
set global=$lg(globals,globalInd)
quit:global=""
set globalData="^"_global
set globalData=$na(#globalData#("x","y","SL"))
set id=$o(#globalData#(id),1,name)
if id'="" {
set AtEnd=0
set typ=$get(#globalData#(id,"Typ"))
set format=$get(#globalData#(id,"Format"))
set Row=$lb(id,global,name,typ,format)
set $li(qHandle,3)=id
quit
} elseif $i(globalInd) {
set id=""
set $li(qHandle,2)=globalInd
}
}
Quit $$$OK
}
And then you can execute statement like this
CALL pkg.classname_test()
And as a result it will be something like on this picture
If all of the globals are the same then you could do this but it is likely that your globals are all different making a single storage map unlikely. Do you already have a data dictionary/meta data system that describes your existing globals? If so I would consider writing a conversion from your existing data dictionary definition to cache classes.

How to make a test fail with the testthat package?

I have a test for which if the prerequisites are not met (e.g., missing file or something) I would like to make it fail.
Just for clarification, here's an example I'd like to do:
test_that("...", {
if ( ... precondition to execute the test is not met... ) {
expect_true(FALSE) # Make it fail without going further
}
expect_that( ... real test here ...)
})
Now my question is: Is there any fail()-like expectation in the testthat package or I have to write expect_true(FALSE) all the time?
There isn't a fail function in testthat at the moment. I think you want something like
fail <- function(message = "Failure has been forced.", info = NULL, label = NULL)
{
expect_that(
NULL,
function(message)
{
expectation(FALSE, message)
},
info,
label
)
}
Usage is, for example,
test_that("!!!", fail())
Failure is not an option...
Try using stop:
test_that("testingsomething", {
if(file.exists("foo.txt")){
stop("foo.txt already exists")
}
foo = writeFreshVersion("foo.txt")
expect_true(file.exists("foo.txt"))
}
)

Getting a cmdlet's dynamic parameters via reflection

Powershell exposes some parameters, "dynamic parameters", based on context. The MSDN page explains the mechanism pretty well, but the skinny is that to find out about these one must call GetDynamicParameters(), which returns a class containing the additional parameters. I need to get these parameters via reflection, and (here's the crux of it), in a ReflectionOnly context (that is, the types are loaded with ReflectionOnlyLoadFrom). So, no Assembly.InvokeMember("GetDynamicParameters").
Can this be done?
No. Reflection works against static assembly metadata. Dynamic parameters in powershell are added at runtime by the command or function itself.
Perhaps this helps:
1: Defintion of the dynamic parameters
#===================================================================================
# DEFINITION OF FREE FIELDS USED BY THE CUSTOMER
#-----------------------------------------------------------------------------------
# SYNTAX: #{ <FF-Name>=#(<FF-Number>,<isMandatory_CREATE>,<isMandatory_UPDATE>); }
$usedFFs = #{
"defaultSMTP"=#(2,1,0); `
"allowedSMTP"=#(3,1,0); `
"secondName"=#(100,1,0); `
"orgID"=#(30001,1,0); `
"allowedSubjectTypeIDs"=#(30002,1,0); `
}
# FF-HelpMessage for input
$usedFFs_HelpMSG = #{ 2="the default smtp domain used by the organizaiton. Sampel:'algacom.ch'"; `
3="comma seperated list of allowed smtp domains. Sampel:'algacom.ch,basel.algacom.ch'"; `
100="an additional organization name. Sampel:'algaCom AG')"; `
30001="an unique ID (integer) identifying the organization entry"; `
30002="comma seperated list of allowed subject types. Sampel:'1,2,1003,10040'"; `
}
2: definition of function that builds the dynamic parameters
#-------------------------------------------------------------------------------------------------------
# Build-DynParams : Used to build the dynamic input parameters based on $usedFFs / $usedFFs_HelpMSG
#-------------------------------------------------------------------------------------------------------
function Build-DynParams($type) {
$paramDictionary = New-Object -Type System.Management.Automation.RuntimeDefinedParameterDictionary
foreach($ffName in $usedFFs.Keys) {
$ffID = $usedFFs.Item($ffName)[0]
$dynAttribCol = New-Object -Type System.Collections.ObjectModel.Collection[System.Attribute]
$dynAttrib = New-Object System.Management.Automation.ParameterAttribute
$dynAttrib.ParameterSetName = "__AllParameterSets"
$dynAttrib.HelpMessage = $usedFFs_HelpMSG.Item($ffID)
switch($type) {
"CREATE" { $dynAttrib.Mandatory = [bool]($usedFFs.Item($ffName)[1]) }
"UPDATE" { $dynAttrib.Mandatory = [bool]($usedFFs.Item($ffName)[2]) }
}
$dynAttribCol.Add($dynAttrib)
$dynParam = New-Object -Type System.Management.Automation.RuntimeDefinedParameter($ffName, [string], $dynAttribCol)
$paramDictionary.Add($ffName, $dynParam)
}
return $paramDictionary
}
3. Function that makes use of the dynamic params
#-------------------------------------------------------------------------------------------------------
# aAPS-OrganizationAdd : This will add a new organization entry
#-------------------------------------------------------------------------------------------------------
Function aAPS-OrganizationAdd {
[CmdletBinding()]
Param(
[Parameter(Mandatory=$true,HelpMessage="The name of the new organization")]
[String]$Descr,
[Parameter(Mandatory=$false,HelpMessage="The name of the parent organization")]
[String]$ParentDescr=$null,
[Parameter(Mandatory=$false,HelpMessage="The status of the new organization [1=Active|2=Inactive]")]
[int]$Status = 1,
[Parameter(Mandatory=$false,HelpMessage="If you want to see the data of the deactivated object")]
[switch]$ShowResult
)
DynamicParam { Build-DynParams "CREATE" }
Begin {}
Process {
# do what oyu want here
}
End {}
}

Resources