When creating a PartitionSetDefinition in Dagster you can pass in a 'mode' that will swap the resources used (for testing purposes you may want to use cloud storage in PROD but use local storage for local development
A mode requires you to specify a set of config values that are usually provided in an environment yaml file but when you create a PartitionSetDefinition like below you can only pass the mode. This is usually done by setting a preset on the pipeline and using that for the run but PartitionSetDefinition only allows the setting of a mode not a preset.
date_partition_set = PartitionSetDefinition(
name="date_partition_set",
pipeline_name="my_pipeline",
partition_fn=get_date_partitions,
run_config_fn_for_partition=run_config_for_date_partition,
mode="test"
)
How can you provide the necessary preset/environment values for this?
One way I've found to do this is to load the presets into the run config when the run config is created for each partition using some utilities that Dagster provides. Found this in some of their unit tests:
test_base.yaml has the typical preset configs corresponding to test mode.
from dagster.utils import file_relative_path, load_yaml_from_globs
def run_config_for_date_partition(partition):
date = partition.value
config_path = file_relative_path(__file__, os.path.join("../my_pkg/environments/", relative_path))
config_dict = load_yaml_from_globs(
config_path("test_base.yaml"),
)
table_name = "table1"
input_config = {"config": {"start_date": date, "table_name": table_name}}
config_dict["solids"] = {
"download_snow_incremental_table": {**input_config}
}
return config_dict
date_partition_set = PartitionSetDefinition(
name="date_partition_set",
pipeline_name="my_pipeline",
partition_fn=get_date_partitions,
run_config_fn_for_partition=run_config_for_date_partition,
mode="test"
)
Related
Using: telegraf version 1.23.1
Thats the workflow Telegraf => Influx => Grafana.
I am using telegraf to check my metrics on a shared server. So far so good, i already could initalize the Telegraf uWSGI Plugin and display the data of my running django projects in grafana.
Problem
Now i wanted to check some folder size too with the [[inputs.filecount]] Telegraf Plugin and this works also well. However i do not need Metrics for every 10s for this plugin. So i change the interval like mentioned in the Documentation in the [[inputs.filecount]] Plugin.
telegraf.conf
[agent]
interval = "10s"
round_interval = true
metric_batch_size = 1000
metric_buffer_limit = 10000
collection_jitter = "5s"
flush_interval = "10s"
flush_jitter = "0s"
#... PLUGIN
[[inputs.filecount]]
# set different interval for this input plugin every 10min
interval=“600s”
collection_jitter=“20s”
# Default from Doc =>
directories = ["/home/myserver/logs", "/home/someName/growingData, ]
name = "*"
recursive = true
regular_only = false
follow_symlinks = false
size = "0B"
mtime = "0s"
After restarting Telegram with Supervisor it crashed because it could not parse the new lines.
supervisor.log
Error running agent: Error loading config file /home/user/etc/telegraf/telegraf.conf: Error parsing data: line 208: invalid TOML syntax
So that are these lines i added because i thought that is how the Doc it mention it.
telegraf.conf
# set different interval for this input plugin every 10min
interval=“600s”
collection_jitter=“20s”
Question
So my question is. How can i change or setup the interval for a single input plugin in telegraf?
Or do i have to apply a different TOML syntax like [[inputs.filecount.agent]] or so?
I assume that i do not have to change any output interval also? Because i assume even though its currently 10s, if this input plugin only pulls/inputs data every 600s it should not matter, some flush cycle will push the Data to influx .
How can i change or setup the interval for a single input plugin in telegraf?
As the link you pointed to shows, individual inputs can set the interval and collection_jitter options. There is no difference in the TOML syntax for example I can do the following for the memory input plugin:
[[inputs.mem]]
interval="600s"
collection_jitter="20s"
I assume that i do not have to change any output interval also?
Correct, these are independent of each other.
line 208: invalid TOML syntax
Knowing what exactly is on line 208 and around that line will hopefully resolve your issue and get you going again. Also make sure your quotes that you used are correct. Sometimes when people copy and paste quotes they get ” vs " which can cause issues!
So I’m trying to make a development environment that’s easily reproducible (staying away from home-manager currently to understand Nix better). After enough searching around I figured out how to make a few custom derivations, use buildEnv for package sets, and use ~/.config/nixpkgs/config.nix to do overrides. I’m working now to setup zsh and oh-my-zsh which have a ton of configuration options, but the only documentation I can find seems to suggest adding them to configuration.nix, which is a NixOS option I can’t use.
Currently my config.nix code looks something like this:
let
pkgs = import <nixpkgs> {};
in {
allowUnfree = true;
programs = {
zsh = {
enable = true;
promptInit = "source ${pkgs.zsh-powerlevel9k}/share/zsh-powerlevel9k/powerlevel9k.zsh-theme";
ohMyZsh = {
enable = true;
plugins = ["autojump"];
theme = "powerlevel9k/powerlevel9k";
};
};
};
packageOverrides = pkgs: with pkgs; rec {
all = buildEnv {
name = "all";
paths = with pkgs; [
tmuxinator
zsh
oh-my-zsh
autojump
...
];
};
};
}
My understanding so far is that within ~/.config/nixpkgs/config.nix, there should be a single config set which contains things like the overrides function and corresponds to documentation examples of config.programs.zsh.enable, etc. However, nothing I write in that programs section affects or causes a different ouput of any of my programs.
What am I missing? How can I affect the configuration options listed here (https://github.com/NixOS/nixpkgs/blob/master/nixos/modules/programs/zsh/zsh.nix)?
You seem to be trying to use home-manager's config without using home-manager itself. As you can see in the NixOS module you linked, this actually sets up /etc/zshrc etc, so it's not intended for use in a user-local config and won't do anything there. If you look at the corresponding home-manager module, you'll see that it basically reimplements the whole module for user-local purposes. So you won't get far with this approach without relying on home-manager.
In my project, I want to get different environment value according to different OS, just like get A from Windows and get B from Linux, I try to use filter function like the code shown below:
filter {"system:windows"}
local value = os.getenv("A")
filter {"system:linux"}
local value = os.getenv("B")
or use configuration like that:
configuration {"windows"}
local value = os.getenv("A")
configuration {"linux"}
local value = os.getenv("B")
When I run the premake5.lua, it will return an error: attempt to concatenate a nil value.
Is there anything I misunderstand? How can I implement it correctly?
Another option:
if os.is("windows") then
...
else if os.is("macosx") then
...
else if os.is("linux") then
...
end
Use os.get to determine which platform you are currently running on.
if os.get() == "windows" then
...
end
There's the command sbt flywayMigrate from flywaydb.org. The command requires use to set flywayUrl, flywayUser, and flywayPassword beforehand. It was good so far.
Now I want to be able to use sbt flywayMigrate for two different environment; Their variables should be different.
I tried to make two new commands: sbt flywayMigrateDev and sbt flywayMigrateProd. I couldn't figure out how to connect the new commands to flywayMigrate.
I tried creating a new scope. But I couldn't figure out how to wire the variables and tasks properly.
I wonder if anyone can give me an example on how to do this. I'd like to see a code example.
We can simplify the problem to:
There's the command sbt flywayMigrate that depends on flywayUrl. How do we allow the command to use different flywayUrls by calling sbt commands (or any other way is good, too)?
Thank you!
You should use config for this.
Example .sbt file contents:
// Set up your configs.
lazy val prodConfig = config("prod")
lazy val devConfig = config("dev")
// Set up any configuration that's common between dev and prod.
val commonFlyway = Seq(
// For the sake of example, a couple of shared settings.
flywayUser := "pg_admin",
flywayLocations := Seq("filesystem:migrations")
)
// Set up prod and dev.
inConfig(prodConfig)(flywayBaseSettings(prodConfig) ++ commonFlyway)
flywayUrl.in(prodConfig) := "jdbc:etc:proddb.somecompany.com"
// Or however you want to load your production password.
flywayPassword.in(prodConfig) := sys.env.getOrElse("PROD_PASSWD", "(unset)")
inConfig(devConfig)(flywayBaseSettings(prodConfig) ++ commonFlyway)
flywayUrl.in(devConfig) := "jdbc:etc:devdb.somecompany.com"
flywayPassword.in(devConfig) := "development_passwd"
Now you can run prod:flywayMigrate and dev:flywayMigrate to migrate production and development, respectively.
See the Flyway docs page for other examples.
I have overridden the onProject function for the vs2012 action which generates some cpp files and then tries to include them in the project
--cant override the generateProject directly
--so have to override at the action level
premake.override( premake.action._list.vs2012, 'onProject', function(base, prj)
if premake.project.iscpp(prj) then
--generate files
--print( "Generating extra files ...")
local extraFiles = mine.getExtraFiles(prj)
for _,file in ipairs( extraFiles ) do
p.generate( file, nil, mine.generateExtraFile )
mine.addFileToSources(file)
end
end
--Generate regular stuff
base(prj)
end)
function mine.getExtraFiles(prj)
local extraFiles = {}
--works out what files to generate and add relevant info to table
return extraFiles
end
--this function is passed as a callback to premake.generate
function mine.generateExtraFile(extraFile)
--write contents of file
end
This is the function that attempts to add each generated file to the project
function mine.addFileToSources(extraFile)
local prj = extraFile.prj
local cfg = extraFile.cfg
local groups = premake.vstudio.vc2010.categorizeSources(prj)
local compiledFiles = groups.ClCompile or {}
--create a new file config for generated file
local filename = path.join(extraFile.location, extraFile.filename)
local fcfg = premake.fileconfig.new( filename, prj)
premake.fileconfig.addconfig(fcfg, cfg)
--add the config to the project's sources
table.insert(compiledFiles, fcfg)
compiledFiles[filename] = fcfg
--add to the projects source tree
--this bit is copied from premake.project.getsourcetree
-- The tree represents the logical source code tree to be displayed
-- in the IDE, not the physical organization of the file system. So
-- virtual paths are used when adding nodes.
-- If the project script specifies a virtual path for a file, disable
-- the logic that could trim out empty root nodes from that path. If
-- the script writer wants an empty root node they should get it.
local flags
if fcfg.vpath ~= fcfg.relpath then
flags = { trim = false }
end
-- Virtual paths can overlap, potentially putting files with the same
-- name in the same folder, even though they have different paths on
-- the underlying filesystem. The tree.add() call won't overwrite
-- existing nodes, so provide the extra logic here. Start by getting
-- the parent folder node, creating it if necessary.
local tr = premake.project.getsourcetree(prj)
local parent = premake.tree.add(tr, path.getdirectory(fcfg.vpath), flags)
local node = premake.tree.insert(parent, premake.tree.new(path.getname(fcfg.vpath)))
-- Pass through value fetches to the file configuration
setmetatable(node, { __index = fcfg })
end
For the most part - this all works:
The files are generated correctly and to correct location
The files are also included in the vcxproj file correctly
My problem is that the vcxproj.filters file is not being generated.
When I run premake I get this error:
Generating myproject.vcxproj.filters...Error: [string "src/actions/vstudio/vs2010_vcxproj_filters...."]:82: attempt to index field 'parent' (a nil value)
which corresponds to the function premake.vstudio.vc2010.filterGroup(prj, groups, group)
I get that the new fcfg I created needs to have a parent but I can't work out where or what I should be adding it to.
Can anyone help?
EDIT 1
I've got things working by adding this line to the end of function mine.addFileToSources(extraFile)
fcfg.parent = parent
This gives the file config a parent node so everything works out but I feel kinda dirty doing this so I'll look at following Citron's advice
EDIT 2
Overriding the bakefiles was much cleaner and neater. It wasn't as straightforward as Citron's code since I needed the information from the baked files in order to carry out my file generation but I am now confident that my code is correct and will possibly work with other exporters than vstudio too.
Here's my new code:
premake.override( premake.oven, 'bakeFiles', function(base, prj)
--bake the files as normal
local bakedFiles = base(prj)
if premake.project.iscpp(prj) then
--gather information about what files to generate and how
local extraFiles = mine.getExtraFiles(prj, bakedFiles)
for _,file in ipairs( extraFiles ) do
--do the generation
premake.generate( file, file.extension, mine.generateExtraFile )
--add the new files
local filename = premake.filename(file, file.extension)
table.insert(file.cfg.files, filename)
-- This should be the first time I've seen this file, start a new
-- file configuration for it. Track both by key for quick lookups
-- and indexed for ordered iteration.
assert( bakedFiles[filename] == nil )
local fcfg = premake.fileconfig.new(filename, file.prj)
bakedFiles[filename] = fcfg
table.insert(bakedFiles, fcfg)
premake.fileconfig.addconfig( bakedFiles[filename], file.cfg)
end
--sort the baked files again - since we have added to them
table.sort(bakedFiles, function(a,b)
return a.vpath < b.vpath
end)
end
return bakedFiles
end)
I don't know what the problem is with your code (a bit too much to read, and not enough time :p) but if you just want to add some generated files to your project tree, I would advise you to override premake.oven.bakeFiles instead.
This is what I used to add files generated by Qt in my addon. See premake.extensions.qt.customBakeFiles on https://github.com/dcourtois/premake-qt/blob/master/qt.lua
Basically in the bakeFiles override, you can just browse your projects, and insert files in the list easily. Then, if those added files need some custom configuration, you can then override premake.fileconfig.addconfig. See premake.extensions.qt.customAddFileConfig in the aforementioned addon.
In this addconfig override, you'll have access to the files and you will be able to modify their configuration object: you can add custom build rules, special options, etc.
It's not a direct answer to your specific question, but I hope it will help you achieve what you need.