How can I use OmegaConf custom interpolation with Hydra - fb-hydra

How can I use OmegaConf custom interpolation with Hydra?
Some background:
One can define a custom interpolation for square root:
from omegaconf import OmegaConf
import math
OmegaConf.register_resolver("sqrt", lambda x: math.sqrt(float(x)))
And use it with this config.yaml:
foo: ${sqrt:9}
Loading and printing foo:
cfg = OmegaConf.load('config.yaml')
print(cfg.foo)
Outputs 3.0
When trying this with Hydra:
import hydra
#hydra.main(config_path="config.yaml")
def main(cfg):
print(cfg.foo)
if __name__ == "__main__":
main()
I get the following error:
Unsupported interpolation type sqrt
full_key: foo
reference_type=Optional[Dict[Any, Any]]
object_type=dict
How can I register my resolver when using Hydra?

You can register your custom resolver ahead of time:
config.yaml:
foo: ${sqrt:9}
main.py:
from omegaconf import OmegaConf
import math
import hydra
OmegaConf.register_new_resolver("sqrt", lambda x: math.sqrt(float(x)))
#hydra.main(config_path=".", config_name="config")
def main(cfg):
print(cfg.foo)
if __name__ == "__main__":
main()
This will print 3.0.
This approach will work just as well with the Compose API. The evaluation of the custom resolver is happening when you access the node (lazily).
You just need to register the resolver before you access it.

Related

How to load Hydra parameters from previous jobs (without having to use argparse and the compose API)?

I'm using Hydra for training machine learning models. It's great for doing complex commands like python train.py data=MNIST batch_size=64 loss=l2. However, if I want to then run the trained model with the same parameters, I have to do something like python reconstruct.py --config_file path_to_previous_job/.hydra/config.yaml. I then use argparse to load in the previous yaml and use the compose API to initialize the Hydra environment. The path to the trained model is inferred from the path to Hydra's .yaml file. If I want to modify one of the parameters, I have to add additional argparse parameters and run something like python reconstruct.py --config_file path_to_previous_job/.hydra/config.yaml --batch_size 128. The code then manually overrides any Hydra parameters with those that were specified on the command line.
What's the right way of doing this?
My current code looks something like the following:
train.py:
import hydra
#hydra.main(config_name="config", config_path="conf")
def main(cfg):
# [training code using cfg.data, cfg.batch_size, cfg.loss etc.]
# [code outputs model checkpoint to job folder generated by Hydra]
main()
reconstruct.py:
import argparse
import os
from hydra.experimental import initialize, compose
if __name__ == '__main__':
parser = argparse.ArgumentParser()
parser.add_argument('hydra_config')
parser.add_argument('--batch_size', type=int)
# [other flags and parameters I may need to override]
args = parser.parse_args()
# Create the Hydra environment.
initialize()
cfg = compose(config_name=args.hydra_config)
# Since checkpoints are stored next to the .hydra, we manually generate the path.
checkpoint_dir = os.path.dirname(os.path.dirname(args.hydra_config))
# Manually override any parameters which can be changed on the command line.
batch_size = args.batch_size if args.batch_size else cfg.data.batch_size
# [code which uses checkpoint_dir to load the model]
# [code which uses both batch_size and params in cfg to set up the data etc.]
This is my first time posting, so let me know if I should clarify anything.
If you want to load the previous config as is and not change it, use OmegaConf.load(file_path).
If you want to re-compose the config (and it sounds like you do, because you added that you want override things), I recommend that you use the Compose API and pass in parameters from the overrides file in the job output directory (next to the stored config.yaml), but concatenate the current run parameters.
This script seems to be doing the job:
import os
from dataclasses import dataclass
from os.path import join
from typing import Optional
from omegaconf import OmegaConf
import hydra
from hydra import compose
from hydra.core.config_store import ConfigStore
from hydra.core.hydra_config import HydraConfig
from hydra.utils import to_absolute_path
# You can also use a yaml config file instead of this Structured Config
#dataclass
class Config:
load_checkpoint: Optional[str] = None
batch_size: int = 16
loss: str = "l2"
cs = ConfigStore.instance()
cs.store(name="config", node=Config)
#hydra.main(config_path=".", config_name="config")
def my_app(cfg: Config) -> None:
if cfg.load_checkpoint is not None:
output_dir = to_absolute_path(cfg.load_checkpoint)
original_overrides = OmegaConf.load(join(output_dir, ".hydra/overrides.yaml"))
current_overrides = HydraConfig.get().overrides.task
hydra_config = OmegaConf.load(join(output_dir, ".hydra/hydra.yaml"))
# getting the config name from the previous job.
config_name = hydra_config.hydra.job.config_name
# concatenating the original overrides with the current overrides
overrides = original_overrides + current_overrides
# compose a new config from scratch
cfg = compose(config_name, overrides=overrides)
# train
print("Running in ", os.getcwd())
print(OmegaConf.to_yaml(cfg))
if __name__ == "__main__":
my_app()
~/tmp$ python train.py
Running in /home/omry/tmp/outputs/2021-04-19/21-23-13
load_checkpoint: null
batch_size: 16
loss: l2
~/tmp$ python train.py load_checkpoint=/home/omry/tmp/outputs/2021-04-19/21-23-13
Running in /home/omry/tmp/outputs/2021-04-19/21-23-22
load_checkpoint: /home/omry/tmp/outputs/2021-04-19/21-23-13
batch_size: 16
loss: l2
~/tmp$ python train.py load_checkpoint=/home/omry/tmp/outputs/2021-04-19/21-23-13 batch_size=32
Running in /home/omry/tmp/outputs/2021-04-19/21-23-28
load_checkpoint: /home/omry/tmp/outputs/2021-04-19/21-23-13
batch_size: 32
loss: l2

Url Persian not found in django/passenger

sample Url:
site.com/category/%D9%81%D8%AA%D9%88%DA%AF%D8%B1%D8%A7%D9%81%DB%8C_%D8%B1%DB%8C%D9%86%D9%88%D9%BE%D9%84%D8%A7%D8%B3%D8%AA%DB%8C/
url config:
url(r'^category/(?P<page_slug>.*)/$', views.category, name='category'),
passenger config:
import imp
import os
import sys
sys.path.insert(0, os.path.dirname(__file__))
wsgi = imp.load_source('wsgi', 'photography/wsgi.py')
application = wsgi.application
wsgi config:
"""
WSGI config for photography project.
It exposes the WSGI callable as a module-level variable named ``application``.
For more information on this file, see
https://docs.djangoproject.com/en/1.10/howto/deployment/wsgi/
"""
import os
from django.core.wsgi import get_wsgi_application
os.environ.setdefault("DJANGO_SETTINGS_MODULE", "photography.settings")
#os.environ["DJANGO_SETTINGS_MODULE"] = "photography.settings"
application = get_wsgi_application()
but response 404 not found in url!
in problem for all url persian slug.
when change config wsgi to unquote:
from urllib.parse import unquote
def application(environ, start_fn):
environ['PATH_INFO'] = unquote(environ['PATH_INFO'])
app = get_wsgi_application()
print(environ)
return app(environ, start_fn)
change url into:
site.com/category/%C3%99%C2%81%C3%98%C2%AA%C3%99%C2%88%C3%9A%C2%AF%C3%98%C2%B1%C3%98%C2%A7%C3%99/tag/%D9%81%D8%AA%D9%88%DA%AF%D8%B1%D8%A7%D9%81%DB%8C-%D9%BE%D8%B2%D8%B4%DA%A9%DB%8C/
But there is an open problem !
I applied all the changes I found with the search, but there is still a problem!
Output one of the changes in the wsgi:
App 3585081 output: set_script_prefix(get_script_name(environ))
App 3585081 output: File "/home/sepandteb/virtualenv/sepandteb/3.5/lib/python3.5/site-packages/django/core/handlers/wsgi.py", line 210, in get_script_name
App 3585081 output: return script_name.decode(UTF_8)
App 3585081 output: UnicodeDecodeError: 'utf-8' codec can't decode byte 0xc3 in position 27: unexpected end of data
use encode('utf-8') for Persian character and set # -*- coding: utf-8 -*- on top of file.
for example when you save data on DB or read on DB use this function.
i.e :
slug.encode('utf-8')
You can use "uri_to_iri" in the view, for example:
from django.shortcuts import get_object_or_404
from django.utils.encoding import uri_to_iri
def blog_detail(request, slug):
post= get_object_or_404(Post, slug=uri_to_iri(slug))

from numexpr import evaluate on Quantopian

I'm trying to get some technical ind. with some of those commands in this link:https://github.com/enigmampc/catalyst/blob/master/catalyst/pipeline/factors/equity/technical.py,
but in the quant.notebook I'm not able to get "from numexpr import evaluate", so evaluate is not defined.
How can I solve this?
from numexpr import evaluate
class FastochasticOscillator(CustomFactor):
inputs=(USEquityPricing.close,USEquityPricing.high,USEquityPricing.low)
window_safe=True
window_length=14
def compute(self, today, assets, out, closes, highs, lows):
highest_high= nanmax(highs, axis=0)
lowest_low= nanmin(lows, axis=0)
latest_close= closes[-1]
evaluate(
'((tc - ll) / (hh - ll)) * 100',
local_dict={
'tc':latest_close,
'll':lowest_low,
'hh':highest_high,
},
global_dict={},
out=out,
)
K= FastochasticOscillator(window_length=14)
return Pipeline(columns={
'K':K,
},screen=base)
I'm working on the Quantopian notebook and when I attempt to import it gives me this: InputRejected: Importing evaluate from numexpr raised an ImportError. Did you mean to import errstate from numpy?
Actually I do not find a way to import numexpr on Quantopian but on Jupyter it do not give problems. Therefore the problem is related to the online IDE. Moreover, I simply re-write the FastOsc ind. in another way to use it inside the pipeline in the quantopian online IDE.
class Fast(CustomFactor):
inputs=(USEquityPricing.close,USEquityPricing.high,USEquityPricing.low)
window_length=14
def compute(self, today, assets, out, close, high, low):
highest_high= nanmax(high, axis=0)
lowest_low= nanmin(low, axis=0)
latest_close= close[-1]
out[:]= ((latest_close - lowest_low) / (highest_high - lowest_low)*100)

How do i import math into kvlang?

how do I import math so that I can assign a labeltext to math.pi?????????
I am only going to provide .kv snippet
#: import math
… some code
Label:
id: Label10
text: str(sin(int(Label2.text) * math.pi/180)
The error:
raise ParserException(self, ln, 'Invalid import syntax')
ParserException: Parser: File "*******************", line 2:
...
1:#: kivy 1.10.1
2:#: import math
3:
4::
...
Invalid import syntax
The syntax is #:import imported_name imported_object, so you could do #:import math math to import the math module with the name math.
This seems trivial in this case, but also extends to things like #:import sin math.sin with the same syntax.

Imports and ill-behaved SBT plugins

I'm trying to define parsers in a build.sbt file.
I'm using this plugin, by adding this line in plugins.sbt:
addSbtPlugin("com.gilt" % "sbt-dependency-graph-sugar" % "0.7.4")
When specifiying this in build.sbt:
import sbt.complete.DefaultParsers._
val servers = token(
literal("desarrollo") |
literal("parametrizacion")
)
SBT complains with this error message:
reference to literal is ambiguous;
reference to token is ambiguous;
it is imported twice in the same scope by
import sbt.complete.DefaultParsers._
and import _root_.gilt.DependencyGraph._
How can I avoid this namespace clashing of basic SBT classes?.
One solution is this:
import sbt.complete.{DefaultParsers ⇒ DP}
import sbt.complete.DefaultParsers._
val servers = DP.token(
DP.literal("desarrollo") |
DP.literal("parametrizacion")
)
I don't fully like it, because it adds clutter.
The ideal solution is to hide unwanted imports.
This solution creates a new scope, so there's no interference with imports:
name := "MyProject"
{
import sbt.complete.DefaultParsers._
val servers = token(
"desarrollo" | "parametrizacion"
)
}

Resources