During development of a new app based on Django, I noticed a memory corruption. I got two functions which use this class:
class ConfigMap():
data = list()
def add(self, entry: CusDeployPhone):
for row in self.data:
if row.phone_var.varid == entry.phone_var.varid:
return self
self.data.append(entry)
return self
def get(self):
return self.data
Function #1
def gen_config_model(request, deploy_phone):
deploy_phone_general = CusDeployPhone.objects.filter(phone_model=7)
config_list_model = ConfigMap()
for entry in deploy_phone:
config_list_model.add(entry)
for entry in deploy_phone_general:
config_list_model.add(entry)
Function #2
def gen_config_endpoint(request):
config_list_endpoint = ConfigMap()
for entry in deploy_model:
config_list_endpoint.add(entry)
for entry in deploy_phone_general:
config_list_endpoint.add(entry)
Both functions return the data in the list. I noticed that when calling the endpoint view, I also see the data when loading the model-one!
Somehow the class gets corrupted or merged with the existing one. Why?
The variable is local to the function.
I know there are issues with lists (references / pointers) but why in this case?
Issue solved:
Python Stack Corruption?
The children variable was declared as a class-level variable so it is
shared amongst all instances of your Nodes. You need to declare it an
instance variable by setting it in the initializer.
Change declaration to initializer.
def __init__(self):
self.children = []
...
Related
Is there a built-in facility or some operator that will run a sensor and negate its status? I am writing a workflow that needs to detect that an object does not exist in order to proceed to eventual success. I have a sensor, but it detects when the object does exist.
For instance, I would like my workflow to detect that an object does not exist. I need almost exactly S3KeySensor, except that I need to negate its status.
The use case you are describing is checking key in S3, if exist wait otherwise continue workflow. As you mentioned this is a Sensor use case. The S3Hook has function check_for_key that checks if key exist so all needed is just to wrap it with Sensor poke function..
A simple basic implementation would be:
from airflow.providers.amazon.aws.hooks.s3 import S3Hook
from airflow.sensors.base import BaseSensorOperator
class S3KeyNotPresentSensor(BaseSensorOperator):
""" Waits for a key to not be present in S3. """
template_fields: Sequence[str] = ('bucket_key', 'bucket_name')
def __init__(
self,
*,
bucket_key: str,
bucket_name: Optional[str] = None,
aws_conn_id: str = 'aws_default',
verify: Optional[Union[str, bool]] = None,
**kwargs,
):
super().__init__(**kwargs)
self.bucket_name = bucket_name
self.bucket_key = [bucket_key] if isinstance(bucket_key, str) else bucket_key
self.aws_conn_id = aws_conn_id
self.verify = verify
self.hook: Optional[S3Hook] = None
def poke(self, context: 'Context'):
return not self.get_hook().check_for_key(self.bucket_key, self.bucket_name)
def get_hook(self) -> S3Hook:
"""Create and return an S3Hook"""
if self.hook:
return self.hook
self.hook = S3Hook(aws_conn_id=self.aws_conn_id, verify=self.verify)
return self.hook
I ended up going another way. I can use the trigger_rule argument of (any) Task -- by setting it to one_failed or all_failed on the next task I can play around with the desired status.
For example,
file_exists = FileSensor(task_id='exists', timeout=3, poke_interval=1, filepath='/tmp/error', mode='reschedule')
sing = SmoothOperator(task_id='sing', trigger_rule='all_failed')
file_exists >> sing
It requires no added code or operator, but has the possible disadvantage of being somewhat surprising.
Replying to myself in the hope that this may be useful to someone else. Thanks!
How do I get a counter to increase within a command function? For example:
global counter
counter = 0
#client.command(pass_context=True)
async def pick(ctx):
counter += 1
Every time I try to do this, it gives me this error:
UnboundLocalError: local variable 'counter' referenced before assignment
I have tried so many ways to get this to work but I can not figure it out to save my life as well as my loved ones.
There are several ways to acomplish what you want.
For one you can, as mentioned in hopethatsacleanwet's answer, just global the variable name so you can access the one in the global scope and not the local scope.
#client.command()
async def pick():
global counter
counter += 1
you also could, as mentioned in benjin's answer, use a cog to bind the variable to a scope the function has access to.
class MyCog:
def __init__(self, bot):
self.bot = bot
self.counter = 0
#commands.command()
async def pick(self):
self.counter += 1
def setup(bot):
bot.add_cog(MyCog(bot))
you could even bind the counter to the bot
client.counter = 0
#client.command()
async def pick():
bot.counter += 1
I reccomend you read up on python's namespaces
You can try creating a cog with a class and using self.counter. You do this by creating a separate file that contains the Class, creating a setup function at the bottom, and then using load_extension in your main code where you run the bot. Example code below.
bot.py
from discord.ext import commands
client = commands.Bot(command_prefix='!')
client.load_extension('cog')
client.run('TOKEN')
cog.py
from discord.ext import commands
class TestCog:
def __init__(self, bot):
self.bot = bot
self.counter = 0
#commands.command()
async def pick(self):
self.counter += 1
await self.bot.say('Counter is now %d' % self.counter)
def setup(bot):
bot.add_cog(TestCog(bot))
The reason that error is occurring is because Python is trying to define counter in a local scope within the pick command. In order to access the global variable you need to "redefine" it as global in the local context. Changing the pick command to this will fix it:
#client.command(pass_context=True)
async def pick(ctx):
global counter
counter += 1
Just for learning purpose, I tried to set a dictionary as a global variable in accumulator the add function works well, but I ran the code and put dictionary in the map function, it always return empty.
But similar code for setting list as a global variable
class DictParam(AccumulatorParam):
def zero(self, value = ""):
return dict()
def addInPlace(self, acc1, acc2):
acc1.update(acc2)
if __name__== "__main__":
sc, sqlContext = init_spark("generate_score_summary", 40)
rdd = sc.textFile('input')
#print(rdd.take(5))
dict1 = sc.accumulator({}, DictParam())
def file_read(line):
global dict1
ls = re.split(',', line)
dict1+={ls[0]:ls[1]}
return line
rdd = rdd.map(lambda x: file_read(x)).cache()
print(dict1)
For anyone who arrives at this thread looking for a Dict accumulator for pyspark: the accepted solution does not solve the posed problem.
The issue is actually in the DictParam defined, it does not update the original dictionary. This works:
class DictParam(AccumulatorParam):
def zero(self, value = ""):
return dict()
def addInPlace(self, value1, value2):
value1.update(value2)
return value1
The original code was missing the return value.
I believe that print(dict1()) simply gets executed before the rdd.map() does.
In Spark, there are 2 types of operations:
transformations, that describe the future computation
and actions, that call for action, and actually trigger the execution
Accumulators are updated only when some action is executed:
Accumulators do not change the lazy evaluation model of Spark. If they
are being updated within an operation on an RDD, their value is only
updated once that RDD is computed as part of an action.
If you check out the end of this section of the docs, there is an example exactly like yours:
accum = sc.accumulator(0)
def g(x):
accum.add(x)
return f(x)
data.map(g)
# Here, accum is still 0 because no actions have caused the `map` to be computed.
So you would need to add some action, for instance:
rdd = rdd.map(lambda x: file_read(x)).cache() # transformation
foo = rdd.count() # action
print(dict1)
Please make sure to check on the details of various RDD functions and accumulator peculiarities because this might affect the correctness of your result. (For instance, rdd.take(n) will by default only scan one partition, not the entire dataset.)
For accumulator updates performed inside actions only, their value is
only updated once that RDD is computed as part of an action
I'm using pytest.mark to give my tests kwargs. However, if I use the same mark on both the class and a test within the class, the class's mark overrides the mark on the function when the same kwargs are used for both.
import pytest
animal = pytest.mark.animal
#animal(species='croc') # Mark the class with a kwarg
class TestClass(object):
#animal(species='hippo') # Mark the function with new kwarg
def test_function(self):
pass
#pytest.fixture(autouse=True) # Use a fixture to inspect my function
def animal_inspector(request):
print request.function.animal.kwargs # Show how the function object got marked
# prints {'species': 'croc'} but the function was marked with 'hippo'
Where'd my hippo go and how can I get him back?
There are unfortunately various pytest bugs related to this, I'm guessing you're running into one of them. The ones I found are related to subclassing which you don't do there though.
So I've been digging around in the pytest code and figured out why this is happening. The marks on the functions are applied to the function at import time but the class and module level marks don't get applied on the function level until test collection. Function marks happen first and add their kwargs to the function. Then class marks overwrite any same kwargs and module marks further overwrite any matching kwargs.
My solution was to simply create my own modified MarkDecorator that filters kwargs before they are added to the marks. Basically, whatever kwarg values get set first (which seems to always be by a function decorator) will always be the value on the mark. Ideally I think this functionality should be added in the MarkInfo class but since my code wasn't creating instances of that I went with what I was creating instances of: MarkDecorator. Note that I only change two lines from the source code (the bits about keys_to_add).
from _pytest.mark import istestfunc, MarkInfo
import inspect
class TestMarker(object): # Modified MarkDecorator class
def __init__(self, name, args=None, kwargs=None):
self.name = name
self.args = args or ()
self.kwargs = kwargs or {}
#property
def markname(self):
return self.name # for backward-compat (2.4.1 had this attr)
def __repr__(self):
d = self.__dict__.copy()
name = d.pop('name')
return "<MarkDecorator %r %r>" % (name, d)
def __call__(self, *args, **kwargs):
""" if passed a single callable argument: decorate it with mark info.
otherwise add *args/**kwargs in-place to mark information. """
if args and not kwargs:
func = args[0]
is_class = inspect.isclass(func)
if len(args) == 1 and (istestfunc(func) or is_class):
if is_class:
if hasattr(func, 'pytestmark'):
mark_list = func.pytestmark
if not isinstance(mark_list, list):
mark_list = [mark_list]
mark_list = mark_list + [self]
func.pytestmark = mark_list
else:
func.pytestmark = [self]
else:
holder = getattr(func, self.name, None)
if holder is None:
holder = MarkInfo(
self.name, self.args, self.kwargs
)
setattr(func, self.name, holder)
else:
# Don't set kwargs that already exist on the mark
keys_to_add = {key: value for key, value in self.kwargs.items() if key not in holder.kwargs}
holder.add(self.args, keys_to_add)
return func
kw = self.kwargs.copy()
kw.update(kwargs)
args = self.args + args
return self.__class__(self.name, args=args, kwargs=kw)
# Create my Mark instance. Note my modified mark class must be imported to be used
animal = TestMarker(name='animal')
# Apply it to class and function
#animal(species='croc') # Mark the class with a kwarg
class TestClass(object):
#animal(species='hippo') # Mark the function with new kwarg
def test_function(self):
pass
# Now prints {'species': 'hippo'} Yay!
imagine there is a set of objects, like:
class oven():
pass
class delta_t_pause():
pass
class caliberBox():
pass
class caliberRectl():
pass
class caliberRound():
pass
and so on. There is a class Button(QtWidgets.QPushButton) which receives two variables when evoked by mouse click. On variable carries an id the other data to be stored in the object.
What I want to do is, create an instance of the object matching the id. Assigning the data to it might be done by obj = Object(data), which is no big deal.
Does anyone has some useful ideas how to achieve that? I'm kinda stuck here.
class Button(QtWidgets.QPushButton):
def __init__(self, senderid, items, parent):
super().__init__(senderid, items, parent)
print(senderid)
self.obj = items
def mousePressEvent(self,event):
QtWidgets.QPushButton.mousePressEvent(self, event)
if event.button() == QtCore.Qt.LeftButton:
print(event.button(),' pressed')
is what I have so far...
Something like
if senderid is in list:
obj = Object(data)
but how do I select the right Object
All programming is done in Python-3.4 and PyQt5.5.
cheers,
Christian
You could keep a dictionary that maps ids to classes. Then you can use the dict to create your object.
objects = {4: oven, 8: delta_t_pause, 15: caliberBox}
if senderid in objects:
obj = objects[senderid](data)