How to debug "TypeError: Can't pickle objects in acquisition wrappers." in Plone - plone

I have a handler that adds a Member to a Group. The last line in this handler causes an error:
TypeError: Can't pickle objects in acquisition wrappers.
> /home/mnieber/.buildout/eggs/ZODB3-3.10.3-py2.6-linux-i686.egg/ZODB/serialize.py(431)_dump()
430 self._p.dump(classmeta)
--> 431 self._p.dump(state)
432 self._file.truncate()
In the pdb debugger I can see that indeed Plone is trying to pickle a value that is an Acquisition wrapper:
ipdb> state
((((<PloneUser 'newuser#usecm.com'>, ('Default_Group',), 'maarten#usecm.com', ('PAS',)),),),)
ipdb> type(state[0][0][0][0])
<type 'Acquisition.ImplicitAcquisitionWrapper'>
However, I cannot see which object is being pickled, and therefore I have no idea which part of my code needs fixing. My question is: how should I go about debugging this error? I have tried looking at all the stack frames, but none of them reveal which object is being serialized.
The handler is this one (run_insecure is a decorator that I use to temporarily install a new security manager that avoids a NotAuthorized error when adding the new member):
#adapter(IPrincipalCreatedEvent)
#run_insecure
def userCreatedHandler(event):
portal_groups = getToolByName(getSite(), "portal_groups")
membersGroup = portal_groups.getGroupById('Default_Group')
membersGroup.addMember(event.principal)
The full error is this one:
Traceback (innermost last):
Module ZPublisher.Publish, line 134, in publish
Module Zope2.App.startup, line 301, in commit
Module transaction._manager, line 89, in commit
Module transaction._transaction, line 329, in commit
Module transaction._transaction, line 443, in _commitResources
Module ZODB.Connection, line 567, in commit
Module ZODB.Connection, line 623, in _commit
Module ZODB.Connection, line 658, in _store_objects
Module ZODB.serialize, line 422, in serialize
Module ZODB.serialize, line 431, in _dump
TypeError: Can't pickle objects in acquisition wrappers.
> /home/mnieber/.buildout/eggs/ZODB3-3.10.3-py2.6-linux-i686.egg/ZODB/serialize.py(431)_dump()
430 self._p.dump(classmeta)
--> 431 self._p.dump(state)
432 self._file.truncate()

I got this kind of problem with pickle, and solved by debugging like you did.
Pickle (used to store objects in ZODB) is trying to serialize your PloneUser, and raising this acquisition wrapper error.
In my case, I was wrapping the portal_workflow object into another class, and had to inherit it from pickle.Pickler, and override __getstate__ method to solve my problem.
This method is called by pickle in order to serialize your object. If you override this method, and return your object.__dict__ without this PloneUser, than this error would not be raised.
This question (although with not your exact problem) have more info about what I'm trying to say.
Nice you could solve your problem.

Related

Using Camembert pre-trained model with DeepPavlov

I'm learning how to use DeepPavlov and can't figure how to use it for NER using the Camembert (french) pre-trained model. My goal is to tag a short paragraph in french.
The docs from deeppavlov explicitly list the Camembert model as a viable transformer architecture. I tried to follow as best as I could but I keep getting this error when I try to build the model.
>>> ner_model = build_model('ner_ontonotes_bert_mult')
/home/philippe/.local/lib/python3.10/site-packages/torch/nn/init.py:405: UserWarning: Initializing zero-element tensors is a no-op
warnings.warn("Initializing zero-element tensors is a no-op")
Some weights of the model checkpoint at camembert-base were not used when initializing CamembertForTokenClassification: ['lm_head.dense.weight', 'lm_head.bias', 'lm_head.layer_norm.bias', 'lm_head.layer_norm.weight', 'lm_head.decoder.weight', 'lm_head.dense.bias']
- This IS expected if you are initializing CamembertForTokenClassification from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).
- This IS NOT expected if you are initializing CamembertForTokenClassification from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).
Some weights of CamembertForTokenClassification were not initialized from the model checkpoint at camembert-base and are newly initialized: ['classifier.weight', 'classifier.bias']
You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.
2023-01-29 09:58:13.308 ERROR in 'deeppavlov.core.common.params'['params'] at line 108: Exception in <class 'deeppavlov.models.torch_bert.torch_transformers_sequence_tagger.TorchTransformersSequenceTagger'>
Traceback (most recent call last):
File "/home/philippe/.local/lib/python3.10/site-packages/deeppavlov/core/common/params.py", line 102, in from_params
component = obj(**dict(config_params, **kwargs))
File "/home/philippe/.local/lib/python3.10/site-packages/deeppavlov/models/torch_bert/torch_transformers_sequence_tagger.py", line 182, in __init__
super().__init__(optimizer=optimizer,
File "/home/philippe/.local/lib/python3.10/site-packages/deeppavlov/core/models/torch_model.py", line 98, in __init__
self.load()
File "/home/philippe/.local/lib/python3.10/site-packages/deeppavlov/models/torch_bert/torch_transformers_sequence_tagger.py", line 295, in load
self.crf = CRF(self.n_classes).to(self.device)
File "/home/philippe/.local/lib/python3.10/site-packages/deeppavlov/models/torch_bert/crf.py", line 13, in __init__
super().__init__(num_tags=num_tags, batch_first=batch_first)
File "/home/philippe/.local/lib/python3.10/site-packages/torchcrf/__init__.py", line 40, in __init__
raise ValueError(f'invalid number of tags: {num_tags}')
ValueError: invalid number of tags: 0
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/home/philippe/.local/lib/python3.10/site-packages/deeppavlov/core/commands/infer.py", line 55, in build_model
component = from_params(component_config, mode=mode)
File "/home/philippe/.local/lib/python3.10/site-packages/deeppavlov/core/common/params.py", line 102, in from_params
component = obj(**dict(config_params, **kwargs))
File "/home/philippe/.local/lib/python3.10/site-packages/deeppavlov/models/torch_bert/torch_transformers_sequence_tagger.py", line 182, in __init__
super().__init__(optimizer=optimizer,
File "/home/philippe/.local/lib/python3.10/site-packages/deeppavlov/core/models/torch_model.py", line 98, in __init__
self.load()
File "/home/philippe/.local/lib/python3.10/site-packages/deeppavlov/models/torch_bert/torch_transformers_sequence_tagger.py", line 295, in load
self.crf = CRF(self.n_classes).to(self.device)
File "/home/philippe/.local/lib/python3.10/site-packages/deeppavlov/models/torch_bert/crf.py", line 13, in __init__
super().__init__(num_tags=num_tags, batch_first=batch_first)
File "/home/philippe/.local/lib/python3.10/site-packages/torchcrf/__init__.py", line 40, in __init__
raise ValueError(f'invalid number of tags: {num_tags}')
ValueError: invalid number of tags: 0
I downloaded the camembert-base model from https://huggingface.co/camembert-base and copied the files in .deeppavlov/models/camembert-base directory.
Then I figured the deeppavlov's model 'ner_ontonotes_bert_mult' was the best for my use so I edited the config file and changed thoses lines in the metadata section at the end. The docs from DeepPavlov ask to change the TRANSFORMER value, witch I did, and I changed the MODEL_PATH so it point to the files I downloaded previously.
"variables": {
"ROOT_PATH": "~/.deeppavlov",
"DOWNLOADS_PATH": "{ROOT_PATH}/downloads",
"MODELS_PATH": "{ROOT_PATH}/models",
"TRANSFORMER": "camembert-base",
"MODEL_PATH": "{MODELS_PATH}/camembert-base"
},
I am aware that I should have copied the config file to a new one with a different name but this should not be a problem.
Then in python I did the following :
from deeppavlov import configs, build_model
build_model('ner_ontonotes_bert_mult')`
And then I get the error mentioned before. I am lost and don't know where to look from now.

Error when exporting from BigQuery to MySQL

I am trying to export a table from BigQuery to Google Cloud MySQL database.
I found this operator called BigQueryToMySqlOperator (documented here https://airflow.apache.org/docs/apache-airflow-providers-google/stable/_api/airflow/providers/google/cloud/transfers/bigquery_to_mysql/index.html?highlight=bigquerytomysqloperator#module-airflow.providers.google.cloud.transfers.bigquery_to_mysql)
When I deploy the DAG containing this task onto cloud composer, the task always failed with the error
Traceback (most recent call last):
File "/opt/python3.8/lib/python3.8/site-packages/airflow/models/taskinstance.py", line 1113, in _run_raw_task
self._prepare_and_execute_task_with_callbacks(context, task)
File "/opt/python3.8/lib/python3.8/site-packages/airflow/models/taskinstance.py", line 1287, in _prepare_and_execute_task_with_callbacks
result = self._execute_task(context, task_copy)
File "/opt/python3.8/lib/python3.8/site-packages/airflow/models/taskinstance.py", line 1317, in _execute_task
result = task_copy.execute(context=context)
File "/opt/python3.8/lib/python3.8/site-packages/airflow/providers/google/cloud/transfers/bigquery_to_mysql.py", line 166, in execute
for rows in self._bq_get_data():
File "/opt/python3.8/lib/python3.8/site-packages/airflow/providers/google/cloud/transfers/bigquery_to_mysql.py", line 138, in _bq_get_data
response = cursor.get_tabledata(
File "/opt/python3.8/lib/python3.8/site-packages/airflow/providers/google/cloud/hooks/bigquery.py", line 2508, in get_tabledata
return self.hook.get_tabledata(*args, **kwargs)
File "/opt/python3.8/lib/python3.8/site-packages/airflow/providers/google/cloud/hooks/bigquery.py", line 1284, in get_tabledata
rows = self.list_rows(dataset_id, table_id, max_results, selected_fields, page_token, start_index)
File "/opt/python3.8/lib/python3.8/site-packages/airflow/providers/google/common/hooks/base_google.py", line 412, in inner_wrapper
raise AirflowException(
airflow.exceptions.AirflowException: You must use keyword arguments in this methods rather than positional
I don't really understand why it is throwing out this error. Can anyone help me figuring out what went wrong or how should I export data from BigQuery to MySQL DB? Much thanks for your help!
EDIT: My operator code would basically look like this
transfer_data = BigQueryToMySqlOperator(
task_id='task_id',
dataset_table='origin_bq_table',
mysql_table='dest_table_name',
replace=True,
)
Based on the stacktrace, you are most likely using apache-airflow-providers-google==2.2.0.
airflow.exceptions.AirflowException: You must use keyword arguments in
this methods rather than positional
This error originates from the GoogleBaseHook, which can be traced back the BigQueryToMySqlOperator.
BigQueryToMySqlOperator > BigQueryHook > BigQueryConnection > BigQueryCursor > get_tabledata
The reason why you are getting the AirflowException is because get_tabledata
is called as part of the execute method.
Unforuntately, the test for the operator is not comprehensive since it only checks whether or not the method was called was the correct parameters.
I think this will require a new release of the google provider where the cursor in BigQueryToMySqlOperator calls list_rows with keyword arguments instead of get_tabledata, which calls list_rows with positional arguments.
I have also made a Github Issue in the Airflow repository.

Flask-Migrate: index has table none

I am trying to create the migration:
python manage.py db migrate
Getting the following stack trace:
I tracked it down to the
[2019-08-12 17:09:11,197] INFO in __init__: Project API
INFO [alembic.runtime.migration] Context impl OracleImpl.
INFO [alembic.runtime.migration] Will assume non-transactional DDL.
INFO [alembic.autogenerate.compare] Detected removed table 'program'
INFO [alembic.autogenerate.compare] Detected removed index 'alerts_idx' on 'alerts'
INFO [alembic.autogenerate.compare] Detected removed table 'alerts'
INFO [alembic.autogenerate.compare] Detected removed index 'error_log_err_date_idx' on 'error_log'
Traceback (most recent call last):
File ".vscode\extensions\ms-python.python-2019.8.29288\pythonFiles\ptvsd_launcher.py", line 43, in <module>
main(ptvsdArgs)
File ".vscode\extensions\ms-python.python-2019.8.29288\pythonFiles\lib\python\ptvsd\__main__.py", line 432, in main
run()
File "Desktop\Project\MVP\myVE\lib\site-packages\alembic\autogenerate\compare.py", line 612, in obj_removed
modify_ops.ops.append(ops.DropIndexOp.from_index(obj.const))
File "Desktop\Project\MVP\myVE\lib\site-packages\alembic\operations\ops.py", line 1026, in from_index
index.table.name,
AttributeError: 'NoneType' object has no attribute 'name'
Trimed for bravity
I had to put a print statement right before line:
print(index)
File "Desktop\Project\MVP\myVE\lib\site-packages\alembic\operations\ops.py", line 1026, in from_index
When I looked at the index on the table it created a "virtual"/"calculated" column trun(some_date_column) so it makes sense that it would not have a table associated.
Removed the index and alembic started working.
Wanted to share with anyone who might hit this in the future.

debugging ZEXP import - no module named

I am attempting to zexp import a site from what should have been the same Plone version and addon packages. However, when I do so I get
Module ZPublisher.Publish, line 60, in publish
Module ZPublisher.mapply, line 77, in mapply
Module ZPublisher.Publish, line 46, in call_object
Module OFS.ObjectManager, line 620, in manage_importObject
Module OFS.ObjectManager, line 638, in _importObjectFromFile
Module ZODB.ExportImport, line 92, in importFile
Module transaction._transaction, line 260, in savepoint
Module transaction._transaction, line 257, in savepoint
Module transaction._transaction, line 690, in __init__
Module ZODB.Connection, line 1123, in savepoint
Module ZODB.Connection, line 587, in _commit
Module ZODB.ExportImport, line 176, in _importDuringCommit
ImportError: No module named MyOldPackage.Content
This is an old Archetypes class that has since been migrated to a replacement in Dexterity. I thought I had removed all references but apparently something got missed. I get that I have to remove all objects that use this class but there's the rub - how do I find which objects use them?
I thought I had found something in the CMFEditions tools but that might be a red herring. Trying to export portal_historiesstorage yields the same error so I thought perhaps some old versions of past content were stuck there. Deleted shadow storage and rebuilt it, so it's now blank (ZMI page confirms this). But I'm still getting the error when trying to import just this tool - how is it still trying to reference this class? This is why I think it might be a red herring and it's inheriting something from the portal object, but I'm having a rough time trying to find where this class is referenced.
edit:
It looks like I have something bad in zvc_repo - I'm not sure the difference in what shadow storage and this are supposed to do. Here's a proposed script. Can anyone with better CMFEditions or ZopeVersionControl comment on how this sounds?
tool = portal.portal_historiesstorage
from ZODB.broken import PersistentBroken
bad_repo_ids = set()
for sequence in tool.zvc_repo._histories:
for version in tool.zvc_repo[sequence]._versions:
obj = tool.zvc_repo[sequence].getVersionById(version)._data._object.object
if isinstance(obj, PersistentBroken):
bad_repo_ids.add(sequence)
for bid in bad_repo_ids:
del tool.zvc_repo._histories[bid]

products.sqlalchemypas-1.0-py2.6.egg AttributeError: getGroupsForPrincipal

I am using products.sqlalchemypas-1.0-py2.6.egg for authenticating user from MSSQL Table. Authentication work as expected but now I'm trying implementaing groups plugin to
get groups from different table. What happening is when I'm trying to loggin its giving me error saying AttributeError: getGroupsForPrincipal.
Error Traceback is ..
2012-02-21T15:33:14 INFO Zope Ready to handle requests
2012-02-21T15:39:25 ERROR Zope.SiteErrorLog 1329838765.580.598770330561 http://localhost:8060/dev/login_form
Traceback (innermost last):
Module ZPublisher.Publish, line 115, in publish
Module ZPublisher.BaseRequest, line 596, in traverse
Module Products.PluggableAuthService.PluggableAuthService, line 235, in validate
Module Products.PluggableAuthService.PluggableAuthService, line 735, in _findUser
Module Products.PluggableAuthService.PluggableAuthService, line 668, in _getGroupsForPrincipal
AttributeError: getGroupsForPrincipal
My defination in plugin.py is ...
def getGroupsForPrincipal(self, principal=getSecurityManager().getUser().getId(),request=None):
"Getting groups from SIMS"
import pdb; pdb.set_trace()
groups = []
results = self.simsGroupForUser(username=principal)
for row in results.dictionaries():
group = row.get('group')
groups.append(group)
return groups
Don't know why its not able to reach this method in plugin.py however there is implatemented block where I did define this interface to implement resulting showing groups interface in my acl_user pas object.
[added]
I've tried to import my plugin in debugger and tried to reach this method and have same error so I dont know Do I need to define anything specifically to pick this method in my pas. I did define in my implements class to impelement IGroupsPlugin.
Any comment is great help as always.
I don't think you method definition does what you expect it to. principal=getSecurityManager().getUser().getId() will calculate the default parameter at import time rather than at method execution time.
Just found that My file has wrong indentation, that why it was giving attributes error. Thanks all for your time and comments.

Resources