I am trying to create the migration:
python manage.py db migrate
Getting the following stack trace:
I tracked it down to the
[2019-08-12 17:09:11,197] INFO in __init__: Project API
INFO [alembic.runtime.migration] Context impl OracleImpl.
INFO [alembic.runtime.migration] Will assume non-transactional DDL.
INFO [alembic.autogenerate.compare] Detected removed table 'program'
INFO [alembic.autogenerate.compare] Detected removed index 'alerts_idx' on 'alerts'
INFO [alembic.autogenerate.compare] Detected removed table 'alerts'
INFO [alembic.autogenerate.compare] Detected removed index 'error_log_err_date_idx' on 'error_log'
Traceback (most recent call last):
File ".vscode\extensions\ms-python.python-2019.8.29288\pythonFiles\ptvsd_launcher.py", line 43, in <module>
main(ptvsdArgs)
File ".vscode\extensions\ms-python.python-2019.8.29288\pythonFiles\lib\python\ptvsd\__main__.py", line 432, in main
run()
File "Desktop\Project\MVP\myVE\lib\site-packages\alembic\autogenerate\compare.py", line 612, in obj_removed
modify_ops.ops.append(ops.DropIndexOp.from_index(obj.const))
File "Desktop\Project\MVP\myVE\lib\site-packages\alembic\operations\ops.py", line 1026, in from_index
index.table.name,
AttributeError: 'NoneType' object has no attribute 'name'
Trimed for bravity
I had to put a print statement right before line:
print(index)
File "Desktop\Project\MVP\myVE\lib\site-packages\alembic\operations\ops.py", line 1026, in from_index
When I looked at the index on the table it created a "virtual"/"calculated" column trun(some_date_column) so it makes sense that it would not have a table associated.
Removed the index and alembic started working.
Wanted to share with anyone who might hit this in the future.
Related
I'm learning how to use DeepPavlov and can't figure how to use it for NER using the Camembert (french) pre-trained model. My goal is to tag a short paragraph in french.
The docs from deeppavlov explicitly list the Camembert model as a viable transformer architecture. I tried to follow as best as I could but I keep getting this error when I try to build the model.
>>> ner_model = build_model('ner_ontonotes_bert_mult')
/home/philippe/.local/lib/python3.10/site-packages/torch/nn/init.py:405: UserWarning: Initializing zero-element tensors is a no-op
warnings.warn("Initializing zero-element tensors is a no-op")
Some weights of the model checkpoint at camembert-base were not used when initializing CamembertForTokenClassification: ['lm_head.dense.weight', 'lm_head.bias', 'lm_head.layer_norm.bias', 'lm_head.layer_norm.weight', 'lm_head.decoder.weight', 'lm_head.dense.bias']
- This IS expected if you are initializing CamembertForTokenClassification from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).
- This IS NOT expected if you are initializing CamembertForTokenClassification from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).
Some weights of CamembertForTokenClassification were not initialized from the model checkpoint at camembert-base and are newly initialized: ['classifier.weight', 'classifier.bias']
You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.
2023-01-29 09:58:13.308 ERROR in 'deeppavlov.core.common.params'['params'] at line 108: Exception in <class 'deeppavlov.models.torch_bert.torch_transformers_sequence_tagger.TorchTransformersSequenceTagger'>
Traceback (most recent call last):
File "/home/philippe/.local/lib/python3.10/site-packages/deeppavlov/core/common/params.py", line 102, in from_params
component = obj(**dict(config_params, **kwargs))
File "/home/philippe/.local/lib/python3.10/site-packages/deeppavlov/models/torch_bert/torch_transformers_sequence_tagger.py", line 182, in __init__
super().__init__(optimizer=optimizer,
File "/home/philippe/.local/lib/python3.10/site-packages/deeppavlov/core/models/torch_model.py", line 98, in __init__
self.load()
File "/home/philippe/.local/lib/python3.10/site-packages/deeppavlov/models/torch_bert/torch_transformers_sequence_tagger.py", line 295, in load
self.crf = CRF(self.n_classes).to(self.device)
File "/home/philippe/.local/lib/python3.10/site-packages/deeppavlov/models/torch_bert/crf.py", line 13, in __init__
super().__init__(num_tags=num_tags, batch_first=batch_first)
File "/home/philippe/.local/lib/python3.10/site-packages/torchcrf/__init__.py", line 40, in __init__
raise ValueError(f'invalid number of tags: {num_tags}')
ValueError: invalid number of tags: 0
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/home/philippe/.local/lib/python3.10/site-packages/deeppavlov/core/commands/infer.py", line 55, in build_model
component = from_params(component_config, mode=mode)
File "/home/philippe/.local/lib/python3.10/site-packages/deeppavlov/core/common/params.py", line 102, in from_params
component = obj(**dict(config_params, **kwargs))
File "/home/philippe/.local/lib/python3.10/site-packages/deeppavlov/models/torch_bert/torch_transformers_sequence_tagger.py", line 182, in __init__
super().__init__(optimizer=optimizer,
File "/home/philippe/.local/lib/python3.10/site-packages/deeppavlov/core/models/torch_model.py", line 98, in __init__
self.load()
File "/home/philippe/.local/lib/python3.10/site-packages/deeppavlov/models/torch_bert/torch_transformers_sequence_tagger.py", line 295, in load
self.crf = CRF(self.n_classes).to(self.device)
File "/home/philippe/.local/lib/python3.10/site-packages/deeppavlov/models/torch_bert/crf.py", line 13, in __init__
super().__init__(num_tags=num_tags, batch_first=batch_first)
File "/home/philippe/.local/lib/python3.10/site-packages/torchcrf/__init__.py", line 40, in __init__
raise ValueError(f'invalid number of tags: {num_tags}')
ValueError: invalid number of tags: 0
I downloaded the camembert-base model from https://huggingface.co/camembert-base and copied the files in .deeppavlov/models/camembert-base directory.
Then I figured the deeppavlov's model 'ner_ontonotes_bert_mult' was the best for my use so I edited the config file and changed thoses lines in the metadata section at the end. The docs from DeepPavlov ask to change the TRANSFORMER value, witch I did, and I changed the MODEL_PATH so it point to the files I downloaded previously.
"variables": {
"ROOT_PATH": "~/.deeppavlov",
"DOWNLOADS_PATH": "{ROOT_PATH}/downloads",
"MODELS_PATH": "{ROOT_PATH}/models",
"TRANSFORMER": "camembert-base",
"MODEL_PATH": "{MODELS_PATH}/camembert-base"
},
I am aware that I should have copied the config file to a new one with a different name but this should not be a problem.
Then in python I did the following :
from deeppavlov import configs, build_model
build_model('ner_ontonotes_bert_mult')`
And then I get the error mentioned before. I am lost and don't know where to look from now.
Trying to make my own component based on KubernetesPodOperator. I am able to define and add the component to the list of components but when trying to run it, I get:
Operator 'KubernetesPodOperator' of node 'KubernetesPodOperator' is not configured in the list of available operators. Please add the fully-qualified package name for 'KubernetesPodOperator' to the AirflowPipelineProcessor.available_airflow_operators configuration.
and error:
Traceback (most recent call last):
File "/opt/conda/lib/python3.9/site-packages/tornado/web.py", line 1704, in _execute
result = await result
File "/opt/conda/lib/python3.9/site-packages/elyra/pipeline/handlers.py", line 120, in post
response = await PipelineProcessorManager.instance().process(pipeline)
File "/opt/conda/lib/python3.9/site-packages/elyra/pipeline/processor.py", line 134, in process
res = await asyncio.get_event_loop().run_in_executor(None, processor.process, pipeline)
File "/opt/conda/lib/python3.9/asyncio/futures.py", line 284, in __await__
yield self # This tells Task to wait for completion.
File "/opt/conda/lib/python3.9/asyncio/tasks.py", line 328, in __wakeup
future.result()
File "/opt/conda/lib/python3.9/asyncio/futures.py", line 201, in result
raise self._exception
File "/opt/conda/lib/python3.9/concurrent/futures/thread.py", line 58, in run
result = self.fn(*self.args, **self.kwargs)
File "/opt/conda/lib/python3.9/site-packages/elyra/pipeline/airflow/processor_airflow.py", line 122, in process
pipeline_filepath = self.create_pipeline_file(pipeline=pipeline,
File "/opt/conda/lib/python3.9/site-packages/elyra/pipeline/airflow/processor_airflow.py", line 420, in create_pipeline_file
target_ops = self._cc_pipeline(pipeline, pipeline_name)
File "/opt/conda/lib/python3.9/site-packages/elyra/pipeline/airflow/processor_airflow.py", line 368, in _cc_pipeline
raise ValueError(f"Operator '{component.name}' of node '{operation.name}' is not configured "
ValueError: Operator 'KubernetesPodOperator' of node 'KubernetesPodOperator' is not configured in the list of available operators. Please add the fully-qualified package name for 'KubernetesPodOperator' to the AirflowPipelineProcessor.available_airflow_operators configuration.
After looking through the src code, I can see in the processor_airflow.py these lines:
# This specifies the default airflow operators included with Elyra. Any Airflow-based
# custom connectors should create/extend the elyra configuration file to include
# those fully-qualified operator/class names.
available_airflow_operators = ListTrait(
CUnicode(),
["airflow.operators.slack_operator.SlackAPIPostOperator",
"airflow.operators.bash_operator.BashOperator",
"airflow.operators.email_operator.EmailOperator",
"airflow.operators.http_operator.SimpleHttpOperator",
"airflow.contrib.operators.spark_sql_operator.SparkSqlOperator",
"airflow.contrib.operators.spark_submit_operator.SparkSubmitOperator"],
help="""List of available Apache Airflow operator names.
Operators available for use within Apache Airflow pipelines. These operators must
be fully qualified (i.e., prefixed with their package names).
""",
).tag(config=True)
tho I am unsure if this can be extended from the client.
The available_airflow_operators list is a configurable trait in Elyra. You’ll have to add the fully-qualified package name for the KubernetesPodOperator to this list in order for it to create the DAG correctly.
To do so, generate a config file from the command line with jupyter elyra --generate-config. Open the created file and add the following line (you can add it under the PipelineProcessor(LoggingConfigurable) heading if you prefer to keep the file organized):
c.AirflowPipelineProcessor.available_airflow_operators.append("airflow.providers.cncf.kubernetes.operators.kubernetes_pod.KubernetesPodOperator")
Change that string value to the correct package for your use case if it's not the above (make sure that it ends with the class name of the required operator). If you need to add multiple packages, you can also use extend rather than append.
Edit: here is the link to the relevant documentation
I am trying to export a table from BigQuery to Google Cloud MySQL database.
I found this operator called BigQueryToMySqlOperator (documented here https://airflow.apache.org/docs/apache-airflow-providers-google/stable/_api/airflow/providers/google/cloud/transfers/bigquery_to_mysql/index.html?highlight=bigquerytomysqloperator#module-airflow.providers.google.cloud.transfers.bigquery_to_mysql)
When I deploy the DAG containing this task onto cloud composer, the task always failed with the error
Traceback (most recent call last):
File "/opt/python3.8/lib/python3.8/site-packages/airflow/models/taskinstance.py", line 1113, in _run_raw_task
self._prepare_and_execute_task_with_callbacks(context, task)
File "/opt/python3.8/lib/python3.8/site-packages/airflow/models/taskinstance.py", line 1287, in _prepare_and_execute_task_with_callbacks
result = self._execute_task(context, task_copy)
File "/opt/python3.8/lib/python3.8/site-packages/airflow/models/taskinstance.py", line 1317, in _execute_task
result = task_copy.execute(context=context)
File "/opt/python3.8/lib/python3.8/site-packages/airflow/providers/google/cloud/transfers/bigquery_to_mysql.py", line 166, in execute
for rows in self._bq_get_data():
File "/opt/python3.8/lib/python3.8/site-packages/airflow/providers/google/cloud/transfers/bigquery_to_mysql.py", line 138, in _bq_get_data
response = cursor.get_tabledata(
File "/opt/python3.8/lib/python3.8/site-packages/airflow/providers/google/cloud/hooks/bigquery.py", line 2508, in get_tabledata
return self.hook.get_tabledata(*args, **kwargs)
File "/opt/python3.8/lib/python3.8/site-packages/airflow/providers/google/cloud/hooks/bigquery.py", line 1284, in get_tabledata
rows = self.list_rows(dataset_id, table_id, max_results, selected_fields, page_token, start_index)
File "/opt/python3.8/lib/python3.8/site-packages/airflow/providers/google/common/hooks/base_google.py", line 412, in inner_wrapper
raise AirflowException(
airflow.exceptions.AirflowException: You must use keyword arguments in this methods rather than positional
I don't really understand why it is throwing out this error. Can anyone help me figuring out what went wrong or how should I export data from BigQuery to MySQL DB? Much thanks for your help!
EDIT: My operator code would basically look like this
transfer_data = BigQueryToMySqlOperator(
task_id='task_id',
dataset_table='origin_bq_table',
mysql_table='dest_table_name',
replace=True,
)
Based on the stacktrace, you are most likely using apache-airflow-providers-google==2.2.0.
airflow.exceptions.AirflowException: You must use keyword arguments in
this methods rather than positional
This error originates from the GoogleBaseHook, which can be traced back the BigQueryToMySqlOperator.
BigQueryToMySqlOperator > BigQueryHook > BigQueryConnection > BigQueryCursor > get_tabledata
The reason why you are getting the AirflowException is because get_tabledata
is called as part of the execute method.
Unforuntately, the test for the operator is not comprehensive since it only checks whether or not the method was called was the correct parameters.
I think this will require a new release of the google provider where the cursor in BigQueryToMySqlOperator calls list_rows with keyword arguments instead of get_tabledata, which calls list_rows with positional arguments.
I have also made a Github Issue in the Airflow repository.
I am using products.sqlalchemypas-1.0-py2.6.egg for authenticating user from MSSQL Table. Authentication work as expected but now I'm trying implementaing groups plugin to
get groups from different table. What happening is when I'm trying to loggin its giving me error saying AttributeError: getGroupsForPrincipal.
Error Traceback is ..
2012-02-21T15:33:14 INFO Zope Ready to handle requests
2012-02-21T15:39:25 ERROR Zope.SiteErrorLog 1329838765.580.598770330561 http://localhost:8060/dev/login_form
Traceback (innermost last):
Module ZPublisher.Publish, line 115, in publish
Module ZPublisher.BaseRequest, line 596, in traverse
Module Products.PluggableAuthService.PluggableAuthService, line 235, in validate
Module Products.PluggableAuthService.PluggableAuthService, line 735, in _findUser
Module Products.PluggableAuthService.PluggableAuthService, line 668, in _getGroupsForPrincipal
AttributeError: getGroupsForPrincipal
My defination in plugin.py is ...
def getGroupsForPrincipal(self, principal=getSecurityManager().getUser().getId(),request=None):
"Getting groups from SIMS"
import pdb; pdb.set_trace()
groups = []
results = self.simsGroupForUser(username=principal)
for row in results.dictionaries():
group = row.get('group')
groups.append(group)
return groups
Don't know why its not able to reach this method in plugin.py however there is implatemented block where I did define this interface to implement resulting showing groups interface in my acl_user pas object.
[added]
I've tried to import my plugin in debugger and tried to reach this method and have same error so I dont know Do I need to define anything specifically to pick this method in my pas. I did define in my implements class to impelement IGroupsPlugin.
Any comment is great help as always.
I don't think you method definition does what you expect it to. principal=getSecurityManager().getUser().getId() will calculate the default parameter at import time rather than at method execution time.
Just found that My file has wrong indentation, that why it was giving attributes error. Thanks all for your time and comments.
I have a handler that adds a Member to a Group. The last line in this handler causes an error:
TypeError: Can't pickle objects in acquisition wrappers.
> /home/mnieber/.buildout/eggs/ZODB3-3.10.3-py2.6-linux-i686.egg/ZODB/serialize.py(431)_dump()
430 self._p.dump(classmeta)
--> 431 self._p.dump(state)
432 self._file.truncate()
In the pdb debugger I can see that indeed Plone is trying to pickle a value that is an Acquisition wrapper:
ipdb> state
((((<PloneUser 'newuser#usecm.com'>, ('Default_Group',), 'maarten#usecm.com', ('PAS',)),),),)
ipdb> type(state[0][0][0][0])
<type 'Acquisition.ImplicitAcquisitionWrapper'>
However, I cannot see which object is being pickled, and therefore I have no idea which part of my code needs fixing. My question is: how should I go about debugging this error? I have tried looking at all the stack frames, but none of them reveal which object is being serialized.
The handler is this one (run_insecure is a decorator that I use to temporarily install a new security manager that avoids a NotAuthorized error when adding the new member):
#adapter(IPrincipalCreatedEvent)
#run_insecure
def userCreatedHandler(event):
portal_groups = getToolByName(getSite(), "portal_groups")
membersGroup = portal_groups.getGroupById('Default_Group')
membersGroup.addMember(event.principal)
The full error is this one:
Traceback (innermost last):
Module ZPublisher.Publish, line 134, in publish
Module Zope2.App.startup, line 301, in commit
Module transaction._manager, line 89, in commit
Module transaction._transaction, line 329, in commit
Module transaction._transaction, line 443, in _commitResources
Module ZODB.Connection, line 567, in commit
Module ZODB.Connection, line 623, in _commit
Module ZODB.Connection, line 658, in _store_objects
Module ZODB.serialize, line 422, in serialize
Module ZODB.serialize, line 431, in _dump
TypeError: Can't pickle objects in acquisition wrappers.
> /home/mnieber/.buildout/eggs/ZODB3-3.10.3-py2.6-linux-i686.egg/ZODB/serialize.py(431)_dump()
430 self._p.dump(classmeta)
--> 431 self._p.dump(state)
432 self._file.truncate()
I got this kind of problem with pickle, and solved by debugging like you did.
Pickle (used to store objects in ZODB) is trying to serialize your PloneUser, and raising this acquisition wrapper error.
In my case, I was wrapping the portal_workflow object into another class, and had to inherit it from pickle.Pickler, and override __getstate__ method to solve my problem.
This method is called by pickle in order to serialize your object. If you override this method, and return your object.__dict__ without this PloneUser, than this error would not be raised.
This question (although with not your exact problem) have more info about what I'm trying to say.
Nice you could solve your problem.