UnicodeDecodeError in PloneFormGen after Plone Security patch 20161129 - plone

After applying the Plone security patch 20161129 our PloneFormGen form does not work anymore, because it contains Umlauts (ä, ö, ß or €). We use PloneFormGen 1.7.20 which is the recommended version for Plone 4.3.9. Here's the stacktrace:
Module zope.tales.tales, line 696, in evaluate
- URL: file:/var/plone/buildout-cache/eggs/Products.PloneFormGen-1.7.20-py2.7.egg/Products/PloneFormGen/skins/PloneFormGen/widget_fieldset_start.pt
- Line 25, Column 10
- Expression: <PythonExpr widget.Description(here)>
- Names:
{'container': <PloneSite at /mysite>,
'context': <FormFolder at /mysite/mitglied-werden>,
'desitelt': <object object at 0x7fe244f734b0>,
'here': <FormFolder at /mysite/mitglied-werden>,
'loop': {u'field': <Products.PageTemplates.Expressions.PathIterator object at 0x16fe5350>},
'nothing': None,
'options': {'args': (),
'state': <Products.CMFFormController.ControllerState.ControllerState object at 0x7fe22c676610>},
'repeat': <Products.PageTemplates.Expressions.SafeMapping object at 0x1d61fc00>,
'request': <HTTPRequest, URL=https://www.example.org/mitglied-werden/fg_base_view_p3>,
'root': <Application at >,
'template': <FSControllerPageTemplate at /mysite/fg_base_view_p3 used for /mysite/mitglied-werden>,
'traverse_subpath': [],
'user': <PloneUser 'siteb390'>}
Module Products.PageTemplates.ZRPythonExpr, line 48, in __call__
- __traceback_info__: widget.Description(here)
Module PythonExpr, line 1, in <expression>
Module Products.PloneFormGen.content.field_utils, line 27, in wDescription
Module zope.i18n, line 107, in translate
UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 20: ordinal not in range(128)
After removing the umlauts it works again, but the form looks a bit unprofessional now.
After searching the web for one hour now for just this error, I have no particular idea which extension causes this (PloneFormGen?) causes this and how to fix it :-/ Even no idea on which direction to look further...

Starting with 1.7.20, all PFG description fields are pushed through the translation mechanism. (Please don't ask me why they do that with user-supplied fields, because the description field is not language-insensitive in the first place.) Ths imposes additional limitations on what letters you can use in the description, as you have found.
Not claiming that this is a bugfix, but in my quick trial, modifying buildout-cache/eggs/Products.PloneFormGen-1.7.20-py2.7.egg/Products/PloneFormGen/content/field_utils.py line 27 from
if value:
value = translate(value, context=instance.REQUEST)
return cgi.escape(value)
to
if value:
value = translate(value.decode('utf-8'), context=instance.REQUEST)
return cgi.escape(value)
made the problem go away.

Related

execution_date jinja resolving as a string

I have an airflow dag that uses the following jinja template: "{{ execution_date.astimezone('Etc/GMT+6').subtract(days=1).strftime('%Y-%m-%dT00:00:00') }}"
This template works in other dags, and it works when the schedule_interval for the dag is set to timedelta(hours=1). However, when we set the schedule interval to 0 8 * * *, it throws the following traceback at runtime:
Traceback (most recent call last):
File "/usr/lib/python2.7/site-packages/airflow/models/__init__.py", line 1426, in _run_raw_task
self.render_templates()
File "/usr/lib/python2.7/site-packages/airflow/models/__init__.py", line 1790, in render_templates
rendered_content = rt(attr, content, jinja_context)
File "/usr/lib/python2.7/site-packages/airflow/models/__init__.py", line 2538, in render_template
return self.render_template_from_field(attr, content, context, jinja_env)
File "/usr/lib/python2.7/site-packages/airflow/models/__init__.py", line 2520, in render_template_from_field
for k, v in list(content.items())}
File "/usr/lib/python2.7/site-packages/airflow/models/__init__.py", line 2520, in <dictcomp>
for k, v in list(content.items())}
File "/usr/lib/python2.7/site-packages/airflow/models/__init__.py", line 2538, in render_template
return self.render_template_from_field(attr, content, context, jinja_env)
File "/usr/lib/python2.7/site-packages/airflow/models/__init__.py", line 2514, in render_template_from_field
result = jinja_env.from_string(content).render(**context)
File "/usr/lib64/python2.7/site-packages/jinja2/environment.py", line 1008, in render
return self.environment.handle_exception(exc_info, True)
File "/usr/lib64/python2.7/site-packages/jinja2/environment.py", line 780, in handle_exception
reraise(exc_type, exc_value, tb)
File "<template>", line 1, in top-level template code
TypeError: astimezone() argument 1 must be datetime.tzinfo, not str
It appears the execution date being passed in is a string, not a datetime object; but I am only able to hit this error on this specific dag, and no others. I've tried deleting the dag entirely and recreating it with no luck.
Looks like astimezone(..) function is misbehaving, it expects a datetime.tzinfo while you are passing it an str argument ('Etc/GMT+6')
TypeError: astimezone() argument 1 must be datetime.tzinfo, not str
While I couldn't make the exact thing work, I believe following achieves pretty much the same effect as what you are trying
{{ execution_date.in_timezone("US/Eastern") - timedelta(days=1) }}
Recall that
execution_date macro is a Pendulum object
in_timezone(..) converts it into a datetime.datetime(..)
then we just add a datetime.timedelta(days=1) to it

Unable to read body returned by HTTPoison

I have this:
HTTPoison.start
case HTTPoison.get("some url") do
{:ok, %HTTPoison.Response{status_code: 200, body: body}} ->
IO.puts(body)
Which produces an exception:
Generated escript grex_cli with MIX_ENV=dev
** (ArgumentError) argument error
(stdlib) :io.put_chars(#PID<0.49.0>, :unicode, [<<60, 33, 100, 111,..... 58, ...>>, 10])
(elixir) lib/kernel/cli.ex:76: anonymous fn/3 in Kernel.CLI.exec_fun/2
How can I read the body and print it?
IO.puts fails with that error because body here is not valid UTF-8 data. If you want to write non UTF-8 binary to stdout, you can use IO.binwrite instead of IO.puts:
IO.binwrite(body)
I got this error when I got compressed content.
So I unzipped it before printing:
IO.puts :zlib.gunzip(body)
You also can check if body is compressed before unzip: https://github.com/edgurgel/httpoison/issues/81#issuecomment-219920101

UnicodeWarning While Using Dictionaries

I've been playing with Python 2.7 for a while, and I'm now tryin to make my own Encryption/Decryption algorithm.
I'm trying to make it support non-Ascii characters.
So this is a part of the dictionnary :
... u'\xe6': '1101100', 'i': '0001000', u'\xea': '1100001', 'm': '0001100', u'\xee': '1100111', 'q': '0010000', 'u': '0010100', u'\xf6': '1110010', 'y': '0011000', '}': '1001111'}
But when I try to convert, by exemple, "é" into binairy, doing
base64 = encrypt[i]
where as encrypt is the name of the dic and i = u"é"
I get this error :
Warning (from warnings module):
File "D:\DeskTop 2\Programs\Projects\4.py", line 174
base64 = encrypt[i]
UnicodeWarning: Unicode equal comparison failed to convert both arguments to Unicode - interpreting them as being unequal
Traceback (most recent call last):
File "D:\DeskTop 2\Programs\Projects\4.py", line 197, in
main()
File "D:\DeskTop 2\Programs\Projects\4.py", line 196, in main
decryption(key, encrypt, decrypt)
File "D:\DeskTop 2\Programs\Projects\4.py", line 174, in decryption
base64 = encrypt[i]
KeyError: '\xf1'
Also, I did start with
# -*- coding: utf-8-*-
Alright, sorry for the useless post.
I found the fix. Basically, I did :
for i in user_input:
base64 = encrypt[i]
but i would be like \0xe
I added
j = i.decode("latin-1")
so j = u"\0xe"
And now it works :D

Alfresco document-derived content is missing <content>

I've created a custom content type derived from document. I am trying to use CMIS to query my Alfresco server (tried with 4.2.b and 4.2.c) programmatically for my documents using python cmislib. I have a pyramid server that takes REST calls and sends them to my Alfresco server using CMIS.
I get this error:
2013-04-11 11:19:25,526 | ERROR | Exception when serving /access_manager/search_noauth
Traceback (most recent call last):
File "/home/hbrown/.virtualenvs/access_manager_master/local/lib/python2.7/site-packages/waitress-0.8.1-py2.7.egg/waitress/channel.py", line 329, in service
task.service()
[...]
File "/home/hbrown/workspace/spt/access_manager/access_manager/views/search.py", line 223, in cmis_main
for result in repo.query(whole_query)
File "/home/hbrown/.virtualenvs/access_manager_master/local/lib/python2.7/site-packages/cmislib/model.py", line 2467, in getContentStream
assert(len(contentElements) == 1), 'Expected to find exactly one atom:content element.'
AssertionError: Expected to find exactly one atom:content element.
I am using getContentStream() to retrieve content. Based on the code comment, I'd say it is the correct API call:
>>> doc.getName()
u'sample-b.pdf'
>>> o = open('tmp.pdf', 'wb')
>>> result = doc.getContentStream()
>>> o.write(result.read())
>>> result.close()
>>> o.close()
>>> import os.path
>>> os.path.getsize('tmp.pdf')
117248
The python code in cmislib clearly expects the document to have XML that includes an element named content, and mine does not.
The calling code looks like this:
from cmislib import CmisClient
SERVER = "localhost"
url = "http://{0}:8080/alfresco/cmisatom".format(SERVER)
client = CmisClient(url, 'admin', 'alfresco')
repo = client.defaultRepository
results = repo.query("select * from wg:bulletin")
print results[0].getContentStream().read()
The XML being operated on in getContentStream looks like this:
<atom:entry>
<atom:author>
<atom:name/>
</atom:author>
<atom:id>http://chemistry.apache.org/aWQtMQ==</atom:id>
<atom:published>2013-04-12T03:22:38Z</atom:published>
<atom:title>Query Result id-1</atom:title>
<app:edited>2013-04-12T03:22:38Z</app:edited>
<atom:updated>2013-04-12T03:22:38Z</atom:updated>
<cmisra:object xmlns:ns3="http://docs.oasis-open.org/ns/cmis/messaging/200908/">
<cmis:properties>
<cmis:propertyInteger displayName="Content Stream Length" localName="contentStreamLength" propertyDefinitionId="cmis:contentStreamLength" queryName="b.cmis:contentStreamLength">
<cmis:value>249</cmis:value>
</cmis:propertyInteger>
<cmis:propertyId displayName="Object Type Id" localName="objectTypeId" propertyDefinitionId="cmis:objectTypeId" queryName="b.cmis:objectTypeId">
<cmis:value>D:wg:bulletin</cmis:value>
</cmis:propertyId>
<cmis:propertyString displayName="Version Series Checked Out By" localName="versionSeriesCheckedOutBy" propertyDefinitionId="cmis:versionSeriesCheckedOutBy" queryName="b.cmis:versionSeriesCheckedOutBy"/>
<cmis:propertyId displayName="Version Series Checked Out Id" localName="versionSeriesCheckedOutId" propertyDefinitionId="cmis:versionSeriesCheckedOutId" queryName="b.cmis:versionSeriesCheckedOutId"/>
<cmis:propertyId displayName="Version series id" localName="versionSeriesId" propertyDefinitionId="cmis:versionSeriesId" queryName="b.cmis:versionSeriesId">
<cmis:value>workspace://SpacesStore/1cd2053d-1fc4-4e85-b780-ba80284f0841</cmis:value>
</cmis:propertyId>
<cmis:propertyString displayName="wg:account" localName="account" propertyDefinitionId="wg:account" queryName="b.wg:account"/>
<cmis:propertyString displayName="Version Label" localName="versionLabel" propertyDefinitionId="cmis:versionLabel" queryName="b.cmis:versionLabel">
<cmis:value>1.0</cmis:value>
</cmis:propertyString>
<cmis:propertyBoolean displayName="Is Latest Version" localName="isLatestVersion" propertyDefinitionId="cmis:isLatestVersion" queryName="b.cmis:isLatestVersion">
<cmis:value>true</cmis:value>
</cmis:propertyBoolean>
<cmis:propertyBoolean displayName="Is Version Series Checked Out" localName="isVersionSeriesCheckedOut" propertyDefinitionId="cmis:isVersionSeriesCheckedOut" queryName="b.cmis:isVersionSeriesCheckedOut">
<cmis:value>false</cmis:value>
</cmis:propertyBoolean>
<cmis:propertyString displayName="Last Modified By" localName="lastModifiedBy" propertyDefinitionId="cmis:lastModifiedBy" queryName="b.cmis:lastModifiedBy">
<cmis:value>admin</cmis:value>
</cmis:propertyString>
<cmis:propertyString displayName="Created by" localName="createdBy" propertyDefinitionId="cmis:createdBy" queryName="b.cmis:createdBy">
<cmis:value>admin</cmis:value>
</cmis:propertyString>
<cmis:propertyDateTime displayName="wg:displayUntil" localName="displayUntil" propertyDefinitionId="wg:displayUntil" queryName="b.wg:displayUntil"/>
<cmis:propertyId displayName="Alfresco Node Ref" localName="nodeRef" propertyDefinitionId="alfcmis:nodeRef" queryName="b.alfcmis:nodeRef">
<cmis:value>workspace://SpacesStore/1cd2053d-1fc4-4e85-b780-ba80284f0841</cmis:value>
</cmis:propertyId>
<cmis:propertyString displayName="wg:email" localName="email" propertyDefinitionId="wg:email" queryName="b.wg:email"/>
<cmis:propertyBoolean displayName="wg:isActive" localName="isActive" propertyDefinitionId="wg:isActive" queryName="b.wg:isActive">
<cmis:value>false</cmis:value>
</cmis:propertyBoolean>
<cmis:propertyString displayName="wg:username" localName="username" propertyDefinitionId="wg:username" queryName="b.wg:username"/>
<cmis:propertyBoolean displayName="Is Latest Major Version" localName="isLatestMajorVersion" propertyDefinitionId="cmis:isLatestMajorVersion" queryName="b.cmis:isLatestMajorVersion">
<cmis:value>true</cmis:value>
</cmis:propertyBoolean>
<cmis:propertyId displayName="Content Stream Id" localName="contentStreamId" propertyDefinitionId="cmis:contentStreamId" queryName="b.cmis:contentStreamId">
<cmis:value>store://2013/4/10/15/29/20b185d0-afae-4a7f-a06e-58eab399bdc9.bin</cmis:value>
</cmis:propertyId>
<cmis:propertyString displayName="Name" localName="name" propertyDefinitionId="cmis:name" queryName="b.cmis:name">
<cmis:value>.pythonrc</cmis:value>
</cmis:propertyString>
<cmis:propertyString displayName="Content Stream MIME Type" localName="contentStreamMimeType" propertyDefinitionId="cmis:contentStreamMimeType" queryName="b.cmis:contentStreamMimeType">
<cmis:value>text/plain</cmis:value>
</cmis:propertyString>
<cmis:propertyDateTime displayName="Creation Date" localName="creationDate" propertyDefinitionId="cmis:creationDate" queryName="b.cmis:creationDate">
<cmis:value>2013-04-10T15:29:18.146-04:00</cmis:value>
</cmis:propertyDateTime>
<cmis:propertyString displayName="Change token" localName="changeToken" propertyDefinitionId="cmis:changeToken" queryName="b.cmis:changeToken"/>
<cmis:propertyString displayName="wg:state" localName="state" propertyDefinitionId="wg:state" queryName="b.wg:state"/>
<cmis:propertyDateTime displayName="wg:displayFrom" localName="displayFrom" propertyDefinitionId="wg:displayFrom" queryName="b.wg:displayFrom"/>
<cmis:propertyString displayName="Checkin Comment" localName="checkinComment" propertyDefinitionId="cmis:checkinComment" queryName="b.cmis:checkinComment"/>
<cmis:propertyString displayName="wg:application" localName="application" propertyDefinitionId="wg:application" queryName="b.wg:application"/>
<cmis:propertyId displayName="Object Id" localName="objectId" propertyDefinitionId="cmis:objectId" queryName="b.cmis:objectId">
<cmis:value>workspace://SpacesStore/1cd2053d-1fc4-4e85-b780-ba80284f0841;1.0</cmis:value>
</cmis:propertyId>
<cmis:propertyBoolean displayName="Is Immutable" localName="isImmutable" propertyDefinitionId="cmis:isImmutable" queryName="b.cmis:isImmutable">
<cmis:value>false</cmis:value>
</cmis:propertyBoolean>
<cmis:propertyBoolean displayName="Is Major Version" localName="isMajorVersion" propertyDefinitionId="cmis:isMajorVersion" queryName="b.cmis:isMajorVersion">
<cmis:value>true</cmis:value>
</cmis:propertyBoolean>
<cmis:propertyString displayName="wg:institution" localName="institution" propertyDefinitionId="wg:institution" queryName="b.wg:institution"/>
<cmis:propertyId displayName="Base Type Id" localName="baseTypeId" propertyDefinitionId="cmis:baseTypeId" queryName="b.cmis:baseTypeId">
<cmis:value>cmis:document</cmis:value>
</cmis:propertyId>
<cmis:propertyString displayName="Content Stream Filename" localName="contentStreamFileName" propertyDefinitionId="cmis:contentStreamFileName" queryName="b.cmis:contentStreamFileName">
<cmis:value>.pythonrc</cmis:value>
</cmis:propertyString>
<cmis:propertyDateTime displayName="Last Modified Date" localName="lastModificationDate" propertyDefinitionId="cmis:lastModificationDate" queryName="b.cmis:lastModificationDate">
<cmis:value>2013-04-10T15:29:23.384-04:00</cmis:value>
</cmis:propertyDateTime>
</cmis:properties>
</cmisra:object>
</atom:entry>
There is clearly no XML element named content here for the python code to extract.
Is this a misconfiguration of my custom content document or is it a change in CMIS that cmislib does not track or am I calling the wrong API function to get the content or something else?
Later: the minimum change to fix this is to make calls to either reload or getAllowableActions.
This was the original code:
def cmis_main(props, settings):
"""
Create a CMIS query based on props and execute against Alfresco
"""
def cmis_query(props, mapping):
"""
Create CMIS query of AND-separated OR-clauses
"""
# Code that formats a query string from dictionaries...
cmis_mapping = {
# Dictionary config for call to cmis_query
# Nothing to see here. Move on.
"app_sids": {
"where_fmt": IN_WHERE_FMT,
"key": "{0}:application".format(CMIS_NAMESPACE),
"fn": set_format,
},
}
cmis_url, cmis_user, cmis_password = cmis_args(settings)
cmisclient = CmisClient(cmis_url, cmis_user, cmis_password)
repo = cmisclient.getDefaultRepository()
whole_query = cmis_query(props, cmis_mapping)
logger.debug(whole_query)
return [
{
'name': result.name,
'content': result.getContentStream().read(),
'content_mime_type': result.properties["cmis:contentStreamMimeType"],
}
for result in repo.query(whole_query)
]
And it was broken. So I changed the code to this:
results = list(repo.query(whole_query))
for result in results:
print(result.getAllowableActions())
# or: result.reload()
return [
{
'name': result.name,
'content': result.getContentStream().read(),
'content_mime_type': result.properties["cmis:contentStreamMimeType"],
}
for result in results
]
And it worked. I changed it to this:
results = list(repo.query(whole_query))
for result in results:
pass
return [
{
'name': result.name,
'content': result.getContentStream().read(),
'content_mime_type': result.properties["cmis:contentStreamMimeType"],
}
for result in results
]
And it broke. So the XML does not appear to be fully loaded in the CMISLIB object.
Try doing results[0].reload() before calling getContentStream(). That shouldn't be required but it may force the object to reload with the content element.

Diazo, parameters and restrictedTraverse

if in my diazo controlpanel > 'Parameter expressions' I put
have_left_portlets = python:context and context.restrictedTraverse('##plone').have_portlets('plone.leftcolumn',context)
I obtain an error only when I'm on the portal homepage:
2012-06-26 16:51:42 ERROR plone.transformchain Unexpected error whilst trying to apply transform chain
Traceback (most recent call last):
File "/Users/vito/.buildout/eggs/plone.transformchain-1.0.2-py2.6.egg/plone/transformchain/transformer.py", line 48, in __call__
newResult = handler.transformIterable(result, encoding)
File "/Users/vito/.buildout/eggs/plone.app.theming-1.0-py2.6.egg/plone/app/theming/transform.py", line 257, in transformIterable
params[name] = quote_param(expression(expressionContext))
File "/Users/vito/.buildout/eggs/Zope2-2.13.13-py2.6.egg/Products/PageTemplates/ZRPythonExpr.py", line 48, in __call__
return eval(self._code, vars, {})
File "PythonExpr", line 1, in <expression>
File "/Users/vito/.buildout/eggs/AccessControl-2.13.7-py2.6-macosx-10.6-x86_64.egg/AccessControl/ImplPython.py", line 675, in guarded_getattr
v = getattr(inst, name)
AttributeError: 'FilesystemResourceDirectory' object has no attribute 'restrictedTraverse'
How I can solve this?
I suspect this is a bug in plone.app.theming: the context isn't set correctly. Strange, though.
Just confirming that the issue exits:
I get about the same traceback, the site itself looks fine, but for every click inside the site I get a the following traceback in my instance fg:
2012-08-10 15:05:05 ERROR plone.transformchain Unexpected error whilst trying to apply transform chain
Traceback (most recent call last):
File "/opt/etc/buildout/eggs/plone.transformchain-1.0.2-py2.6.egg/plone/transformchain/transformer.py", line 48, in __call__
newResult = handler.transformIterable(result, encoding)
File "/opt/etc/buildout/eggs/plone.app.theming-1.0-py2.6.egg/plone/app/theming/transform.py", line 257, in transformIterable
params[name] = quote_param(expression(expressionContext))
File "/opt/etc/buildout/eggs/Zope2-2.13.10-py2.6.egg/Products/PageTemplates/ZRPythonExpr.py", line 48, in __call__
return eval(self._code, vars, {})
File "PythonExpr", line 1, in <expression>
AttributeError: 'FilesystemResourceDirectory' object has no attribute 'Language'
This is because I have the following line in my manifest.cfg (which is about the same as the parameter line in the plone_control_panel:
lang = python: context.Language()
In a way in my case this is sort of logical, since not all content objects have an index called Language().
But the 'context' in this case is apparently refering to the 'FileSystemResourceDirectory' and not to the piece of content you are on?
I'll try with pdb if I can find some more info...

Resources