is there any docs / tutorial about how to setup ZRS ?
I can't find anything useful...
ZRS replicates the whole data.fs, is that right?
is there any way / solution to only replicate and keep in sync parts of the instance (folders) ?
Once you've done it, it goes from "black magic" to "not much to say", really. As you've probably noted, the latest versions are incompatible with Plone's older Zope version, so you need to pin to
zc.zrs == 2.4.4
and in the usecase I have here (which includes zlibstorage), the single stanzas in zeo-conf-additional look like this on the server:
<serverzlibstorage demo>
<zrs demo-Z>
replicate-to 41002
<filestorage demo-ZR>
path ${buildout:zeo-datadir}/filestorage/${buildout:instancename}/demo.fs
blob-dir ${buildout:zeo-datadir}/blobstorage/${buildout:instancename}/demo
</filestorage>
</zrs>
</serverzlibstorage>
like this on the replicate:
<serverzlibstorage demo>
<zrs demo-Z>
replicate-from ${buildout:zeo-host}:41002
keep-alive-delay 60
<filestorage demo-ZR>
path ${buildout:zeo-datadir}/filestorage/${buildout:instancename}/demo.fs
blob-dir ${buildout:zeo-datadir}/blobstorage/${buildout:instancename}/demo
</filestorage>
</zrs>
</serverzlibstorage>
and like this on the client in zope-conf-additional:
<zodb_db demo>
cache-size 9000
<zlibstorage>
<zeoclient>
server ${buildout:zeo-host}:${buildout:zeo-port}
# backup:
server ${buildout:clone-host}:${buildout:clone-port}
read-only-fallback True
storage demo
name demo
# [...]
</zeoclient>
</zlibstorage>
mount-point /fs-demo
</zodb_db>
But, as you remark, you can only replicate entire databases, and from what I gather, the general consensus in the Plone community is that having multiple databases (even one per Plone instance) is very much not a recommended use-case anymore.
Related
I have the following issue:
I use salt stack to manage my minions, which are running in different datacenters. But the package repositories are not consistent: Not all have the latest versions of salt. With Salt stack I can of course work around that, so I added to the top.sls:
'not G#saltversion:3003.1':
- fixes.saltversion
But I don't like that up there. I've tried several variants, but couldn't manage to select minions which have a specific grain less than a specific version. Like in this case: To select all minions which have an older version than 3003.1 to apply a state on them, that gets the package directly from a different repo.
How do I select "less than" of a Grain?
I've googled around already and didn't find anything matching my case. The Docs are also not helpful for my case. I've read about custom matcher: But do I really need to implement a custom matcher for that?
Thanks in advance for your help everyone
Have a look at the following grain: saltversioninfo.
This grain is a list: [ majorversion, patchversion ].
You can target minions with releases later than Fluorine (2019.2.0) like this:
'P#saltversioninfo:0:\b(?:3[0-9]{2}[0-2])\b or ( G#saltversioninfo:0:3003 and G#saltversioninfo:1:0 ):
- match: compound
- fixes.saltversion
This compound match will target minions with release between 3000 and 3003.0.
This is a bit static and you need to modify this after a new release. But I hope this will help you.
EDIT:
The matcher above is untested, I don't have minions with older version. You should test the matcher before with the following salt command:
salt -C 'P#saltversioninfo:0:\b(?:3[0-9]{2}[0-2])\b or ( G#saltversioninfo:0:3003 and G#saltversioninfo:1:0 )' test.ping
I am working on AEM 5.6 to 6.2 upgrade project. There are some nodes in aem 5.6 environment which contains invalid character(as per JCR naming convention like rte[2] is one of the node name which doesn't follow the naming convention)but somehow we are able to replicate those nodes in 5.6 environment. After upgrade it to aem 6.2,it seems like JCR is more restricted and won't allow the nodes to replicate if it is having invalid characters.
Getting the below error in aem 6.2: error:
com.day.cq.replication.ReplicationException: Repository error during node import: Cannot create a new node using a name including an index
Is there any way we can configure AEM 6.2 to stop checking JCR node names?or any other solution?
JCR 2 does not allow [ as a valid character therefore you won't get an easy workaround for this. It's one of the limitations just like the same-named-sibling.
My recommendation will be to modify these nodes before the upgrade/migration to 6.2. This can be complicated and costly for business but 6.2 won't allow it.
As a background [ was allowed in older version due to twisted support for grammar syntax for same-named-siblings.
Assuming that these are all content nodes as nothing out-of-the-box in AEM 5.x follows this naming convention.
Some ways to fix it:
Write a custom servlet to query and rename the paths across all references. You will have to test your content for these renames.
Use Groovy console (https://github.com/OlsonDigital/aem-groovy-console) to rename the nodes.
In either case, you will need to modify the nodes before the migration as the structure is not oak compliant therefore you cannot use crx2oak commit hooks also. This can be done with both in-place upgrade and side-by-side migration. This is similar to the problem with same named siblings that must be corrected before the migration.
Some efficiency techniques that might help:
Write queries to find invalid node names on top-level nodes like /content/mysite-a, /content/mysite-b etc. Don't run root level queries on /content as it might downgrade to
traversal and halt the execution.
Ensure that all references are updated in same commit. If you are using custom servlet, call session.save() only after updating all the node names and it's corresponding references.
As i mention in the comment this replication failure causes because of the oak workspace restriction as the code snippet below
//handle index
if (oakName.contains("["))
{ throw new RepositoryException("Cannot create a new node using a name including an index");
}
and i feel you can't escape this constraint as it it required by the repository to maintain consistency
you can find nodes that ends with '[', by below query
SELECT [jcr:path] FROM [nt:base] WHERE ISDESCENDANTNODE('/content/path/') AND [jcr:path] like '%\['
and to modify the JCR/CRX nodes you can use CURL or SlingPostServlet method
Some helpful posts are blow.
https://github.com/paulrohrbeck/aem-links/blob/master/curl_cheatsheet.md
http://sling.apache.org/site/manipulating-content-the-slingpostservlet-servletspost.html
Can you try migrating using a tool like oak-upgrade and let us know if you are still facing this issue.
The tool is robust and you have the flexibility to configure specific sub-trees for migration using this tool.
I have a Plone 4 site which stopped to rename new Archetypes objects; after creation (as something like /temp/portaltype.2015-04-23.1234567890) and saving the first changes, including giving it a title, it should be renamed to something nicer (/temp/an-object-with-a-meaningful-name), but this doesn't happen anymore.
Perhaps the problem arose when I applied some changes to update Plone from 4.3.3 to 4.3.4 (to make one step at a time); but I have inherited a long versions.cfg which is solely sorted by package names and doesn't include any hints why certain versions were chosen ...
I'm able to go back two months and have a version which does the renaming, but without more knowledge about what to look for, it will be a very time-consuming process of re-applying every single change, rebuilding, starting and testing; but there have not been any changes to my schema definitions. I have a temp browser which is involved in delivering the primary edit form. but this doesn't seem to be the case for the saving action.
Sadly I don't fully understand yet the mechanisms of the base_edit action which should - as far as I understand - call Archetypes.BaseObject.processForm and implicitly ._renameAfterCreation, so I'd be grateful for some pointers how to debug this. Thank you!
Update:
I have a few triggers in my product's configure.zcml, e.g.:
<subscriber
for=".content.portaltype.PortalType
Products.Archetypes.interfaces.IObjectInitializedEvent"
handler=".events.onInitPortalType"/>
… with, in events.py:
def onInitPortalType(self, event):
"""
Called after first edit of new objects?
"""
print '/// onInitPortalType(%(self)r, %(event)r)' % locals()
setInitialOwner(self, event)
setStateToPrivate(self, event)
However, the event doesn't seem to be triggered, since I couldn't find the output in an instance fg session.
Update 2:
I noticed that zope.event had been pinned to a quite old version (3.5.2), so I'm trying to update to 4.3.4 more seriously now (following this how-to). This got me zope.event v4.0.3, but I have a version conflict now:
There is a version conflict.
We already have: zc.recipe.egg 1.3.2.
While:
Installing.
Getting section test.
Initializing section test.
Installing recipe zc.recipe.testrunner.
There seems to be a requirement for zc.recipe.egg < 2dev somewhere, but I can't find it.
Nothing significant changed between Plone 4.3.3 and 4.3.4 on Archetypes. Products.Archetypes changed from 1.9.7 to 1.9.8 and Products.ATContentTypes remains on the same version.
Pointers could be:
There's a _at_rename_after_creation flag, which is True by default. This can be changed on the content type class.
Is your type still activated in portal_factorytool? (Afaik this should have no impact on renaming after creation - but who knows :-))
Any Products.Archetypes.interfaces.IObjectInitializedEvent subscriber?
Issue I had once was, that the tmp id portaltype.2015-04-23.1234567890 had the wrong format and AT did no recognise it as tmp id and therefore it did not rename it after creation. The method AT uses to check if the id is autogenerated --> https://github.com/plone/Products.CMFPlone/blob/4.3.4/Products/CMFPlone/utils.py#L111 AFAIK the problem was, that the meta_type and portal_type was not the same anymore.
I have been using pyudev to look for bluetooth devices and then used the "features" attribute to determine if the device has LE support. However, just recently in the latest version of Ubuntu, udev no longer reports anything for "features".
Here's what details I do get:
$ udevadm info --attribute-walk /sys/class/bluetooth/hci0
looking at device '/devices/pci0000:00/0000:00:1a.0/usb1/1-1/1-1.4/1-1.4:1.0/bluetooth/hci0':
KERNEL=="hci0"
SUBSYSTEM=="bluetooth"
DRIVER==""
ATTR{name}=="ubuntu-0"
ATTR{type}=="BR/EDR"
ATTR{address}=="08:3e:8e:xx:xx:xx"
And then on another device that is continuing to work as I expected I get:
$ udevadm info --attribute-walk /sys/class/bluetooth/hci0
looking at device '/devices/platform/sw-ohci.1/usb3/3-1/3-1:1.0/bluetooth/hci0':
KERNEL=="hci0"
SUBSYSTEM=="bluetooth"
DRIVER==""
ATTR{bus}=="USB"
ATTR{sniff_min_interval}=="80"
ATTR{name}=="linaro-nano-0"
ATTR{type}=="BR/EDR"
ATTR{hci_version}=="6"
ATTR{class}=="0x000000"
ATTR{idle_timeout}=="0"
ATTR{address}=="00:02:72:xx:xx:xx"
ATTR{features}=="0xbffecffedbff7b87"
ATTR{sniff_max_interval}=="800"
ATTR{manufacturer}=="15"
ATTR{hci_revision}=="4096"
Is there some sort of configuration change somewhere that causes the difference in responses?
EDIT
I've found that most of those missing values are now in /sys/kernel/debug/bluetooth/hci0/ but I have no idea why that's so. However the format of the features file is different. Is this controlled by a configuration file, compiler options, or something else?
I think changes have been made in the kernel code to use debugfs instead of sysfs for several bits of information. So, I'll just have to rethink how I get that information now.
We're running a Zope server with an eventually large-ish number of Plone (4) sites. Every now and then, an extension product update comes along and requires a re-install to pick up changes in the profile settings, e.g. new content types.
Manually, this would mean clicking through to every Plone site's portal_quickinstaller, tick the products, press update. This is not very feasible if we're talking about dozens of sites, so I'm trying to automate this. Essentially so far, I have the following living as a Script(Python) in the Zope root:
a = context.restrictedTraverse('/')
p = a['Plone']
print p.getSiteManager()
qi = p.restrictedTraverse('portal_quickinstaller')
print qi
qi.reinstallProducts('LinguaPlone')
(Simplified; in reality I have a longer list instead of the single Plone instance, and I might want to reinstall a longer list of products.)
This fails with the following:
Module Products.CMFQuickInstallerTool.QuickInstallerTool, line 613, in uninstallProducts
Module Products.CMFQuickInstallerTool.InstalledProduct, line 272, in uninstall
Module Products.CMFQuickInstallerTool.InstalledProduct, line 351, in _cascadeRemove
AttributeError: 'BaseGlobalComponents' object has no attribute 'objectItems'
From my debugging attempts so far, the BaseGlobalComponents is the Zope SiteManager returned by the zope.component.getSiteManager. How do I convince quickinstaller to pick up the right one, i.e. the one from the Plone Site it's living in?
Alternatively, how would I go about automating re-installing products in a way that will remain vaguely feasible for larger installations? (ETA: I'm aware this is not the kind of thing you do automatically with a cronjob, but updates of inhouse-developed products can't be avoided, I'm afraid.)
Here's how to change the active local site manager. You won't be able to do this in Restricted Python, so you'll need to turn your Python script into an External Method or browser view.
from zope.app.component.hooks import setHooks, setSite
setHooks()
setSite(site)
The setHooks call only needs to be done once. In Zope 2.12 these calls should be imported instead from zope.site.hooks and in Zope 2.13 from zope.component.hooks.
Keep in mind that calling reinstallProducts is not appropriate for all add-on products, and not recommended unless you've carefully checked what reinstalling does and are sure it won't cause problems. Some products provide upgrade steps that run actions more selectively.
Disclaimer: are you sure you want to do this? Automatically reinstalling and upgrading products to the latest version, blindly and without any testing on a staging instance, is asking for trouble.
Anyway, you can do such a thing using XML-RPC and a little tweaking. This is how you install a product on a live running instance using XML-RPC:
>>> import xmlrpclib
>>> proxy = xmlrpclib.ServerProxy(
"http://admin:passwd#localhost:8080/Plone/portal_quickinstaller"
)
>>> proxy.getProductVersion('Marshall')
'2.0'
>>> proxy.isProductInstalled('Marshall')
'False'
>>> proxy.installProduct('Marshall')
'Registry installed sucessfully.\n'
>>> proxy.isProductInstalled('Marshall')
'True'
To reinstall you need subclass Products.CMFQuickInstallerTool.QuickInstallerTool.py and provide you custom QuickInstallerTool with a method that has the keyword argument "reinstall" set as 'True' by default; something like:
442c442
< swallowExceptions=None, reinstall=False,
---
> swallowExceptions=None, reinstall=True,
452,457c452,457
< if self.isProductInstalled(p):
< prod = self._getOb(p)
< msg = ('This product is already installed, '
< 'please uninstall before reinstalling it.')
< prod.log(msg)
< return msg
---
> #if self.isProductInstalled(p):
> # prod = self._getOb(p)
> # msg = ('This product is already installed, '
> # 'please uninstall before reinstalling it.')
> # prod.log(msg)
> # return msg
Even better: provide your own method for gathering information about versions and reinstalling a product, compatible with the XML-RPC protocol (as you cannot pass keyword arguments).
There might be cleaner ways of doing this via XML-RPC, but portal_quickinstaller is not meant to be used in this way and there may be caveats. Use with caution.
I have got this python script in the Zope root of an instance with 7 Plone Sites. Looks pretty much the same as what you have. It might be that it only works on this Plone 2.5 site (yes, old), but I think it should work on 3.x and 4.x as well. Maybe an an innocent looking difference (that I am overlooking) causes the error in your script; maybe the restrictedTraverses that you do trip it up. (Script edited for clarity.)
SITES = ['site-1', 'site-2']
for site in SITES:
print "Reinstalling LinguaPlone in %s." % site
portal = context[site]
qi = portal.portal_quickinstaller
qi.reinstallProducts(['LinguaPlone'])
First don't do a reinstall it can break your website in many cases.
Next you have to consider that add-ons may provide an upgrade step (usually they will). Use the quickinstaller api to achieve this in PythonScript. It is good but it can also be achieved with a script on the file system. Check the examples here: http://svn.plone.org/svn/plone/plone.org/Products.PloneOrg/trunk/scripts/
Another solution can be to use the Selenium IDE to record the quickinstaller stuff in one site and make a copy paste the results of that tests to run it on another website (very weird isn't it ?)