Personal tools
You are here: Home Weblog
Document Actions
  • Send this page to somebody
  • Print this page
  • Add Bookmarklet



ZODB: How to get and read objects from an undo information

Filed Under:

This is useful for example if you have transactions that cause writes to the database and you don't know what has been written. First, identify the ID of the transaction that you're interested. In the Undo tab of Zope, inspect the checkbox for the transaction and copy the part that looks like an id from its value.

Then, in a zope debugging shell (started with bin/instance debug), I've done:

>>> import base64, cPickle
>>> webtid = "QTY1MjhoaytpMVU9"

# DB will be the "database", as represented in Zope.
>>> db = app._p_jar.db()

# Storage will be an instance of FileStorage. I don't think 
# it's possible to achieve this connected through ZEO.
>>> storage =

# This is the "real" transaction id
>>> tid = base64.decodestring(webtid + "\n")

# Now we'll get a "transaction position", a position 
# in the filestorage where the transaction begins
>>> tpos = storage._txn_find(tid, True)

# This will be the transaction header. This is the info that's 
# shown in the Undo UI.
>>> th = storage._read_txn_header(tpos)
>>> print th.status, th.descr, th.user

# Now we get the "data position", the position in the zodb where 
# the objects of that transaction sit
>>> pos = tpos + th.headerlen()

# This will be the data header
>>> dh = storage._read_data_header(pos)

# we're very much interested in the oid
>>> oid = dh.oid
>>> pickle_data, tid = storage.load(oid)

# heh, zodb is just a pickle store, you knew that, right?
>>> print pickle_data
>>> print cPickle.loads(pickle_data)

# but it's easier to get the object using the zope machinery
>>> obj = app._p_jar[oid] 
>>> print obj

The zope process needs to be configured with a FileStorage, not a ClientStorage (aka Zeo client).


The wrong way to sudo su, and the right way

Filed Under:

I've never had a formal training on Linux. My knowledge comes from my experience working on it for the last 12 years, but some things I've just learned as simple recipes and never bother looking them up.

One such thing was changing users with sudo. I've been doing it as:

sudo su <user> -

Notice the dash at the end, which makes provides a "real" environment for <user>, just as if I would have logged in with that user, directly.

But this has ceased to work properly, apparently due to a bug fix in Linux TTY.

The error that I got was:

tibi@localhost:~$ sudo su zope -                                  
bash: cannot set terminal process group (-1): Inappropriate ioctl for device
bash: no job control in this shell

The solution is to switch the position of the dash, to:

sudo su - <user>


Essential Firefox addons

Filed Under:

Without these addons, Firefox may not be my favourite browser. With them, it's a beast that has no match among all other browsers. So, a top 10 of my favourite extensions:

  • Vimperator: the essential plugin, makes Firefox behave in a modal manner similar to VIM. Includes many shortcuts similar to vim. Integrates an external editor (like GVim) with any textarea. My favourite things with it: really fast opening/closing of tabs, URL address copying and pasting, follow links, etc. Just awsome. The external editor thingy has no easy answer in Chrome, where you'd have to be running an http proxy, because addons are sandboxed and can't access the disk.
  • TreeStyleTabs: another awesome plugin, with no match in the Chrome world, where sidebars are an external window.
  • SessionManager: Firefox has a built-in, but SM is better
  • LastPass: while I have Firefox sync enabled, it is better to be able to access those login information from my mobile devices.
  • Auto Unload Tab: when you have 100+ tabs loaded, it's a requirement. Firefox has an automatic unload tab feature, but it can't be controlled and in my experience, it didn't work well.
  • Ghostery + uBlock: get rid of annoying stuff
  • Flashgot: I use it sometimes to make backups of video streams. For youtube I use youtube-dl.
  • Clearly: easy saving content to Evernote

In the past, Firebug would have been at the front of this list, but right now it's buggy, slows down the browser (switching tabs), so it gets ignored in favour of Chromium.


A simple epub file renaming utility

Filed Under:

I have a couple of epub files that have random names assigned to them, and I wanted to rename them based on their metadata, in the form Author - Title. Below is what I came up with:


from zipfile import ZipFile
import lxml.etree
import os.path
import shutil
import sys

namespaces = {

def main():
    if len(sys.argv) != 2:
        print "Need a path."

    fpath = sys.argv[1]
    zip = ZipFile(fpath)
    meta ="META-INF/container.xml")
    e = lxml.etree.fromstring(meta)
    rootfile = e.xpath("/u:container/u:rootfiles/u:rootfile", 
    path = rootfile.get('full-path')
    opf =
    e = lxml.etree.fromstring(opf)
    title = e.xpath("//dc:title", namespaces=namespaces)[0].text
    author = e.xpath("//dc:creator", namespaces=namespaces)[0].text

    base = os.path.dirname(fpath)
    new_name = "%s - %s.epub" % (author, title)
    shutil.move(fpath, os.path.join(base, new_name))

if __name__ == "__main__":


Getting the superclasses for a python object

Filed Under:

Zope 2 (and Plone) persistent objects usually have an intricate inheritance tree. Finding what classes an object inherits can be a time consuming task, hunting through the various eggs for the relevant source code. Below is a little snippet that shows how to easily get the list of superclasses:

(Pdb) pp type(ff).mro()
(<class ''>,
 <class 'archetypes.schemaextender.field.TranslatableExtensionField'>,
 <class 'archetypes.schemaextender.field.BaseExtensionField'>,
 <class ''>,
 <class 'Products.Archetypes.Field.ObjectField'>,
 <class 'Products.Archetypes.Field.Field'>,
 <class 'Products.Archetypes.Layer.DefaultLayerContainer'>,
 <class ''>,
 <class 'Products.Archetypes.Field.ImageField'>,
 <class 'Products.Archetypes.Field.FileField'>,
 <type 'ExtensionClass.Base'>,
 <type 'object'>)

 Credit goes to the original post where I found this.


Version conflict: zc.buildout's version of madness

Filed Under:

I'm not even trying to understand what happens, because it's aggravating to see buildouts fail like this:

  Getting section zope2.
  Initializing section zope2.
  Installing recipe plone.recipe.zope2install.
Error: There is a version conflict.
We already have: setuptools 0.6c9

or, worse, this:

  Getting section zope2.
  Initializing section zope2.
  Installing recipe plone.recipe.zope2install.
Error: There is a version conflict.
We already have: zc.buildout 1.5.2

Well, technically I know what happens: for example, zc.buildout is latest 2.0a1 now, but I've already installed 1.5.2 in my virtualenv (the bootstrap process failed hard, there are a tons of bugs there, I've had more failures in bootstrap then success, lately) and I had one product in my buildout which depended on zc.buildout, so it tried to pull the latest, only to get a version conflict.

I wish zc.buildout would behave more inteligently. Even specifying 

prefer-final = true

didn't do much to solve my problems. The only way to solve the problem was to add the following to the buildout.cfg file:

versions = versions

zc.buildout =1.5.2
setuptools = 0.6c9

UPDATE: a better solution is to have:

zc.buildout =
setuptools =


Building PIL with JPEG support on Ubuntu 11.04

Filed Under:

I had problems building PIL with jpeg support on the latest Ubuntu. There are now two libjpeg libraries: one called libjpeg62 and one libjpeg8. Every guide on the net explaining how to compile PIL with jpeg support points to installing libjpeg62-dev. Needless to say, libjpeg8-dev is actually needed to properly build PIL. My reason for initially avoiding libjpeg8 is that it causes libsdlimage-dev to be uninstalled, so it looks like I'll have to juggle packages whenever I want to compile something that requires SDL.


Export/import users in and out of Plone

Filed Under:

A dirty quick method of importing and exporting the users (only usernames and passwords) out of Plone, using 2 external methods. Code below, not much else to say.

import cPickle

def export(self):
    pas = self.acl_users
    users = pas.source_users
    passwords = users._user_passwords
    result = dict(passwords)

    f = open('/tmp/out.blob', 'w')
    cPickle.dump(result, f)

    return "done"

def import_users(self):
    pas = self.acl_users
    users = pas.source_users
    f = open('/tmp/out.blob')
    res = cPickle.load(f)

    for uid, pwd in res.items():
        users.addUser(uid, uid, pwd)

    return "done"


Set product configuration globally in zope.conf

Filed Under:

I have a Zope product that needs to write in a centralized location, across multiple instances. The classic Python solution would be to write a variable in a module and read that location from the code, but this feels unelegant in an environment that uses zc.buildout for deployment. The solution I have found is, as follows:

In buildout.cfg, in the instance part definition, add:

zope-conf-additional =
        mylocation ${buildout:directory}/var/mylocation

Next, inside the product code I have:

from App.config import getConfiguration
import os

conf = getConfiguration()
dest = conf.environment['mylocation']
if not os.path.exists(dest):

There were 2 things that I had to research for this task: reading the global zope configuration (that's done with App.config.getConfiguration()) and the fact that you can't add arbitrary key/values in zope.conf and have to use the <environment> section.


Running Products.Gloworm on Plone 4

Filed Under:

For some reason, the TTW developer tools tend to get neglected in the Plone world. A valuable tools such as Clouseau has fallen out of favour and now Gloworm, the @@manage-viewlets replacement/complement won't run in Plone 4 (at least at version 1.0, which is the latest right now on PyPI).

Fortunately, Gloworm has been updated in svn trunk. To get the latest version you need to add it to sources (in buildout.cfg or develop.cfg):

eggs +=

Products.Gloworm = svn

Next, run in the shell:

#bin/develop co Products.Gloworm
#bin/develop rb


A pattern for programatically creating Plone content

Filed Under:

I'm importing content from a legacy system to a new website that I'm doing with Plone 4 (wow! what an improvement, in speed and technology) and was looking at the existing documentation on how to programatically create new Plone content. The issue I'm having with the existing documentation is that it's incomplete. It won't give you automatically created ids, you'll have to manually call mutators if you don't know any better, etc.

This is what I have come up with (this code runs in a browser view):

_id = self.context.generateUniqueId("Document")
_id = self.context.invokeFactory(type_name=type_name, id=_id)
ob = self.context[_id]
     description = "text...",
     subject     = ('tag1', 'tag2'),
     title       = "some title...",
_id = ob.getId()

One thing I notice is that Plone 4 starts faster, comes with easily pluggable developer tools, and in general feels a lot more polished then any previous Plone releases. It's a good system to work with.


A miniguide to Dolmen packages

Filed Under:

I'm finally starting a long-overdue project which I have decided to do with Dolmen. As usual, I start by studying its source code and the packages that are available for it. By itself it can will get me about 60% with the requirements for my project, so it's a pretty good starting base. I plan to also study and use some of the menhir.* packages, which are pretty good as generic CMS content types.

Dolmen is an application development framework based on Grok and ZTK which also provides a CMS (Content Management System) out of the box. Dolmen is being made with four main objectives in mind: easily pluggable, rock solid and fast content type development, readability and speed.
Users and group management in Dolmen
Provides a breadcrumbs navigation for the Dolmen applications. It registers a viewlet to render the links.
Provides a useable "clipboard", that allows you to cut, copy and paste your objects.
Is a collection of tools to work with containers in Dolmen applications.
Provides out-of-the-box utilities for Dolmen applications content.
Provides ready-to-use components to get a fully functional and extensively pluggable User Interface for a Dolmen application
Forms and viewlets to edit ZopeDublinCore metadata
Viewlets and utilities for permission-aware searching of objects in a Dolmen site.
Roles and permissions for a Dolmen site
The basic Dolmen objects that serve as roots of Dolmen sites
Allows basic management of alternate views
Basic components for authentication
Zope sessions implementation using beaker
A layer above zope.file using ZODB blobs as a storage facility. It offers a BlobFile content type and a BlobProperty property for complex schemas.
A set of interfaces that apply to basic Python types, to better integrate them with ZCA
Base classes and utilities to create content types
Additional fields usable in schemas. At this moment there's just GlobalClass
Allows you to manage and store files within the ZODB. It takes the core functionalities of, and simplifies them, using Grok for views and adapters registrations.
A package in charge of providing basic functionalities to work with zeam.form Forms.
A package which helps developers create their C.R.U.D forms using Grok, zeam.form and dolmen.content. It provides a collection of base classes to add, edit, and access content. It innovates by providing adapters to customize the fields of a form.
Aims to provide the most flexible and explicit way to create and manage menus and their entries with Grok.
A simple layer on top of zc.async to provide queuing of tasks. Not ready?
Is a thin layer above zc.relation, allowing a simple and straightforward implementation of standalone relationships between objects.
Defines a clear high-level API to deal with pluggable storage components.
Is package specialized in Thumbnail generation. Using the mechanisms, it allows a pluggable and flexible thumbnail storage.
A package that walks hand-in-hand with dolmen.file. It provides a useable and pluggable way to render the dolmen.file.FileField in a zeam.form Form.
A thin layer above dolmen.widget.file providing a widget suitable to fields implementing IImageField. It adds, thanks to dolmen.thumbnailer a preview of the uploaded image in both input and display mode.
A package that provides a useable and pluggable way to render a text field as a WYSIWG editor in a zeam.form Form.
Nothing here
Allows registration of icons and associating them with content types
Allows concatanation and packing of browser resources (css and js)
An example document content type
An example file content type
An example folder content type
An example image content type
An example photoalbum content type
An example rstdocument content type
An example user content type
Registers a jquery based library for HTML tables sorting
Simple commenting system with avatar integration
A viewlet that provides a livesearch box
A viewlet providing a navigation tree
A tagging engine based on the lovely.tag
A complete skin for a Dolmen site.
A skin for Snappy sites
The Snappy, a video sharing sample site
Mimetype transform utilities. Not finished?
Utilities to mark files as Flash and allow to view them.
A video player for flash movies
Convert video files to flash movies and thumbnails
A form library designed to be grokish and simple
zope.schema integration for zeam.form. It provides widgets and default CRUD style actions.
A few tutorials for Dolmen
A Paste script extension that allows quick bootstrapping of new Dolmen projects

To download all the packages, I've ctrl+selected the git repositories names from, pasted them into a repositories.txt file and ran the following script:


import subprocess
import os

f = open('repositories.txt')
for line in f.readlines():
    git = line.strip()
    pkg = git
    if pkg.endswith('.git'):
        pkg = ".".join(pkg.split('.')[:-1])

    if os.path.exists(pkg):
        subprocess.check_call(['git', 'pull'], cwd=pkg)
        subprocess.check_call(['git', 'clone', 'git://' + git])



Migrating content (folders) from Plone 3 to Plone 4 via zexp import

Filed Under:

I had a need (and a problem) moving some content from a Zope 2.10/ Plone 3.3 instance to a Zope 2.12/Plone 4 instance. The path I have chosen was that of the least resistence, which for me was exporting the folder I was interested as a zexp file from the old instance and importing it in the new Plone instance. According to some members of the #plone IRC channel, this method of getting content from one zope instance to another is not possible, or at least not supported. I supposed that's correct, zexp import works best for moving content between identical zope instances, but, as they say, necessity is the mother of learning.

The issue is that the implementation of folders has changed from Plone 3 to 4 to use BTrees, which greatly improves performance. The problem is that, when viewing imported folders, I got the following traceback:

Traceback (innermost last):

    * Module ZPublisher.Publish, line 116, in publish
    * Module ZPublisher.BaseRequest, line 434, in traverse
    * Module Products.CMFCore.DynamicType, line 150, in __before_publishing_traverse__
    * Module Products.CMFDynamicViewFTI.fti, line 215, in queryMethodID
    * Module Products.CMFDynamicViewFTI.fti, line 182, in defaultView
    * Module Products.CMFPlone.PloneTool, line 840, in browserDefault
    * Module Products.CMFPlone.PloneTool, line 708, in getDefaultPage
    * Module Products.CMFPlone.utils, line 81, in getDefaultPage
    * Module, line 32, in getDefaultPage
    * Module, line 75, in getDefaultPage
    * Module Products.BTreeFolder2.BTreeFolder2, line 337, in has_key

AttributeError: 'NoneType' object has no attribute 'has_key' 

The solution was to call @@migrate-btrees on the imported folder, which fixes that folder and makes it conform to the latest implementation.

One final note, the default Plone buildout doesn't have a folder called "import" anywhere in the buildout, so one needs to be created inside the "client home folder", which is the folder of your plone and buildout instance, the one that hosts the bin, parts and var folders.


Some issues with zc.recipe.egg's python option

Filed Under:

I've recently had to integrate a script/package into a Plone 2.5 buildout that runs on top of Python 2.4. Due to that package's dependence of a sane imaplib (and the one in Python 2.4 is buggy), I had to run the script with python2.6. To make a script run on a different python, you need to do:

recipe = zc.regipe.egg
eggs = 
python = python26

The python26 option is actually the name of a buildout part that configures the python executable path

python = /usr/bin/python26

Now the problems. I've had various buildouts fail with a message "Cannot find egg myegg". After a bit of effort, we managed to trace the cause to this problem:

First, the python path in the [python26] part was incorect. Second, even if it pointed to the proper binary, the -devel packages for that python needed to be installed.

Well, now I know. Hopefully I'll remember it for the next time when I'll encounter the problem.


Can you do this on your shiny Mac?

Filed Under:

Probably you can, but you have never done it because you have a shiny interface for everything. I'm talking about this discovery of mine:

svn diff | kompare -

What it does is to take the output from svn diff and pipe it into Kompare, a merge/diff utility from the KDE Project. I can do this from the command line, straight from the directory that I'm in, and bang! I get a nice graphical overview, complete with the tree structure that I can navigate to see what I'm about to commit. Honestly, this little command gets me excited everytime I run it.


Generating products outside of the Products.* namespace with ArchGenXML

Filed Under:

I'm a die hard in regards to ArchGenXML usage. The number of things to know about when creating new content types for Plone is just too high. Package structure, Zope package registration, content types registration, QuickInstaller registration, GenericSetup profiles, skins registration, workflows, etc. I can go in and do changes to the code, and add to it, but generating it from scratch is a gigantic task, especially for my use case, where I need to start a new project with about 7 content types.

Now to the problem: ArchGenXML assumes (and hardcodes in its templates) the Products.* prefix for your package. I don't have any problems with it, but my employer uses a different package structure so for uniformity I need to follow their standards. 15 minutes of poking and changing through agx enabled me to change it so it would generate GS xml files with the proper namespace (based on a new model level TGV named "namespace" that I have created). After assessing the difficulty of the task and being under time pressure, I've decided to go the dumb route, which I'm documenting below:

My generation script is something like this:

./archgenxml -c archgenxml.cfg myproduct.zuml 

#rename Products.myproduct to ns.myproduct
find myproduct/* -type f -print | xargs sed -i 's/Products\.myproduct/ns\.myproduct/g'

#in the xml type profiles, replace myproduct by ns.myproduct
find myproduct/* -type f -print | xargs sed -i 's/>myproduct</>ns\.myproduct</g'

#in the profiles.zcml, rename the profile to ns.myproduct
find myproduct/* -type f -print | xargs sed -i 's/title=\"myproduct\"/title=\"ns\.myproduct\"/g'

Notice the sed lines, which change the source code to point to "ns.myproduct" instead of "Products.myproduct". Now there's just two more problems, which are actually AGX bugs:

  • the FilesystemDirectoryViews registered for the skin are incorect, so you'll need to insert this in ns/myproduct/
from Products.CMFCore import utils
from Globals import package_home
from os.path import dirname
ppath = utils.ProductsPath
DirectoryView.registerDirectory('skins', product_globals)
utils.ProductsPath = ppath
  • trying to get AGX to output the code inside a two level deep folder doesn't work (it crashes), so you'll need to move this script file and the model just above the "myproduct" folder, inside the "ns" namespace.

That's about it. Hopefully I'll find the time to fix the two bugs and add the namespace improvement in the next week. Still need to do the review for Plone for Education, which I have received for free from the publisher.


Another cause for buildout failures: system distributed Python

Filed Under:

Always compile your own! One day I'll even remember that...

I've had a buildout bootstrap process failure, this time a weird one, perhaps I should document the bug and report it.

The latest Ubuntu version which I have installed (Lucid Lynx) comes with a package called python-pkg-resources, which packages pkg_resources, which used to be available only through the setuptools distribution. Buildout's tries to guess if Setuptools or Distribute are installed by checking the availability of pkg_resources; by guessing wrong it all comes to a crash at the end.

I'm not very interested in debugging these types of problems anymore. Distribution/packaging tools should just work. I want to focus on my work, not debug the toolchain. No more corner cases or whatver. So I'm gonna compile separate Pythons in the future, especially when dealing with older Zope/Plones.

Dear PyPi uploaders: don't use a download URL, upload your package instead!

Pypi's biggest mistake: allowing package entries without any upload

I think this is the Python Index biggest mistake, the one which makes it unreliable for serious development environments: exposing package entries with no real package files and just a download URL. To see what I'm talking about, just examine the PyPI records for BeautifulSoup or IPython, packages that are very common in buildouts. As soon as the author and publisher of that package has a hosting problem, the developer that uses that package also has a problem. Buildouts will completely fail and this will cause dead times and frustration for the developers.

Yes, there are a couple of PyPi mirrors, but they only mirror files hosted by PyPi. The central PyPi site will probably have better performance and availability then what individual groups and developers can provide and it's always easier to mirror one single website than many, so there's no shame or loss of pride in using the PyPi to host your files. Please do so!



Test if developer mode is set in Zope 3 and Grok

Filed Under:

I've started an application that uses Dolmen, a lightweight CMS built on top of Grok, and I want to be able to "rollup" the megrok.resource files based on the devmode setting that was set in the zope.conf file. After a bit of digging, I came up with this code that tells me the current devmode setting for the running instance:

from import applicationController
from import IRuntimeInfo

def get_debug_mode():
    """Returns the devmode setting for a running zope instance"""

    rti = IRuntimeInfo(applicationController)
    return rti.getDeveloperMode() == "On"



A "don't do" for internationalizing Django templates

Filed Under:

I'm internationalizing a Pinax website and I've encountered this piece of code in a template:

<input type="submit" value="{% trans "invite" %}"/>
{% blocktrans %}{{ other_user }} to be a friend.{% endblocktrans %}

The message ids for this code would be two separate blocks: "invite" and " %{other_user}s to be a friend". Both offer very little in terms of context and make the translators job difficult. Correct, in my point of view, would be the more convoluted form of:

{% blocktrans %}
<input type="submit" value="invite"/>
{{ other_user }} to be a friend.
{%  endblocktrans %}

This implies that the translators know enough HTML to notice that the value attribute needs to be translated, but the end result is a lot more flexible and provides real context to them.

TL;DR: don't split paragraphs into separate translation units. It's a NO-NO.

UPDATE: I have found what is probably the worst example of how to create a translatable template. Remember, don't assume the English language resembles anything like another language.

{% trans "edited by user" %} {{ obj.editor.username }} {% trans "at"%} {{ obj.modified|date:"H:i" }}

This should be done this way:

{% blocktrans with obj.editor.username as editor_username and obj.modified|date:"H:i" as obj_modified
edited by user {{ editor_username }} at {{ obj_modified }}
{% endblocktrans %}
RSS 2.0
Powered by Quills
Creative Commons License
This work is licensed under a Creative Commons Attribution 3.0 License.

Powered by Plone CMS, the Open Source Content Management System

This site conforms to the following standards: