I have a couple of epub files that have random names assigned to them, and I wanted to rename them based on their metadata, in the form Author - Title. Below is what I came up with:

#!/usr/bin/python

from zipfile import ZipFile
import lxml.etree
import os.path
import shutil
import sys

namespaces = {
        'u':"urn:oasis:names:tc:opendocument:xmlns:container",
        'xsi':"http://www.w3.org/2001/XMLSchema-instance",
        'opf':"http://www.idpf.org/2007/opf", 
        'dcterms':"http://purl.org/dc/terms/",
        'calibre':"http://calibre.kovidgoyal.net/2009/metadata",
        'dc':"http://purl.org/dc/elements/1.1/",
        }

def main():
    if len(sys.argv) != 2:
        print "Need a path."
        sys.exit(1)

    fpath = sys.argv[1]
    zip = ZipFile(fpath)
    meta = zip.read("META-INF/container.xml")
    e = lxml.etree.fromstring(meta)
    rootfile = e.xpath("/u:container/u:rootfiles/u:rootfile", 
            namespaces=namespaces)[0]
    path = rootfile.get('full-path')
    opf = zip.read(path)
    e = lxml.etree.fromstring(opf)
    title = e.xpath("//dc:title", namespaces=namespaces)[0].text
    author = e.xpath("//dc:creator", namespaces=namespaces)[0].text


    base = os.path.dirname(fpath)
    new_name = "%s - %s.epub" % (author, title)
    shutil.move(fpath, os.path.join(base, new_name))

if __name__ == "__main__":
    main()

Zope 2 (and Plone) persistent objects usually have an intricate inheritance tree. Finding what classes an object inherits can be a time consuming task, hunting through the various eggs for the relevant source code. Below is a little snippet that shows how to easily get the list of superclasses:

(Pdb) pp type(ff).mro()
(<class 'plone.app.blob.subtypes.image.ExtensionBlobField'>,
 <class 'archetypes.schemaextender.field.TranslatableExtensionField'>,
 <class 'archetypes.schemaextender.field.BaseExtensionField'>,
 <class 'plone.app.blob.field.BlobField'>,
 <class 'Products.Archetypes.Field.ObjectField'>,
 <class 'Products.Archetypes.Field.Field'>,
 <class 'Products.Archetypes.Layer.DefaultLayerContainer'>,
 <class 'plone.app.blob.mixins.ImageFieldMixin'>,
 <class 'Products.Archetypes.Field.ImageField'>,
 <class 'Products.Archetypes.Field.FileField'>,
 <type 'ExtensionClass.Base'>,
 <type 'object'>)

 Credit goes to the original post where I found this.

I'm not even trying to understand what happens, because it's aggravating to see buildouts fail like this:

While:
  Installing.
  Getting section zope2.
  Initializing section zope2.
  Installing recipe plone.recipe.zope2install.
Error: There is a version conflict.
We already have: setuptools 0.6c9

or, worse, this:

While:
  Installing.
  Getting section zope2.
  Initializing section zope2.
  Installing recipe plone.recipe.zope2install.
Error: There is a version conflict.
We already have: zc.buildout 1.5.2

Well, technically I know what happens: for example, zc.buildout is latest 2.0a1 now, but I've already installed 1.5.2 in my virtualenv (the bootstrap process failed hard, there are a tons of bugs there, I've had more failures in bootstrap then success, lately) and I had one product in my buildout which depended on zc.buildout, so it tried to pull the latest, only to get a version conflict.

I wish zc.buildout would behave more inteligently. Even specifying 

prefer-final = true

didn't do much to solve my problems. The only way to solve the problem was to add the following to the buildout.cfg file:

[buildout]
...
versions = versions

[versions]
zc.buildout =1.5.2
setuptools = 0.6c9

I had problems building PIL with jpeg support on the latest Ubuntu. There are now two libjpeg libraries: one called libjpeg62 and one libjpeg8. Every guide on the net explaining how to compile PIL with jpeg support points to installing libjpeg62-dev. Needless to say, libjpeg8-dev is actually needed to properly build PIL. My reason for initially avoiding libjpeg8 is that it causes libsdlimage-dev to be uninstalled, so it looks like I'll have to juggle packages whenever I want to compile something that requires SDL.

A dirty quick method of importing and exporting the users (only usernames and passwords) out of Plone, using 2 external methods. Code below, not much else to say.

import cPickle

def export(self):
    pas = self.acl_users
    users = pas.source_users
    passwords = users._user_passwords
    result = dict(passwords)

    f = open('/tmp/out.blob', 'w')
    cPickle.dump(result, f)
    f.close()

    return "done"

def import_users(self):
    pas = self.acl_users
    users = pas.source_users
    f = open('/tmp/out.blob')
    res = cPickle.load(f)
    f.close()

    for uid, pwd in res.items():
        users.addUser(uid, uid, pwd)

    return "done"

I have a Zope product that needs to write in a centralized location, across multiple instances. The classic Python solution would be to write a variable in a config.py module and read that location from the code, but this feels unelegant in an environment that uses zc.buildout for deployment. The solution I have found is, as follows:

In buildout.cfg, in the instance part definition, add:

zope-conf-additional =
    <environment>
        mylocation ${buildout:directory}/var/mylocation
    </environment>

Next, inside the product code I have:

from App.config import getConfiguration
import os

conf = getConfiguration()
dest = conf.environment['mylocation']
if not os.path.exists(dest):
    os.mkdir(dest)

There were 2 things that I had to research for this task: reading the global zope configuration (that's done with App.config.getConfiguration()) and the fact that you can't add arbitrary key/values in zope.conf and have to use the <environment> section.

For some reason, the TTW developer tools tend to get neglected in the Plone world. A valuable tools such as Clouseau has fallen out of favour and now Gloworm, the @@manage-viewlets replacement/complement won't run in Plone 4 (at least at version 1.0, which is the latest right now on PyPI).

Fortunately, Gloworm has been updated in svn trunk. To get the latest version you need to add it to sources (in buildout.cfg or develop.cfg):

[buildout]
eggs +=
     Products.Gloworm

[sources]
Products.Gloworm = svn https://weblion.psu.edu/svn/weblion/weblion/Products.Gloworm/trunk/

Next, run in the shell:

#bin/develop co Products.Gloworm
#bin/develop rb

I'm importing content from a legacy system to a new website that I'm doing with Plone 4 (wow! what an improvement, in speed and technology) and was looking at the existing documentation on how to programatically create new Plone content. The issue I'm having with the existing documentation is that it's incomplete. It won't give you automatically created ids, you'll have to manually call mutators if you don't know any better, etc.

This is what I have come up with (this code runs in a browser view):

_id = self.context.generateUniqueId("Document")
_id = self.context.invokeFactory(type_name=type_name, id=_id)
ob = self.context[_id]
ob.edit(
     description = "text...",
     subject     = ('tag1', 'tag2'),
     title       = "some title...",
 )
ob._renameAfterCreation(check_auto_id=True)
_id = ob.getId()

One thing I notice is that Plone 4 starts faster, comes with easily pluggable developer tools, and in general feels a lot more polished then any previous Plone releases. It's a good system to work with.

I've recently had to integrate a script/package into a Plone 2.5 buildout that runs on top of Python 2.4. Due to that package's dependence of a sane imaplib (and the one in Python 2.4 is buggy), I had to run the script with python2.6. To make a script run on a different python, you need to do:

[myscript]
recipe = zc.regipe.egg
eggs = 
     myegg
     IMAPClient
python = python26

The python26 option is actually the name of a buildout part that configures the python executable path

[python26]
python = /usr/bin/python26

Now the problems. I've had various buildouts fail with a message "Cannot find egg myegg". After a bit of effort, we managed to trace the cause to this problem:

First, the python path in the [python26] part was incorect. Second, even if it pointed to the proper binary, the -devel packages for that python needed to be installed.

Well, now I know. Hopefully I'll remember it for the next time when I'll encounter the problem.

Probably you can, but you have never done it because you have a shiny interface for everything. I'm talking about this discovery of mine:

svn diff | kompare -

What it does is to take the output from svn diff and pipe it into Kompare, a merge/diff utility from the KDE Project. I can do this from the command line, straight from the directory that I'm in, and bang! I get a nice graphical overview, complete with the tree structure that I can navigate to see what I'm about to commit. Honestly, this little command gets me excited everytime I run it.

I'm a die hard in regards to ArchGenXML usage. The number of things to know about when creating new content types for Plone is just too high. Package structure, Zope package registration, content types registration, QuickInstaller registration, GenericSetup profiles, skins registration, workflows, etc. I can go in and do changes to the code, and add to it, but generating it from scratch is a gigantic task, especially for my use case, where I need to start a new project with about 7 content types.

Now to the problem: ArchGenXML assumes (and hardcodes in its templates) the Products.* prefix for your package. I don't have any problems with it, but my employer uses a different package structure so for uniformity I need to follow their standards. 15 minutes of poking and changing through agx enabled me to change it so it would generate GS xml files with the proper namespace (based on a new model level TGV named "namespace" that I have created). After assessing the difficulty of the task and being under time pressure, I've decided to go the dumb route, which I'm documenting below:

My generation script is something like this:

./archgenxml -c archgenxml.cfg myproduct.zuml 

#rename Products.myproduct to ns.myproduct
find myproduct/* -type f -print | xargs sed -i 's/Products\.myproduct/ns\.myproduct/g'

#in the xml type profiles, replace myproduct by ns.myproduct
find myproduct/* -type f -print | xargs sed -i 's/>myproduct</>ns\.myproduct</g'

#in the profiles.zcml, rename the profile to ns.myproduct
find myproduct/* -type f -print | xargs sed -i 's/title=\"myproduct\"/title=\"ns\.myproduct\"/g'

Notice the sed lines, which change the source code to point to "ns.myproduct" instead of "Products.myproduct". Now there's just two more problems, which are actually AGX bugs:

• the FilesystemDirectoryViews registered for the skin are incorect, so you'll need to insert this in ns/myproduct/__init__.py

from Products.CMFCore import utils
from Globals import package_home
from os.path import dirname
ppath = utils.ProductsPath
utils.ProductsPath.append(dirname(package_home(product_globals)))
DirectoryView.registerDirectory('skins', product_globals)
utils.ProductsPath = ppath

• trying to get AGX to output the code inside a two level deep folder doesn't work (it crashes), so you'll need to move this script file and the model just above the "myproduct" folder, inside the "ns" namespace.

That's about it. Hopefully I'll find the time to fix the two bugs and add the namespace improvement in the next week. Still need to do the review for Plone for Education, which I have received for free from the publisher.

I've started an application that uses Dolmen, a lightweight CMS built on top of Grok, and I want to be able to "rollup" the megrok.resource files based on the devmode setting that was set in the zope.conf file. After a bit of digging, I came up with this code that tells me the current devmode setting for the running instance:

from zope.app.applicationcontrol.applicationcontrol import applicationController
from zope.app.applicationcontrol.interfaces import IRuntimeInfo

def get_debug_mode():
    """Returns the devmode setting for a running zope instance"""

    rti = IRuntimeInfo(applicationController)
    return rti.getDeveloperMode() == "On"

I'm internationalizing a Pinax website and I've encountered this piece of code in a template:

<input type="submit" value="{% trans "invite" %}"/>
{% blocktrans %}{{ other_user }} to be a friend.{% endblocktrans %}

The message ids for this code would be two separate blocks: "invite" and " %{other_user}s to be a friend". Both offer very little in terms of context and make the translators job difficult. Correct, in my point of view, would be the more convoluted form of:

{% blocktrans %}
<input type="submit" value="invite"/>
{{ other_user }} to be a friend.
{%  endblocktrans %}

This implies that the translators know enough HTML to notice that the value attribute needs to be translated, but the end result is a lot more flexible and provides real context to them.

TL;DR: don't split paragraphs into separate translation units. It's a NO-NO.

UPDATE: I have found what is probably the worst example of how to create a translatable template. Remember, don't assume the English language resembles anything like another language.

{% trans "edited by user" %} {{ obj.editor.username }} {% trans "at"%} {{ obj.modified|date:"H:i" }}

This should be done this way:

{% blocktrans with obj.editor.username as editor_username and obj.modified|date:"H:i" as obj_modified
edited by user {{ editor_username }} at {{ obj_modified }}
{% endblocktrans %}

Yet another angry rant, caused, of course, by using Django in anger. Nothing wrong with using something in anger, that's the real way I learn something. Zope 3 even has an online book on how to use it in anger.

That said, Django's makemessages administrative command sucks by being way too inflexible to anything but the ideal Django development environment. My environment looks like this: I have a project based on Pinax, which I'm developing and deploying using zc.buildout. My source code sits in src, where I have several packages. I also have a "localsettings.py" module located in the root of the buildout, because I don't want to have it inside the src folder. Pinax is located in parts/Pinax, and it's actually a git checkout, based on my own fork of Pinax. Pinax doesn't have translations at the moment (I think I saw a ticket in its tracker about reintroducing a translation package), so I'm on my own here with regards to translation.

With this setup, it is close to impossible for me to generate anything useful without a lot of hacking and swearing. Makemessages insists on being run from inside a Django project, and when I did that, it complains about missing localsettings module. Pointing the root of the buildout as pythonpath didn't do anything. A good thing that I have already extracted messages from the templates, before switching to the buildout project structure.

Some solutions that I have found:

• I can extract messages from the Pinax python modules using this homegrown script:
  

PYFILES=/tmp/pyfiles
PINAX=parts/Pinax/pinax/ 
BASE=/home/tibi/work/ProjectBuildout/src/project/locale/ro/LC_MESSAGES/
POTFILE_PYTHON=$BASE/python.pot
POTFILE_TEMPLATES=$BASE/templates.pot
POTFILE=$BASE/django.pot
POFILE=$BASE/django.po

#extract messages from python code
find $PINAX | grep ".*py$" > /tmp/pyfiles
touch $POTFILE_PYTHON
xgettext -j -L python -d django -f $PYFILES -o $POTFILE_PYTHON

#merge the templates + python messages into one pot file
msgcat -o $POTFILE $POTFILE_TEMPLATES $POTFILE_PYTHON

#merge the potfile with the po file
msgmerge -U -N $POFILE $POTFILE

• I have copied all the templates from pinax and its associated applications inside a template folder in my project. Now I can generate the po file, from my src/project folder, with

../../bin/py ./../../manage.py makemessages -e .py -e .html -l ro

Of course, I can't run this over the other apps and packages in my src/ folder to extract messages from the python modules, so I am forced to adjust the first script to take those folders into consideration.

A bug creeped in one of my Django 1.1 projects that is in beta-testing right now: the Pinax wiki app looks for a REMOTE_ADDR value in request.META, which was not set in my environment. My environment is a pretty standard (as far as this setup goes) nginx + fcgi (flup on the django side) + django. Further work on this issue revealed, step by step, that:

• REMOTE_ADDR needs to be somehow set by a Django middleware, based on an http header,
  
• so I've added django.middleware.http.SetRemoteAddrFromForwardedFor to the list of loaded middleware
• but that middleware is deprecated in Django 1.1 and does nothing, so I rewrote this middleware based on Chapter 15 of Django Book
• this new middleware did its job, but for some reason flup stripped that header from the request and the proper values never got to Django
• this made me look for a replacement for flup, which I found in django-cpserver.

This package adds a new admin command, 'runcpserver', which replaces the default development server by something more appropriate to production. The word on the 'net-streets' is that it runs well and does its job, so I'll be using it for the time being. The recommended solution is to run apache+mod_wsgi and proxy that to nginx, but right now I don't want the extra administrative overhead that Apache represents. If I can't get enough "juice" out of one instance of cpserver, I'll just add an extra instance and balance them with nginx or haproxy.

Probably this is documented somewhere in the Django docs, but, I mean, who has time to read the docs? :-) In the urls.py file for my project I didn't do a

from django.conf.urls.defaults import *

because that's just bad style. Instead I've just imported what I needed: patterns, include and url. Later on I got this error in my email:

Traceback (most recent call last):

  File "/home/zope/djangoprojects/lib/python2.5/site-packages/Django-1.1.1-py2.5.egg/django/core/handlers/base.py", line 118, in get_response
    callback, param_dict = resolver.resolve404()

  File "/home/zope/djangoprojects/lib/python2.5/site-packages/Django-1.1.1-py2.5.egg/django/core/urlresolvers.py", line 263, in resolve404
    return self._resolve_special('404')

  File "/home/zope/djangoprojects/lib/python2.5/site-packages/Django-1.1.1-py2.5.egg/django/core/urlresolvers.py", line 255, in _resolve_special
    callback = getattr(self.urlconf_module, 'handler%s' % view_type)

AttributeError: 'module' object has no attribute 'handler404'

Just looking at the error I could determine the cause: my urlconf module (urls.py) didn't have a handler404 view defined, so I just gave up and replaced my specific imports with the asterisk import.

On a side note, what's up with Django logging? By default, the only way to log errors in production environments is by email. Really!? If I would want emails, I'd setup the syslog to email me those entries. I already have the logwatch emailing me stuff that happens on my server. One step above is django-db-log, which I haven't tried yet, but which seems to be logging errors to the db. Word is that real logging, using the logging python module, will be introduced in Django 1.2.

I'm not a Django developer, but I have a project that involves deploying a LFS site, and it took me a bit of time and effort to understand how to customize its templates and resources. This all may seem obvious to the any Django developer, but I'm not, so I think this info needs to be out there.

As a side note, at first I have reviewed Satchmo, as it seemed to have more traction in the Django community (being a Zope/Plone developer, I am more aware of the previous work done by the LFS developer, the EasyShop). The Satchmo installation, for the un-initiated, seems to be a nightmare of many different settings that needed to be tweaked, skeletons that don't work OOTB, at least in the way I have installed them. On top of all this, it seems to think that I also need to develop some sort of project that will integrate with the Satchmo shop (but this may be due to Django). What happened to treating your product as the end product? After a bit of effort, I had one instance running and ready to review it. Several days, in another Linux virtual machine (I've switched from VMWare to VirtualBox), I've tried to install Satchmo again, this time using djangorecipe. I didn't have the same patience, though.

A short look at LFS convinced me to at least give it a try: there is a buildout that can be installed easily (although there were a few dependencies that I had to install), the "backend" is polished compared to the default django admin views used by Satchmo, and, while it is simpler, less featured, than Satchmo, it does everything I need.

Onward to customizing the LFS: my goal is to be able to change templates (for example, the base layout) and publish a new logo.

I've created a new folder "mytheme" inside the buildout, with two subfolders: "templates" and "static". The path to the templates folder needs to be inserted in the "TEMPLATES_DIRS" variables from settings.py. To customize the base template, for example, I've created a "lfs/base.html" file inside the "templates" folder. Its content can be copied from the original in the lfs_theme folder. I've added a new line in the header section, pointing to a new CSS file that will customize the base LFS stylesheet.

<link rel="stylesheet" type="text/css" href="{{ MEDIA_URL }}mytheme/css/custom.css" />

As a side note, this templates folder can be used for overriding any other template: templates in Django are usually denoted by a relative path, and you need only to replicate this path inside our overriding templates folder.

Next, overriding and creating new resources: I have created a "static" folder inside "mytheme" and I've linked to it inside the lfs_project/media folder. Inside the "mytheme" folder I can now host images and files, in 2 new "css" and "img" subfolders. Back in my customized base.html, I have changed the logo path:

<img class="logo" src="{{ MEDIA_URL }}mytheme/img/logo.jpg" alt="Logo" />

That's about it: I got a custom CSS, the ability to customize templates and to point to new images. I'm happy for now.

As I have said in the beginning, this is all basic stuff. No harm done in pointing to the obvious though, as there seem to be many new developers flocking to Django, all with different level of expertise.

I'm checking out a large repository and svn keeps getting blocked, for some reason (bad network, maybe). When running update again, it complains that the checkout should be cleaned, but the cleanup operation doesn't succeed, with something like this:

tibi@xps:~/svn/vs/data$ svn cleanup                    
svn: In directory '.'                                                
svn: Error processing command 'modify-wcprop' in '.'                 
svn: 'setup.config' is not under version control

Searching on then net for a solution didn't yield anything, so the one that I have found was to empty the .svn/log file (by default is read only) and then to run svn up again.

I've upgraded CacheFu (Products.CacheSetup) to the latest 1.2 for some of the websites that I manage, in a Plone 2.5 cluster. One of them had a problem that I haven't been able to track previously, due to limited time: on a reinstall of CacheSetup, due to product upgrades, CacheFu couldn't be installed anymore. The traceback was something like:

this product has already been installed without Quickinstaller!failed:
Traceback (most recent call last):

  File "/home/zope/z29/Products/CMFQuickInstallerTool/QuickInstallerTool.py", line 330, in installProduct

  File "/home/zope/p25/parts/zope2/lib/python/Products/ExternalMethod/ExternalMethod.py", line 225, in __call__
    try: return f(*args, **kw)

  File "/home/zope/p25/eggs/Products.CacheSetup-1.2-py2.4.egg/Products/CacheSetup/Extensions/Install.py", line 35, in install
    policy_utils.addCachePolicies(self, out)

  File "/home/zope/p25/eggs/Products.CacheSetup-1.2-py2.4.egg/Products/CacheSetup/Extensions/policy_utils.py", line 72, in addCachePolicies
    p.addCacheRules(rules)

  File "/home/zope/p25/eggs/Products.CacheSetup-1.2-py2.4.egg/Products/CacheSetup/Extensions/policy_2.py", line 13, in addCacheRules
    rules.invokeFactory(id=id, type_name='PolicyHTTPCacheManagerCacheRule')

  File "/home/zope/z29/Products/CMFCore/PortalFolder.py", line 408, in invokeFactory

  File "/home/zope/z29/Products/CMFCore/TypesTool.py", line 934, in constructContent

  File "/home/zope/z29/Products/CMFCore/TypesTool.py", line 343, in constructInstance

  File "/home/zope/z29/Products/CMFCore/TypesTool.py", line 574, in _constructInstance

  File "", line 6, in addPolicyHTTPCacheManagerCacheRule

  File "/home/zope/p25/parts/zope2/lib/python/OFS/ObjectManager.py", line 301, in _setObject
    v = self._checkId(id)

  File "/home/zope/z29/Products/CMFCore/Skinnable.py", line 223, in _checkId

  File "/home/zope/p25/parts/zope2/lib/python/OFS/ObjectManager.py", line 95, in checkValidId
    raise BadRequest, (

BadRequest: The id "httpcache" is invalid - it is already in use.

One other weird thing are the paths in this traceback: /home/zope/z29 doesn't exist anymore, the database was moved from a different server. I think it's related to the persistent product entries in the Control_Panels, which can be cleared. Not a big problem. In the log, there was also an entry related to this traceback:

2009-02-28 17:01:18 CRITICAL txn.-1223480432 A storage error occurred during the second phase of the two-phase commit.  Resources may be in an inconsistent state

Now, the solution is really simple, but I needed to debug the policy_2 module to find this: 

rules.invokeFactory(id=id, type_name='PolicyHTTPCacheManagerCacheRule')

didn't work because there was a document in the root called "rules". Nobody expects the spamish Acquisition! And I didn't either... Lesson? Zope 2 was designed to be too smart for its own good, thus violating the KISS principle. Still love it, though.

While testing software to play music from a computer on my network, I have discovered Elisa, which is a wonderful piece of software built in Python and zope.interface. The package manager helpfully installed a python-zopeinterface package, which turned out to break one of the apps I've been working on (one of the packages that is used depends on a more recent version of zope.interface and breaks with a missing object import). Adding an explicit dependency on zope.interface>=3.5 didn't help either. The egg was installed and a reference to it was inserted in the generated script wrapper for the buildout's bin folder, but the system zope.interface was found.

The solution that I have found was to make sure the zope.interface dependency is listed in the last position in the install_requires section of setup.py. This has the effect of placing the zope.interface egg path first in the generated script, and thus solving the problem.

UPDATE: On another project I'm working on, this solution didn't work. Buildout would complain about a version conflict and would drop the building process. The solution was to setup a separate virtualenv bootstrapped with --no-site-packages and use the python from that virtualenv to bootstrap the buildout environment. I think it's a bug in zc.buildout, as it should have obeyed the versions section of buildout, plus the explicit dependency in install_requires of my package setup.py