I had a need (and a problem) moving some content from a Zope 2.10/ Plone 3.3 instance to a Zope 2.12/Plone 4 instance. The path I have chosen was that of the least resistence, which for me was exporting the folder I was interested as a zexp file from the old instance and importing it in the new Plone instance. According to some members of the #plone IRC channel, this method of getting content from one zope instance to another is not possible, or at least not supported. I supposed that's correct, zexp import works best for moving content between identical zope instances, but, as they say, necessity is the mother of learning.

The issue is that the implementation of folders has changed from Plone 3 to 4 to use BTrees, which greatly improves performance. The problem is that, when viewing imported folders, I got the following traceback:

Traceback (innermost last):

    * Module ZPublisher.Publish, line 116, in publish
    * Module ZPublisher.BaseRequest, line 434, in traverse
    * Module Products.CMFCore.DynamicType, line 150, in __before_publishing_traverse__
    * Module Products.CMFDynamicViewFTI.fti, line 215, in queryMethodID
    * Module Products.CMFDynamicViewFTI.fti, line 182, in defaultView
    * Module Products.CMFPlone.PloneTool, line 840, in browserDefault
    * Module Products.CMFPlone.PloneTool, line 708, in getDefaultPage
    * Module Products.CMFPlone.utils, line 81, in getDefaultPage
    * Module plone.app.layout.navigation.defaultpage, line 32, in getDefaultPage
    * Module plone.app.layout.navigation.defaultpage, line 75, in getDefaultPage
    * Module Products.BTreeFolder2.BTreeFolder2, line 337, in has_key

AttributeError: 'NoneType' object has no attribute 'has_key' 

The solution was to call @@migrate-btrees on the imported folder, which fixes that folder and makes it conform to the latest implementation.

One final note, the default Plone buildout doesn't have a folder called "import" anywhere in the buildout, so one needs to be created inside the "client home folder", which is the folder of your plone and buildout instance, the one that hosts the bin, parts and var folders.

I've recently had to integrate a script/package into a Plone 2.5 buildout that runs on top of Python 2.4. Due to that package's dependence of a sane imaplib (and the one in Python 2.4 is buggy), I had to run the script with python2.6. To make a script run on a different python, you need to do:

[myscript]
recipe = zc.regipe.egg
eggs = 
     myegg
     IMAPClient
python = python26

The python26 option is actually the name of a buildout part that configures the python executable path

[python26]
python = /usr/bin/python26

Now the problems. I've had various buildouts fail with a message "Cannot find egg myegg". After a bit of effort, we managed to trace the cause to this problem:

First, the python path in the [python26] part was incorect. Second, even if it pointed to the proper binary, the -devel packages for that python needed to be installed.

Well, now I know. Hopefully I'll remember it for the next time when I'll encounter the problem.

Probably you can, but you have never done it because you have a shiny interface for everything. I'm talking about this discovery of mine:

svn diff | kompare -

What it does is to take the output from svn diff and pipe it into Kompare, a merge/diff utility from the KDE Project. I can do this from the command line, straight from the directory that I'm in, and bang! I get a nice graphical overview, complete with the tree structure that I can navigate to see what I'm about to commit. Honestly, this little command gets me excited everytime I run it.

I'm a die hard in regards to ArchGenXML usage. The number of things to know about when creating new content types for Plone is just too high. Package structure, Zope package registration, content types registration, QuickInstaller registration, GenericSetup profiles, skins registration, workflows, etc. I can go in and do changes to the code, and add to it, but generating it from scratch is a gigantic task, especially for my use case, where I need to start a new project with about 7 content types.

Now to the problem: ArchGenXML assumes (and hardcodes in its templates) the Products.* prefix for your package. I don't have any problems with it, but my employer uses a different package structure so for uniformity I need to follow their standards. 15 minutes of poking and changing through agx enabled me to change it so it would generate GS xml files with the proper namespace (based on a new model level TGV named "namespace" that I have created). After assessing the difficulty of the task and being under time pressure, I've decided to go the dumb route, which I'm documenting below:

My generation script is something like this:

./archgenxml -c archgenxml.cfg myproduct.zuml 

#rename Products.myproduct to ns.myproduct
find myproduct/* -type f -print | xargs sed -i 's/Products\.myproduct/ns\.myproduct/g'

#in the xml type profiles, replace myproduct by ns.myproduct
find myproduct/* -type f -print | xargs sed -i 's/>myproduct</>ns\.myproduct</g'

#in the profiles.zcml, rename the profile to ns.myproduct
find myproduct/* -type f -print | xargs sed -i 's/title=\"myproduct\"/title=\"ns\.myproduct\"/g'

Notice the sed lines, which change the source code to point to "ns.myproduct" instead of "Products.myproduct". Now there's just two more problems, which are actually AGX bugs:

• the FilesystemDirectoryViews registered for the skin are incorect, so you'll need to insert this in ns/myproduct/__init__.py

from Products.CMFCore import utils
from Globals import package_home
from os.path import dirname
ppath = utils.ProductsPath
utils.ProductsPath.append(dirname(package_home(product_globals)))
DirectoryView.registerDirectory('skins', product_globals)
utils.ProductsPath = ppath

• trying to get AGX to output the code inside a two level deep folder doesn't work (it crashes), so you'll need to move this script file and the model just above the "myproduct" folder, inside the "ns" namespace.

That's about it. Hopefully I'll find the time to fix the two bugs and add the namespace improvement in the next week. Still need to do the review for Plone for Education, which I have received for free from the publisher.

I've had a buildout bootstrap process failure, this time a weird one, perhaps I should document the bug and report it.

The latest Ubuntu version which I have installed (Lucid Lynx) comes with a package called python-pkg-resources, which packages pkg_resources, which used to be available only through the setuptools distribution. Buildout's bootstrap.py tries to guess if Setuptools or Distribute are installed by checking the availability of pkg_resources; by guessing wrong it all comes to a crash at the end.

I'm not very interested in debugging these types of problems anymore. Distribution/packaging tools should just work. I want to focus on my work, not debug the toolchain. No more corner cases or whatver. So I'm gonna compile separate Pythons in the future, especially when dealing with older Zope/Plones.

I think this is the Python Index biggest mistake, the one which makes it unreliable for serious development environments: exposing package entries with no real package files and just a download URL. To see what I'm talking about, just examine the PyPI records for BeautifulSoup or IPython, packages that are very common in buildouts. As soon as the author and publisher of that package has a hosting problem, the developer that uses that package also has a problem. Buildouts will completely fail and this will cause dead times and frustration for the developers.

Yes, there are a couple of PyPi mirrors, but they only mirror files hosted by PyPi. The central PyPi site will probably have better performance and availability then what individual groups and developers can provide and it's always easier to mirror one single website than many, so there's no shame or loss of pride in using the PyPi to host your files. Please do so!

I've started an application that uses Dolmen, a lightweight CMS built on top of Grok, and I want to be able to "rollup" the megrok.resource files based on the devmode setting that was set in the zope.conf file. After a bit of digging, I came up with this code that tells me the current devmode setting for the running instance:

from zope.app.applicationcontrol.applicationcontrol import applicationController
from zope.app.applicationcontrol.interfaces import IRuntimeInfo

def get_debug_mode():
    """Returns the devmode setting for a running zope instance"""

    rti = IRuntimeInfo(applicationController)
    return rti.getDeveloperMode() == "On"

I'm internationalizing a Pinax website and I've encountered this piece of code in a template:

<input type="submit" value="{% trans "invite" %}"/>
{% blocktrans %}{{ other_user }} to be a friend.{% endblocktrans %}

The message ids for this code would be two separate blocks: "invite" and " %{other_user}s to be a friend". Both offer very little in terms of context and make the translators job difficult. Correct, in my point of view, would be the more convoluted form of:

{% blocktrans %}
<input type="submit" value="invite"/>
{{ other_user }} to be a friend.
{%  endblocktrans %}

This implies that the translators know enough HTML to notice that the value attribute needs to be translated, but the end result is a lot more flexible and provides real context to them.

TL;DR: don't split paragraphs into separate translation units. It's a NO-NO.

UPDATE: I have found what is probably the worst example of how to create a translatable template. Remember, don't assume the English language resembles anything like another language.

{% trans "edited by user" %} {{ obj.editor.username }} {% trans "at"%} {{ obj.modified|date:"H:i" }}

This should be done this way:

{% blocktrans with obj.editor.username as editor_username and obj.modified|date:"H:i" as obj_modified
edited by user {{ editor_username }} at {{ obj_modified }}
{% endblocktrans %}

I wonder what possible explanation there is for the behaviour of the date template filter.

Uses the same format as PHP's date() function (http://php.net/date) with some custom extensions.

I understand where Django comes from, but I think this sort of things should be more aligned with the rest of the Python world.

I fail to see how

{% block %} 
...
{% endblock %}

is in any way better or "less scary" then, let's say

<dj:block>
...
</dj:block>

Yet another rant, this time triggered by the error I got when writing this piece of code:

{% blocktrans with offer.offerer.username as offerer_username 
               and offer.offered_time|date as offerer_date %}
...
{% endblocktrans %}

One more thing to grudge about is that vim, even with djangohtml syntax type installed, is not very knowledgeble about how to format the template file (it treats the tags as regular piece of text). Probably this could be fixed, so I shouldn't complain about this too much.

I think Django templates are compiled to python code, so it's natural that they're treated in an imperative, dumb way, but that's not the only way of doing things. For example, Chameleon is another templating library that compiles its templates as python code, has no problem working with an XML based templating language frontends.

I have many grudges against the django templating language and its templates (in short, I hate them), so I'm gathering evidence to support what my "spider sense" tells me. Today the template tag system goes under fire.

Given the following template fragment:

{% load i18n %}
{% load avatar_tags %}
{% load voting_tags %}
{% load pagination_tags %}
{% load extra_voting_tags %}
{% load in_filter %}
{% load extra_tagging_tags %}
{% load sorting_tags %}

Which one is responsible for the following "anchor" tag?

{% anchor "hotness" "reddit-like hotness" %}

That's the equivalent of diving into a python module, with lots of "from X import *" at the top. Where do you find the definition of a symbol? At least, if it were Python, I could do a tag search in Vim, or a "go to definition" in Eclipse. If this practice is frowned upon in the rest of the Python world, why are so many programmers praising the Django templating system? Am I the only mad man here? My problems with this tag is that it doesn't translate the content, so I'll need to grep for its source and change it.

The template tags in Django are about extending the templating language, as to provide the programmer with new and specialized ways to interact with the template and its environment. The reason for this "tag inflation" is that the django templating language, for all its richness (by tags and filters numbers, I mean), is really limited. Python expressions are not allowed, and for every imaginable use case, there needs to be a tag, specialized or not.

How would Zope 3 solve, for example, a problem similar to the one the "anchor" tag handles? Well, rendering a special link for a content item could be as easy as

<a href="" tal:replace="someobject/@@hotness_link"><img src="hotness.gif" /></a>

Is this better? I think so. I'm editing HTML, and the <a /> tag is way better in expressing what the end result will be, compared to a simple {% anchor %} tag. Even more, the <img /> tag inside is purely cosmetic, just to cue the viewer of what the final result will be. The entire <a /> tag, with its content, will be replaced by whatever result is rendered calling the the someobject/@@hotness_link view. Finding the source of the hotness_link view is easily introspectable TTW using a debug tool such as lovely.skinbrowser.

The ZPT templates from Zope 3 can also give you a mechanism where you can add new expression types, but there's just one or two packages in the wild that define new expression types. Now compare this to the regular Django projects, were defining new tags is something that almost all projects do.

In conclusion, even though Django templates are much more imperative then ZPT, which are very declarative, they don't achieve the power and simplicity that they strive for.

Yet another angry rant, caused, of course, by using Django in anger. Nothing wrong with using something in anger, that's the real way I learn something. Zope 3 even has an online book on how to use it in anger.

That said, Django's makemessages administrative command sucks by being way too inflexible to anything but the ideal Django development environment. My environment looks like this: I have a project based on Pinax, which I'm developing and deploying using zc.buildout. My source code sits in src, where I have several packages. I also have a "localsettings.py" module located in the root of the buildout, because I don't want to have it inside the src folder. Pinax is located in parts/Pinax, and it's actually a git checkout, based on my own fork of Pinax. Pinax doesn't have translations at the moment (I think I saw a ticket in its tracker about reintroducing a translation package), so I'm on my own here with regards to translation.

With this setup, it is close to impossible for me to generate anything useful without a lot of hacking and swearing. Makemessages insists on being run from inside a Django project, and when I did that, it complains about missing localsettings module. Pointing the root of the buildout as pythonpath didn't do anything. A good thing that I have already extracted messages from the templates, before switching to the buildout project structure.

Some solutions that I have found:

• I can extract messages from the Pinax python modules using this homegrown script:
  

PYFILES=/tmp/pyfiles
PINAX=parts/Pinax/pinax/ 
BASE=/home/tibi/work/ProjectBuildout/src/project/locale/ro/LC_MESSAGES/
POTFILE_PYTHON=$BASE/python.pot
POTFILE_TEMPLATES=$BASE/templates.pot
POTFILE=$BASE/django.pot
POFILE=$BASE/django.po

#extract messages from python code
find $PINAX | grep ".*py$" > /tmp/pyfiles
touch $POTFILE_PYTHON
xgettext -j -L python -d django -f $PYFILES -o $POTFILE_PYTHON

#merge the templates + python messages into one pot file
msgcat -o $POTFILE $POTFILE_TEMPLATES $POTFILE_PYTHON

#merge the potfile with the po file
msgmerge -U -N $POFILE $POTFILE

• I have copied all the templates from pinax and its associated applications inside a template folder in my project. Now I can generate the po file, from my src/project folder, with

../../bin/py ./../../manage.py makemessages -e .py -e .html -l ro

Of course, I can't run this over the other apps and packages in my src/ folder to extract messages from the python modules, so I am forced to adjust the first script to take those folders into consideration.

I managed to figure out how to beat one awkward piece of workflow when starting new project: I always start hacking on a project, then I notice that I haven't been working on a svn checkout (it's the chicken or the egg problem). So I need to do a svn import, delete my copy of the project, checkout the svn version and start hacking at my project again. This workaround is agravated, though, when working with buildouts. I can't svn import the entire folder, because lots of generated folders and files will end up in subversion, which I don't want. Cleaning my original, running import and regenerate the buildout is a workflow killer.

The solution is simple, and goes like this: instead of importing the entire buildout folder, I can run

svn import . http://my/subversion/path/ --depth empty -m "initial import"

or I can just create a folder in my repository with svn mkdir

Next, checkout the empty folder from the repository into my buildout

svn co http://my/subversion/path/ .

Now I can cherry-pick whatever I want committed, with svn add. Problem solved!

A bug creeped in one of my Django 1.1 projects that is in beta-testing right now: the Pinax wiki app looks for a REMOTE_ADDR value in request.META, which was not set in my environment. My environment is a pretty standard (as far as this setup goes) nginx + fcgi (flup on the django side) + django. Further work on this issue revealed, step by step, that:

• REMOTE_ADDR needs to be somehow set by a Django middleware, based on an http header,
  
• so I've added django.middleware.http.SetRemoteAddrFromForwardedFor to the list of loaded middleware
• but that middleware is deprecated in Django 1.1 and does nothing, so I rewrote this middleware based on Chapter 15 of Django Book
• this new middleware did its job, but for some reason flup stripped that header from the request and the proper values never got to Django
• this made me look for a replacement for flup, which I found in django-cpserver.

This package adds a new admin command, 'runcpserver', which replaces the default development server by something more appropriate to production. The word on the 'net-streets' is that it runs well and does its job, so I'll be using it for the time being. The recommended solution is to run apache+mod_wsgi and proxy that to nginx, but right now I don't want the extra administrative overhead that Apache represents. If I can't get enough "juice" out of one instance of cpserver, I'll just add an extra instance and balance them with nginx or haproxy.

Probably this is documented somewhere in the Django docs, but, I mean, who has time to read the docs? :-) In the urls.py file for my project I didn't do a

from django.conf.urls.defaults import *

because that's just bad style. Instead I've just imported what I needed: patterns, include and url. Later on I got this error in my email:

Traceback (most recent call last):

  File "/home/zope/djangoprojects/lib/python2.5/site-packages/Django-1.1.1-py2.5.egg/django/core/handlers/base.py", line 118, in get_response
    callback, param_dict = resolver.resolve404()

  File "/home/zope/djangoprojects/lib/python2.5/site-packages/Django-1.1.1-py2.5.egg/django/core/urlresolvers.py", line 263, in resolve404
    return self._resolve_special('404')

  File "/home/zope/djangoprojects/lib/python2.5/site-packages/Django-1.1.1-py2.5.egg/django/core/urlresolvers.py", line 255, in _resolve_special
    callback = getattr(self.urlconf_module, 'handler%s' % view_type)

AttributeError: 'module' object has no attribute 'handler404'

Just looking at the error I could determine the cause: my urlconf module (urls.py) didn't have a handler404 view defined, so I just gave up and replaced my specific imports with the asterisk import.

On a side note, what's up with Django logging? By default, the only way to log errors in production environments is by email. Really!? If I would want emails, I'd setup the syslog to email me those entries. I already have the logwatch emailing me stuff that happens on my server. One step above is django-db-log, which I haven't tried yet, but which seems to be logging errors to the db. Word is that real logging, using the logging python module, will be introduced in Django 1.2.