I blog therefore I am (by Carlos de la Guardia)

7 Jun, 2008

Django has quickly become one of the leading web development platforms in the Python world. Still, having used Zope and Plone for years and accustomed to their power, I was unwilling to dedicate much time to it, other than reading a few tutorials and creating a simple application just to taste the waters.

This changed a bit when Google's App Engine came around, since its use of Django meant that I better learn a little more if I wanted to try it some time (I missed out on the first 10,000 invites).

That's why I eagerly accepted the offer from Packt Publishing to send me a review copy of their book Learning Website Development with Django, by Ayman Hourieh. A great opportunity to learn more about Django at the perfect time.

The book is aimed at web developers who are not looking to become experts in the framework, but just want to build a "2.0" web application quickly and painlessly. It is expected that they know some Python, but no great demands are placed on other previous knowledge.

Like most titles from Packt, the book wastes little time on introductions and quickly dives into the subject matter. The reader is put to work from the very beginning, and by page 20 Django is installed and the first project created and launched. I like this practical approach. All the code is available at the book's web site, so it's also easy to follow along.

The book is structured around the development of a social bookmarking application ("Social" and "2.0" seem to be mandatory buzzwords around Django, Rails and some other "new wave" frameworks). Without going into too much detail, each chapter introduces the reader to some new Django concepts needed to build specific features into the application. For example, there are chapters focusing on user management, searching, Ajax and administration interfaces.

By the end of the book, the reader has a working application for storing and sharing bookmarks, complete with tags, tag clouds, RSS, Ajax bells and whistles, voting, friend networks and an administrative interface. She also should end up with a good knowledge of what Django can do and how to use its features. A couple of final chapters on deployment and future directions complete the tour and give the reader some insight on what to do next with her newly gained Django proficiency.

Overall, I think the author explains the concepts well enough and achieves his objective of teaching beginners how to build an application with Django. However, if you have a good knowledge of web development and Python, you may find that you need a deeper source of knowledge about Django as a framework (which by the way is readily available at the free online Django book).

At just over 200 pages and at a price tag of US $40, I find the book somewhat expensive, but it covers the ground it promises, so if you just want to quickly get going with your "web 2.0" application, the book could be worth the price.

22 Apr, 2008

If you need some Plone or Zope 3 training and by any chance were considering Jazkarta's offerings next week (links below), I can get you a small discount and in the process help Alex Clark move plone.org to Plone 3.

I want to contribute to Alex's travel fund for the Paris sprint, but I have no money. However, Jazkarta will give me $50 for every person that signs up for the course using one of the links below. The person who subscribes to the course will also get a $50 discount.

I will donate any money I get from this to Alex Clark's fund. Here are the links:

Plone developer training (April 28-29)

Zope 3 developer training (April 30-May 1)

21 Apr, 2008

If you've been thinking about giving Grok a try for your cool new web application, but felt that the tutorial didn't cover enough ground to really get you started, take a look at the new primer on Grok by the tireless Martijn Faassen (note that this link points directly to a text version on the svn repository, web version should be coming soon).

In this document, aimed at developers, he explains in a very concise fashion all the parts that make up Grok and how they relate to each other. While doing this, he also explains the most important concepts of the Zope 3 Component Architecture (ZCA), which is something that the Zope 3 world wants the Python community to get their hands into (come on, give it a chance, it's only Python).

So now Grok can claim to make Zope 3 easy to use and the ZCA easier to understand, not bad for a band of cavemen.

26 Mar, 2008

Supervisor is a Unix process monitoring system that can automatically start and monitor any number of services for you and restart them whenever they stop unexpectedly. It is written in Python, and is very extensible. It includes a command line client program and also a web interface for easy remote process management.

Supervisor can be extended directly with Python or using any other language via XML-RPC. Its event notification protocol makes programming new event listeners a very simple task.

I used Supervisor recently to set up a Plone installation and I heartily recommend it to anyone who wants to monitor server processes like those used by a Plone site.

My site was using two Zope instances with ZEO, the Pound load balancer and the Varnish cache.
Supervisor allows me to handle all of these via its remote web interface, from where I can start, stop or even check the log files of any process. Supervisor lets you assign a priority to every program that it controls, so that it is possible to start or stop all services with one command in a specific order. Here is a screen capture of the web interface:

Supervisor can be installed with easy_install, which makes it very convenient to use with any Python installation. Integrating it into a Plone site, where zc.buildout is the favorite deployment mechanism, is also very easy. Here's what I added to my buildout.cfg file to use it:

parts = supervisor

[supervisor]
recipe = repoze.recipe.egg
eggs = supervisor

In addition to that, supervisor needs a configuration file. I added an etc directory to my buildout with the following supervisord.conf file in it:

[inet_http_server]
port=127.0.0.1:9999

username=admin
password=admin

[supervisord]
logfile=%(here)s/../var/log/supervisord.log
logfile_maxbytes=50MB
logfile_backups=10 
loglevel=info
pidfile=%(here)s/../var/supervisord.pid
nodaemon=false

[rpcinterface:supervisor]
supervisor.rpcinterface_factory = supervisor.rpcinterface:make_main_rpcinterface

[supervisorctl]
serverurl=http://127.0.0.1:9999

[program:zeo]
command = %(here)s/../parts/zeoserver/bin/runzeo
priority = 10

[program:zope_instance1]
command = %(here)s/../parts/instance1/bin/runzope
priority = 20
redirect_stderr = true

[program:zope_instance2]
command = %(here)s/../parts/instance2/bin/runzope
priority = 20
redirect_stderr = true

[program:pound]
command = bin/pidproxy %(here)s/../parts/pound/var/pound.pid %(here)s/../bin/pound
priority = 40
redirect_stderr = true

[program:varnish]
command = %(here)s/../bin/varnish
priority = 60
redirect_stderr = true

Note that supervisor starts its subprocesses via fork/exec and thus they must not dameonize. That's why on the example config Zope and ZEO are started using runzope and runzeo rather than zopectl and zeoctl.

For programs which launch another process separately and control that via PID files, like MySQL or Pound in the example above, the pidproxy script can take care of this. Just give it the launch script and the PID file locations in the config and everything will work OK.

Supervisor is a great solution for process monitoring and this post just scratches the surface of what is possible with it. I hope I can post later about using its extension mechanism. In the meantime, I recommend you try it out or at least read the manual and see what it can do.

21 Mar, 2008

Packt Publishing was generous enough to send me for review a copy of their book, Hacking Vim. I was really looking forward to this, but then some family matters prevented me from finishing the task within a reasonable amount of time. This is my late, late, late review.

Hacking Vim is a cookbook for becoming more productive using Vim and GVim. At 200 pages, it is not a large book, but it is packed with recipes and tips. The recipes range from simple stuff, like changing the color scheme, to advanced tasks, like creating and distributing Vim scripts. Some of the recipes apply only to GVim.

The book begins with an introduction, where the author quickly goes through the genealogy of Vim and explains how it differs from similar editors, like Elvis or Vi itself. This is short and to the point. It can easily be skipped if the reader wants to go straight to the recipes.

The recipes are organized into four categories, each devoted a separate chapter: personalization, navigation, production boosters and advanced formatting. Each chapter contains around ten to fifteen recipes. In addition, there are tips scattered all through the book. In general, I liked the format and the table of contents makes it easy to find a given recipe.

Personal favorites in the personalization chapter were spell checking, using abbreviations and modifying key bindings. In the navigation chapter I liked the explanations for search and markers. My favorite chapter was the one about production boosters, specially templates, tag lists, macros and opening files everywhere. On the formatting chapter I looked more into code formatting than text, since I use Vim mostly for development.

The final chapter is a tutorial for creating Vim scripts, covering everything from development to distribution. Instructions are included for creating the scripts using other programming languages, like Python and Ruby.

The appendices are more useful than in the average book. They include a survey of unusual Vim plugins, like games and mail clients. A sort of bonus recipe about storing the Vim configuration file online is also included.

Though I use Vim a lot, I'm not what you could call a power user. I don't think I'll ever write a Vim script, for example. Overall, however, I got some benefit from the book, even though serious practice is needed to really take advantage of some recipes.

That said, things like templates and tag lists can make the light Vim user much more productive without too much effort. Also, there are references to many Vim plugins that further simplify the work needed to get some stuff done. My Plone and Python development immediately became easier because of all this.

In conclusion, if you use Vim and are not an expert, I think this is a valuable book to own, though the US $40 price tag seems a little steep to me (it is also available as an e-book for half of that).

29 Feb, 2008

I recently gave a small introductory talk about Plone at the CONSOL (National Free Software Congress) here in México. The aim was to introduce new developers to Plone.

There were about a couple dozen students present, most of them quite familiar with Drupal or Joomla. They barely had heard about Plone or Python. No one had actually installed Plone at all.

The talk went reasonably well, but I was disappointed in that no one asked a question at the end. A couple of days later, however, one of the students contacted me by email to ask advice on how to install Plone.

He was specially interested in finding out how to upload what he did on his machine to a cheap hosting service shared by him and some friends. You see, with the poor economic situation for many students of state universities in México, the pooling of resources with other friends to have a web development testbed shows a lot of initiative. Even as I was writing to him that he just can't use Plone on that kind of hosting environment, I knew that would be the last of his experiences with Plone.

Of course, customers will always be able to use a real hosting solution for their projects, but new developers sometimes won't. In some countries, like México, this can have an impact in the adoption of Plone, since customers will be scared of going with it and then not finding any developers to maintain and extend their sites.

Now, I don't think that the solution is to try to make Plone run in cheap hosting environments. The effort is just not worth it. What could be done, a thing which would help Plone adoption at all levels, is to pursuit Martin Aspeli's vision of the Big Green Button that turns a Plone site into a static (or even potentially dynamic) version of itself that can be hosted with one of those $5 a month providers.

I have been looking at Entransit, from Enfold Systems, and I think it can do most of what Martin describes on his blog post. However, Entransit's out of the box experience is, according to Alan Runyan, very limited and just a small glimpse of what is possible with it.

Entransit also has little documentation and very few mentions on plone.org, but that could change fast, since it has been tagged as a possibility for delivering Plone content on the 4.0 release notes.

I am currently testing Entransit for a client's project. I'll let you know how it goes.

28 Feb, 2008

You often hear sports figures talking about giving a 110 per cent effort for some important game, but Plone walks the talk. Witness the following screen capture, which is proof that Plone coaxed at least one machine into giving this seemingly impossible effort.

Who says that Plone can't perform?

31 Jan, 2008

I have written a couple of tutorials for using Repoze under mod_wsgi:

• Install Plone 3 behind Apache and mod_wsgi using Repoze
• Installing and setting up Grok under mod_wsgi using Repoze

Give Repoze a try. WSGI will be an important part of the future of Zope and Repoze can get you there now.

28 Jan, 2008

Malthe Borch has a provocative take on what he calls a revolutionary approach to future Plone core development. Basically, create a parallel branch of Plone that uses the same recently componentized packages in the Plone core, but is otherwise free of any backwards compatibility requirements.

This idea is always attractive to developers. Why deal with all the cruft accumulated in years of evolution when you can start with an almost blank slate? A complete rewrite can lead to a cleaner, more modern implementation and in a shorter time too!

The problem is that, first, this is not entirely true. Second, you may lose many users while you do the grand rewrite and even after you are done (assuming some fairly flexible definition of done, because getting that rewrite to the dreamed-of perfect state envisioned at the beginning is really easier said than done).

We've had a very close complete rewrite experience in the Zope community in recent years. Zope 3 aimed to be the replacement for Zope 2, and some very good developers underwent the task of taking all the lessons learned from years of Zope development and distilling those into a new version of Zope that also took into account modern development paradigms.

The result? A new, great, powerful, successful web development library...which almost nobody knows or cares about outside the Zope community. Those who know that it exists are often very confused about its relationship to the old Zope ("please sir, could you tell me how to install Plone on my new Zope 3 instance?"). Heck, I have been following this story for years and still don't know whether to call Zope 3 a library or a framework.

Keeping the Zope name also backfired in unexpected ways, as the project inherited the perception of unwieldy and unpythonic (whatever that means) that it's namesake had acquired in the Python community by this time, thus limiting the number of new adopters.

I think the Zope 3 story shows us that it takes more than a name to maintain continuity between two otherwise disconnected pieces of software. You need an active and ongoing effort to make this connection apparent to outsiders or you will get a lot of confusion. Like Paul Everitt said in the comments to Malthe's post, until the Five project came along, there was no apparent bridge between the projects, thus the Zope 3 name turned out to be misleading.

The truth is, there are many uncertainties in a developers work to even think about working with a framework with an identity crisis. That's what you get with complete rewrites of frameworks with large user bases, because inevitably, a large portion of those users will stick with the old code. A good level of backward compatibility is necessary to maintain the user base. You just can't tell a user that all or most of his accumulated knowledge is no longer useful, even if it's for the better (in many cases, this means better for the developer, not the user).

To be succesful, a revolution needs to get the support of the masses. Thus, its ideals need at least to be based on the people's aspirations and needs. That's why I agree completely with Martin Aspeli's comment about the driving force needing to be at the users, integrators and businesses end. To me, an evolutionary path sprinkled with revolutionary ideas would be wiser than a complete break and rebuild strategy.

One last comment about starting from scratch: even a complete clean slate restart is no piece of cake to get right. The Django web application framework was started from the ground up by another group of talented Python developers and was in controlled use for some time in a production environment before its authors decided to open it up for the world. Not very long after going public, a drastic refactoring of some parts was deemed necessary (the so-called magic removal branch), to "remove warts that Django had accumulated over the years".

In Django's case, I think that refactoring was done early and worked fine, but the point is it was a fairly well controlled project and they still needed to remove warts. I doubt they could get away with something like that at this time. They have too broad an user base for that at this point. Books have been published, tutorials written, sites launched...there's no going back now.

I think Malthe would do a great good to the Plone community if he pursues his ideas and contrasts his approach with Plone frequently, so that some of his work can be integrated with the Plone core at some time. Vudo is a good name, I'd say. For better or worse, let Plone be Plone and continue its evolutionary path.

24 Jan, 2008

A recent post on Coderspiel mentions that Lucene does not use Lucene for indexing their own web site and goes on to criticize projects that don't use their own software on their sites. He asks:

"How is it that some fancy-pants framework is always the right tool for an abstract job and PHP is the right tool for a real job?"

He is mostly referring to Java projects under the Apache umbrella, but I think the question is interesting for the Python world too. Does a web application or project "lose points" if it doesn't use itself on its own site?

The Grok web framework has just launched a new web site, and there was some discussion on their mailing list about using Grok for this task, the dogfood approach, or "the right tool for the job". Ultimately, they settled on Plone, because they need CMS capabilities and Plone is, well, a CMS. Check out their site, I think it went well.

Speaking of Plone, they use Plone itself for their site too, though not the latest version. A prospective user can join the site easily and learn what Plone is all about while using it. In this case, the dogfood approach just works.

Zope, on the other hand, uses itself too (more precisely, a Frankenstein old Plone setup), but the results are far from appealing. For Zope, the dogfood approach seems counterproductive, though I would say that's not because of Zope itself.

Django's main site seems to use "the right tool for the job" and not itself for their main site, but and their showcase Django projects site both use Django. I should say that Django project related sites that I have seen are generally appealing regardless of the approach taken.

Pylons uses Confluence, a commercial wiki, and Turbogears, also sporting a nice looking site, uses moinmoin, at least for their documentation site (not sure what they use for the main site, but there's a chance it could be Turbogears itself).

Of course, how the site really looks is not dependent on the dogfood or right tool approaches, but the Zope case seems a reminder that if you are going to eat your own dogfood, you may want to spice it a little at least.

Overall, I cannot say I see a clear advantage for the dogfood approach. I think it has some value to show what you can do, but it all goes for nought if presented in the wrong way. Ultimately, the mission of a web site is to present the project, which can be accomplished fairly well without using it.

In the case of very specific technologies like Lucene, eating the dogfood may be more important, but as far as I can tell, it hasn't stopped or even slowed down adoption of the technology. Having access to a demo site can sometimes be useful for a developer, but easy to install packages and good documentation are really far more important.

Update: Simon Willison points out that Django's main site does indeed use Django. He and other commenters also rightly mention that many developer sites use a combination of tools, like issue trackers, wikis and svn frontends and it makes no sense to rewrite all of those just to eat your own dogfood. Reinout van Rees gently reminds me that I use the right tool approach for my own blog (I've had clients make fun of me because of this, actually).

22 Jan, 2008

The TIOBE programming community index has declared Python as the language of the year 2007, based on it's growth during the period as compared to other languages.

Without all the hype of last years wunderkind, Ruby, Python advanced from 8th to 6th place, overtaking Perl for the first time ever. Ruby lost one place after Delphi (!) climbed from 12th to 9th place and pushed last years fastest growing language into the 11th position, behind Javascript.

Python had a +2.04 percent growth for the year to get to 5.538 percent overall usage (as you can see, 'fastest growing' doesn't mean 'most used' by a long shot, so don't lose perspective here). Ruby's growth for the year? -0.17 percent. Its growth over the year of the big Ruby on Rails push? +2.15 percent. That's right, without all the noise, Python grew almost the same in 2007 as Ruby did in 2006.

There are several reason why, I think. First, keep in mind the TIOBE index ratings attempt to give an indication of programming language popularity and are based, according to the TIOBE site, on the world-wide availability of skilled engineers, courses and third party vendors. Search engines play a big part in the rating process, as can be seen in the definition at their site.

Now, what caused Python's popularity surge? I'd venture the following guesses:

• The popularity of strong agile web development frameworks, like Django, Pylons and Turbogears. Many a Python fan that had been lured away by RoR's seductive appeal came back to the language because of them.
• The growing use of Python as a system administrator's tool over Perl. Python, like Perl, is present in almost every system and has a nice standard library in place. Unlike Perl, Python is just easy.
• The constant if sometimes unaknowledged push of Zope and it's related technologies. Plone is one of the most heavily used Python based applications out there, and more than ten years later, Zope still continues to innovate and influence the Python development world.

Anyway, it's a great time to be a Python developer. No matter what part of the growth curve you and your favorite Python projects represent, please keep at it.

22 Jan, 2008

Doug Hellmann wonders what tools Python developers find more necessary to their work these days. In my case I would say vim, which got a lot of votes on Doug's comments page. Also, I would probably be out of business if Google didn't exist.

Of the tools mentioned on his site, I've used Wing IDE sometimes and I think it's cool, but somehow I always end up going back to vim. I might try out ipython one of these days, to find out the reason why it got a so many votes.

On the Zope/Plone front, I'd say zc.buildout is becoming one of the more important tools for development and deployment. I don't think I use any Zope or Plone specific tools besides that.

I am a Linux guy, but I guess we are becoming a minority on the Zope/Plone community with OSX growing in popularity (incidentally, call it envy, but I think the Mac experience has to be out of this world for Python developers to accept, and pay big bucks for, an OS with things like broken readline support for the Python interpreter). What tools do OSX developers find invaluable for Python work? Is there anything out there that makes Zope/Plone development easier on that OS?

21 Jan, 2008

I think Repoze is a potentially great thing for the Zope world, so I've been experimenting with getting different Repoze projects running under mod_wsgi and apache, which I think may become the standard deployment solution for production Python WSGI apps.

I recently wrote a step by step guide to setting up Zope and Plone in this manner. Now it's Grok's turn.

Grok is nearing a 1.0 release and is a web development framework which makes the following claims:

1. Easy to use.
2. For Python developers.
3. Powerful.
4. Based on Zope 3 technologies.

For those of you that will immediately jump and say that claim number four is incompatible with one and two above, please at least visit Grok's web site and give it a try. We are really talking about a Python web development framework here.

Getting Grok to run under mod_wsgi is very easy, but as of this writing you will need to check out repoze.grok from the subversion trunk, since it contains fixes to make the WSGI setup work using the included sample files instead of rolling your own.

Once you have the correct version, setting it up is very straightforward. Install mod_wsgi under Apache and make sure you have a Python 2.4 installation ready with setuptools already installed. Check my post on Plone behind mod_wsgi if you need more detailed instructions for this.

Next, install and run repoze.project:

$ easy_install-2.4 -f http://dist.repoze.org repoze.project

Repoze uses virtualenv to create sandboxes for the applications it runs, so that each one has an independent Python environment of its own and avoids egg conflicts with other applications. To create the Grok sandbox:

$ repozeproject repoze.grok --path=/home/cguardia/repoze/grok

Now all you need to do is copy the contents of the apache2.conf file from the etc directory in your sandbox into your Apache configuration (with the appropiate editsfor your site, of course) and rename the sample-users.zcml file in the same directory to users.zcml (or create you own) and Grok is ready to run.

Since this configuration is intended for production use, you will also need to setup a ZEO server and configure the zope process to connect to it. Just run 'bin/mkzeoinstance .' from your sandbox directory and add the required configuration to the zope.conf file. Here is an example:

<zodb_db main>
    cache-size 5000
    <zeoclient>
        server localhost:8100
        storage 1
        cache-size 20MB
        name zeostorage
        var $INSTANCE/var
    </zeoclient>
    mount-point /
</zodb_db>

That's all there is to it, you can now run Grok behind mod_wsgi by restarting the ZEO and Apache processes:

$ bin/zeoctl start

$ sudo apache2ctl start

You Grok application is ready to run under mod_wsgi, and perhaps be combined with other Python WSGI applications on the same site.

6 Jan, 2008

To more and more Python web developers, WSGI, holds the key to the Python web development future. Since there are a number of important web development frameworks and the power of Python makes it really easy to create new ones quickly, interacting with best of breed applications developed in multiple frameworks could soon be the best way to create a new Python web site.

Until relatively recently, Zope and some of its most successful applications, like Plone, ran the risk of missing the WSGI party, but not anymore, now that Repoze is here.

Repoze is a bridge between Zope and WSGI, which has the objectives of both helping Zope developers publish applications using WSGI and, equally important, letting non-Zope web developers use parts of Zope independently.

I recently installed Repoze under WSGI on a test server of mine, and I intend to write about my experiences with it on this blog. This is hopefully the first post of many.

I started with a clean install of Ubuntu Linux 7.10 on a new virtual server. The first step is to install the necessary packages for the correct Python version (Zope currently requires Python 2.4) and also for the Apache server.

Before that, It was necessary to install the required packages for being able to compile and build software using Ubuntu. Be aware that both package installation and Apache module additions usually require root access.

$ apt-get install build-essential

Next, the packages for Python and Apache. Like most packaged Linux distributions, Ubuntu requires a separate install for the development libraries of each piece of software:

$ apt-get install python2.4
$ apt-get install python2.4-dev
$ apt-get install apache2
$ apt-get install apache2-dev

Repoze uses Python's setup tools, so that package is neede as well:

$ apt-get install python-setuptools

Now, the server was ready for mod_wsgi. Since there is no package for this in Ubuntu 7.10, it was necessary to get it directly from the download site:

$ wget http://modwsgi.googlecode.com/files/mod_wsgi-2.0c4.tar.gz

$ tar xzf mod_wsgi-2.0c4.tar.gz
$ cd mod_wsgi-2.0c4
$ ./configure --with-python=/usr/bin/python2.4
$ make
$ make install

Note that it is necessary to compile mod_wsgi using the same Python you will use to run your web site. Since Zope requires 2.4, the --with-python option was used to point to the newly installed Python.

Once mod_wsgi is intalled, the apache server needs to be told about it. On Apache 2, this is done by adding the load declaration and any configuration directives to the sites-available directory.

The load declaration was put on a file named wsgi.load, which contains only this:

LoadModule wsgi_module /usr/lib/apache2/modules/mod_wsgi.so

The configuration directives reside in the file named wsgi.conf, they contain an almost identical configuration to the one shown on the repoze.org site, on the deployment page.

WSGIPythonHome ${sandbox}
WSGIDaemonProcess tmp threads=1 processes=4 maximum-requests=10000 python-path=${sandbox}/lib/python2.4/site-packages
<VirtualHost *:80>
  ServerName my.machine.local
  WSGIScriptAlias /site ${sandbox}/bin/zope2.wsgi
  WSGIProcessGroup tmp
  WSGIPassAuthorization On
  SetEnv HTTP_X_VHM_HOST http://my.machine.local/site
  SetEnv PASTE_CONFIG ${sandbox}/etc/zope2.ini
</VirtualHost>

To really activate this configuration files, another step was required, which is to create soft links for them under the mods-enabled directory of the Apache configuration:

$ cd mods-enabled
$ ln -s ../mods-available/wsgi.load
$ ln -s ../mods-available/wsgi.conf

After this, the server was finally ready for Repoze. The first step was to install repozeproject, which is used to initialize a new Repoze sandbox. As mentioned before, this was done using Python's setup tools:

$ easy_install-2.4 -f http://dist.repoze.org repoze.project

Once repozeproject is ready, it is possible to create any number of repoze applications, which reside in an independent sandbox. This sandbox is really a virtual Python installation based on the same Python used to run the repozeproject command.

For this example, I used repoze.plone, which depends on repoze.zope2 and installs everything that is needed to run Plone (version 3.0.1 as of this writing):

$ repozeproject repoze.plone --path=/home/cguardia/repoze/plone

This command was run using my regular user account and created a plone directory which is actually the Repoze sandbox. The repoze.plone egg that installs this also installs a few sample configuration files, which may be used almost "as is" to run the new site.

To test the installation, first it was needed to add a new user to the Zope site. The command is run from inside the sandbox directory:

$ bin/addzope2user cguardia password

Now the site could be tested using paster:

$ bin/paster serve etc/zope2.ini

Once this was run I had access to the newly installed Zope site at port 8080. I took advantage of this test to add a Plone Site inside, so everything was ready for the WSGI configuration.

The only other requirement for running under mod_wsgi is to install a ZEO server and modify the sandbox's etc/zope.conf file to use a client storage pointing to it. This is very easy, just run (inside the sandbox):

$ bin/mkzeoinst .

This creates runzeo and zeoctl scripts under the bin directory. Since mkzeoinst doesn't delete existing files, I had to make sure that they didn't exist before running it.

Just to save you the trouble of looking it up, here is the section I added to the zope.conf file:

<zodb_db main>
    cache-size 5000
    <zeoclient>
        server localhost:8100
        storage 1
        cache-size 20MB
        name zeostorage
        var $INSTANCE/var
    </zeoclient>
    mount-point /
</zodb_db>

That was it. The site was now ready to work behind Apache and mod_wsgi.

When I tried this I had a lot of problems, because the sample zope2.wsgi script, which is used to run the Zope application process, didn't work at all. After some email and IRC exchanges with ever helpful and friendly Chris McDonough, I ended up fixing it myself and in the process became a commiter to the project (yay!). So, just make sure to run the newly released 0.2.9 version of the repoze.zope2 egg and everything will work ok.

With that out of the way, I just needed to start the ZEO server and, finally, Apache:

$ bin/zeoctl start
$ sudo apache2ctl start

One important thing, since mod_wsgi's daemon mode uses multiple processes, each one needs to execute the whole Zope startup process the first time it is called, so it won't hurt to click a bit through the site when Apache is initially started (or restarted), so that real users don't get these long requests. I hear that mod_wsgi will soon have some kind of preload functionality to deal with this situation better.

There it is. Zope can now run directly behind the Apache server, without need for its own ZServer. As you can see, using Repoze, Zope can now coexist peacefully with other Python frameworks easily using WSGI, but more importantly, it can interact with them. I'll write about some of that magic in another post.

1 Jan, 2008

While working for a client, I had the need to transfer the Data.fs file from the customer's server to my development machine, but I was not authorized to pack the database.

Due to other commitments I was working at night, so I would have to miss a whole day of work on this project if I waited for someone else to pack the db the next day. The file's size was 14GB so transfering it as it was really was out of the question.

I had a regular user ssh acount, so there were a couple of ways available for me to pack a copy of the ZODB, but the Python interpreter seemed to me to be the easiest path to success:

>>> import time
>>> import ZODB.FileStorage
>>> import ZODB.serialize
>>> storage=ZODB.FileStorage.FileStorage('/home/cguardia/Data.fs.copy')
>>> storage.pack(time.time(),ZODB.serialize.referencesf)

Done! The file shrunk from 14GB to 360MB and the transfer was done in time for me to get some work done that night.

In case anyone wonders about this, the referencesf parameter is part of the storage interface and is required for the pack method, even though apparently FileStorage doesn't use it in any way. If you are curious, referencesf is a function that returns the ids of objets inside a pickle. If you want to know still more, then you've got me. This is as far as I'm willing to go.