Turbo Plone theming with XDV/Diazo

Remember 1980's so-bad-it's-good Flash Gordon movie? In the buildup to the final reel, the damsel in distress implores our hero:

"Flash, Flash, I love you, but we only have fourteen hours to save the Earth!"

Fourteen hours?! To the eight-year-old me - accustomed to James Bond's mere minutes for bomb diffusal (done with 007 seconds to spare of course) - that was aeons of time in which to rescue mankind. Did anything take that long?

Of course the adult me - website-building me - knows better. Effort for theming alone can be measured in days, even weeks. But here's a thing: what if you could take an existing theme and slide it neatly into place in front of an existing, powerful content management system? Two big blocks of effort already done - all you had to do was tie them together. XDV makes this idea a reality for Plone. You can get results so quickly with XDV that when I set up a Plone 4 demo site for a potential client recently, I was confident that they could have a version of their current site's theme after a single day's development effort. Yep, a deadline one third tighter than Flash Gordon's!

Want to know how? What follows is a step-by-step tutorial explaining what I did. My aim is to give enough detail that someone new to Plone (an inquisitive web designer maybe?) could follow along. At the end I'll give some pointers on growing your demo site into a fully themed, production Plone site.

The XDV approach

With XDV (or Diazo as it will soon be known) you build up your theme by defining how small chunks of markup fit into the overall HTML structure of your page. It's simple to understand but very flexible and powerful. XDV works independent of Plone (see also XDV's older cousin Deliverance) but is easy to get running on a Plone site thanks to the collective.xdv package (soon to undergo its own name change to plone.app.theming).

What we need for the demo site are:

• Plone 4 with collective.xdv installed,
• a theme file - a page from the client's current site in a local index.html,
• a rules file that defines how XDV should map Plone content into the theme file,
• a CSS file with Plone-specific styles; all other styles come from the client's current site,
• all of those files wrapped up with a few config tweaks into a client.demo package.

Getting started

I can't show you the site I built, so first we need a new client. Hmmm, how about the British Prime Minister's Office? The content is a broad and deep tree of standard pages - the stuff an out-of-the-box Plone excels at. Looks like it's currently a Wordpress site.

Alright, let's build that.

To install Plone on your machine, follow the standard instructions to build a standalone, non-root instance of Plone 4 at $HOME/Plone. If you're on Linux it's as easy as:

$ cd
$ wget http://launchpad.net/plone/4.0/4.0.4/+download/Plone-4.0.4-UnifiedInstaller.tgz
$ tar xzvf Plone-4.0.4-UnifiedInstaller.tgz
$ cd Plone-4.0.4-UnifiedInstaller
$  ./install.sh --password=secret standalone

You can delete the Unified Installer when you're done. Also, Plone does not install anything outside the $HOME/Plone directory, so if you want to remove your site at a later date just delete that directory.

I want to concentrate on XDV, so for these next few steps I'll give the instructions without much background info. You'll find plenty more detail in Plone.org's documentation section.

We will store our theme and config files in a package called number10.demo. Let's create a package template with ZopeSkel:

$ cd $HOME/Plone/zinstance
$ pushd src
$ ../bin/zopeskel plone number10.demo

ZopeSkel will ask a few questions. Hit <Enter> on all of them except 'Register Profile'. Set that as True - we want the package to register a GenericSetup profile.

$ popd

Done. number10.demo now exists in the src directory. Next we need to add our new package and the collective.xdv package to the build. Create a file demo.cfg in $HOME/Plone/zinstance containing:

[buildout]

extends =
    buildout.cfg
    http://good-py.appspot.com/release/collective.xdv/1.0rc11

eggs +=
    collective.xdv
    number10.demo

develop =
    src/number10.demo

The second line of the extends stanza refers to a 'known good set' of collective.xdv's dependencies. 1.0rc11 is currently the latest version.

Run buildout with our configuration file to get collective.xdv and its dependencies:

$ ./bin/buildout -c demo.cfg

Now that all necessary packages are downloaded and ready to go, let's flesh out the number10.demo package. We start by making collective.xdv a dependency so that when we install number10.demo, collective.xdv's configuration is installed too. Edit $HOME/Plone/zinstance/src/number10.demo/number10/demo/profiles/default/metadata.xml as follows:

<?xml version="1.0"?>
<metadata>
  <version>1000</version>
  <dependencies>
    <dependency>profile-collective.xdv:default</dependency>
  </dependencies>
</metadata>

Next, we'll point collective.xdv at our files and configure the domain we want themed. Create $HOME/Plone/zinstance/src/number10.demo/number10/demo/profiles/default/registry.xml:

<?xml version="1.0"?>
<registry>
  <record interface="collective.xdv.interfaces.ITransformSettings"
          field="theme">
    <value>python://number10.demo/index.html</value>
  </record>
  <record interface="collective.xdv.interfaces.ITransformSettings"
          field="rules">
    <value>python://number10.demo/rules.xml</value>
  </record>
  <record interface="collective.xdv.interfaces.ITransformSettings"
          field="domains">
    <value>
     <element>localhost:8080</element>
    </value>
  </record>
  <record interface="collective.xdv.interfaces.ITransformSettings"
          field="enabled">
    <value>True</value>
  </record>
</registry>

That's all the setup done; now we can create the theme and rules files.

First the theme file. We want a page from number10.gov.uk that's fairly typical, not a special case like the homepage. The 'Latest News' page has all the right elements: left hand navigation, breadcrumbs, basic content, etc. Either visit that page in your browser and copy/paste the source into index.html, or download it directly:

$ wget -O src/number10.demo/number10/demo/index.html http://www.number10.gov.uk/news/latest-news

Gotta love wget. Now we need to check that CSS resources will load from our local machine. Edit index.html so that all CSS files are referred to by an absolute - not relative - URL, i.e. have an href like http://www.number10.gov.uk/some/path/foo.css, not /some/path/foo.css. Luckily on the Number10 site they're already absolute so there's nothing to do, but I will feel better if line endings are all the same:

$ dos2unix src/number10.demo/number10/demo/index.html # this step optional

Better! Next, create a basic rules file at $HOME/Plone/zinstance/src/number10.demo/number10/demo/rules.xml:

<?xml version="1.0" encoding="UTF-8"?>
<rules
    xmlns="http://namespaces.plone.org/xdv">

  <!-- Head: title -->
  <replace theme="/html/head/title" content="/html/head/title" />

</rules>

Only one rule to begin with. That's enough to see that the XDV transform is working.

Test the transform

Now let's start Plone, install our number10.demo product and see the XDV transform applied. Start the instance in the foreground:

$ ./bin/instance fg

Running Plone like this outputs the log directly in your terminal - useful for spotting problems.

In your browser, go to http://localhost:8080 where you'll see that "Plone is up and running". Hit the button to create a new site and log in with the username/password given when you installed (admin/secret in my example). On the subsequent 'Create a Plone site' form just hit the 'Create Plone Site' button. Hello Plone! Click 'admin' in the top right corner, choose 'Site Setup' from the dropdown menu, then on the control panel go to 'Add-ons' under 'Plone Configuration. Install number10.demo (found at the bottom of the available add-ons list).

At this point the XDV theme should kick in. But unfortunately not. The log shows a problem:

XSLTParseError: Attribute 'onblur': Failed to compile the expression 'this.value = 'Enter search terms';' in the AVT.

Eek. XDV choked on something nasty. Searching in index.html for 'onblur' we find the culprit: inline JavaScript on an <input> tag. We don't need this, so delete the onfocus and onblur attributes, then refresh the page.

This time success! We have the Number10 theme with the page title coming from Plone

Before we continue, now's a good time to populate Plone with some of Number10's content. We configured collective.xdv to theme http://localhost:8080/Plone, but we can still get to the unXDVed theme on http://127.0.0.1:8080/Plone. Point a tab in your browser there, login, and create some folders. For this tutorial I added and published Latest News, Press Briefings and Press Notices inside Plone's usual News folder (allow Folders to be added at http://127.0.0.1:8080/Plone/news/folder_constraintypes_form), which is enough to test breadcrumbs and left hand navigation. For a real demo you'll want at least all the top-level folders.

Writing rules

Now we're getting into it. The rules file maps Plone's content into the theme file. We'll build up our rules in a couple of stages.

XDV's PyPI page explains how these rules work far better than I could here. You definitely should read it. Some of these rules are lifted straight from that page.

Update rules.xml to look like this:

<?xml version="1.0" encoding="UTF-8"?>
<rules
    xmlns="http://namespaces.plone.org/xdv"
    xmlns:css="http://namespaces.plone.org/xdv+css">

  <!-- Head: title -->
  <replace theme="/html/head/title" content="/html/head/title" />

  <!-- Base tag -->
  <append theme="/html/head" content="/html/head/base" />

  <!-- Body -->
  <prepend theme="/html/body" content="/html/body/attribute::class" />

  <!-- Top level navigation -->
  <replace css:theme="#top-menu ul" css:content="ul#portal-globalnav" />

  <!-- Search -->
  <replace css:theme="form#kbs" css:content="form#livesearch0" />

  <!-- Breadcrumbs -->
  <copy css:theme=".bread_crumb"
        css:content="#portal-breadcrumbs > span:not(:first-child)" />

  <!-- Page title -->
  <copy css:theme=".pages_textnews_news h2"
        content="//span[@id='breadcrumbs-1']//text()" />

  <!-- Second level navigation -->
  <copy css:theme=".pages_sidenav_news"
        css:content="#portal-column-one dl.portletNavigationTree ul.navTreeLevel0 > *" />

  <!-- Footer actions -->
  <replace css:theme="#footer_links ul"
           css:content="#portal-siteactions" />

</rules>

Notice that we use XDV's css namespace to get CSS selectors, which are often much easier to use than XPath. Sometimes XPath is necessary though: for the page title we have to replace the content of an H2 tag in the theme with the content of Plone's page title H1 tag. XPath can select content where CSS cannot.

I don't know how I'd write these rules without Firebug. I'm continually switching between XDVed and unXDVed tabs, using Firebug to identify the bits of markup I need. For complex XPath selectors, Firefinder lets you test them directly in the browser.

Getting there. Breadcrumbs, page title and left hand nav all look good.

xsl:template - XDV's samurai sword

What's wrong with the primary navigation styles? Inspecting the theme markup we see that each <a> in the top menu has a class named after the menu item.

<div id="top-menu">
  <ul>
    <li><a class="current_home">Home</a></li>
    <li><a class="current_home_news_active">News</a></li>
    <li><a class="current_home_communicate">Transparency</a></li>
    <li><a class="current_home_meet">Meet the PM</a></li>
    <li><a class="current_home_history">History and Tour</a></li>
    <li><a class="current_home_nr10tv">Number 10 TV</a></li>
  </ul>
</div>

Eugh. Messy. For Number10's existing styles to 'take' we need to reproduce that markup. It's too complex for the usual XDV rules. Time to unsheath the samurai sword that is xsl:template.

As explained on XDV's PyPI page you can include inline XSL in your rules file. These directives apply directly to the markup coming out of Plone. With xsl:template we can match and rewrite arbitrary Plone content.

<!-- Top level navigation - link class -->
<xsl:template match="//ul[@id='portal-globalnav']//li/a">
  <xsl:variable name="i">
    <xsl:value-of select="1 + count(parent::*/preceding-sibling::*)"/>
  </xsl:variable>
  <xsl:variable name="hardcoded_class">
    <xsl:choose>
      <xsl:when test="$i = 1">current_home</xsl:when>
      <xsl:when test="$i = 2">current_home_news</xsl:when>
      <xsl:when test="$i = 3">current_home_communicate</xsl:when>
      <xsl:when test="$i = 4">current_home_meet</xsl:when>
      <xsl:when test="$i = 5">current_home_history</xsl:when>
      <xsl:when test="$i = 6">current_home_nr10tv</xsl:when>
      <xsl:otherwise>current_home</xsl:otherwise>
    </xsl:choose>
  </xsl:variable>
  <xsl:variable name="active">
    <xsl:if test="parent::*[@class='selected']">_active</xsl:if>
  </xsl:variable>
  <xsl:copy>
    <xsl:copy-of select="@*"/>
      <xsl:attribute name="class">
        <xsl:value-of select="$hardcoded_class"/><xsl:value-of select="$active" />
      </xsl:attribute>
    <xsl:copy-of select="node()"/>
  </xsl:copy>
</xsl:template>

Don't forget to add xmlns:xsl="http://www.w3.org/1999/XSL/Transform" to the <rules> element in your rules file.

What's happening here? Here's a pseudocode version:

For every anchor element in Plone's portal-globalnav:

1. Let i = the current position in the list.
2. Let hardcoded_class = the appropriate hardcoded class name, dependent on i.
3. Let active = '_active' if the anchor's parent <li> has the class 'selected'
4. Add a class attribute to the anchor with the value of hardcoded_class + active.

And it works:

Why do I call xsl:template a samurai sword? Because xsl:template will chop your fingers off! It's incredibly useful but best used very sparingly. The code above is hard to understand - I would not ship it in a production theme. Far better to fix how the CSS is done or - if markup changes are necessary - to override the plone.global_sections navigation viewlet to produce the markup we want.

No doubt that samurai swords are cool. Having discovered you have access to such a weapon, maybe you're tempted to do some serious slicing of Plone's default markup. Here's a sharp tip for you: Don't! Overuse of xsl:template will cripple the maintainability of your site. Rewritten the markup for an Archetypes editing widget in your rules file? If the upgrade from Plone 4.x to Plone 4.y includes a change in the widget's underlying .pt file, you've got a hard-to-catch and hard-to-debug broken form lurking in your site. Tempted to reformat the folder_listing page? You'll likely break the user interface because the JavaScript-generated markup for that page will not pass through your rules.

When in a real theme might you use xsl:template then? When you can write a very generic snippet that applies site wide - something that's easy to read at a glance. For example, by convention Plone's portlets are formatted as an HTML definition list element. Using xsl:template you could neatly change that markup to divs across the whole site.

So you've heeded the ol' great-power-great-responsibility mantra and have resolved to master xsl:template and use it wisely. How do you begin your XSLT training? Personally I found Evan Lenz does a great job of demystifying the technology; Evan's O'Reilly book is currently my favourite source of XSLT answers.

The content column

Almost there. Now for the main content. Here're the basic rules to add to your rules.xml:

<!-- Page title -->
<xsl:template match="//h1[contains(concat(' ', normalize-space(@class), ' '),
                     ' documentFirstHeading ')]">
  <h2 class="page_title_border"><xsl:copy-of select="./text()"/></h2>
</xsl:template>

<!-- Content -->
<copy css:theme="#inner_container div:first-child"
         css:content="#portal-column-content > *:not(:first-child)" />
<!-- Drop junk content -->
<drop css:theme="#skip_to_content" />
<drop css:theme="#rssholder" />
<drop css:theme="#inner_container div:not(:first-child)" />
<drop css:theme="#inner_container ~ div.navigation" />

<!-- Content class -->
<xsl:template match="//div[@id='content']">
  <div id='content' class='pages_post_news'>
    <xsl:apply-templates />
  </div>
</xsl:template>

Note the syntax in the first rule. Matching a class on an HTML element in XPath requires a fairly elaborate construct. You usually cannot do a simple *[@class='foo'] because the string must match exactly - that syntax won't match <div class="foo bar"> for example. This works:
*[contains(concat(' ', normalize-space(@class), ' '), ' foo ')]
Ugly, isn't it? If you need XPath that matches a couple of classes and not another, you quickly end up with a multiline monster of a rule. Laying out the code nicely helps, but I'd love to find a way to define custom XPath selectors in XDV...

The two xsl:template hacks force classes necessary to trigger Number10's existing CSS.

Now as a regular user we can navigate around the XDV site and see real pages of content. But as a user with content editing privilege, we get an ugly, unstyled edit bar.

The easiest way to get the edit bar looking good is to use just enough of Plone's own CSS.

Plone 4's theme is called Sunburst. Open the main CSS file at $HOME/Plone/buildout-cache/eggs/plonetheme.sunburst-1.0.6-py2.6.egg/plonetheme/sunburst/skins/sunburst_styles/public.css. Create your own CSS file at $HOME/Plone/zinstance/src/number10.demo/number10/demo/demo.css and copy/paste all the lines starting from /* @group Status messages */ up to but not including /* @group Control panel */. Add the following at the top of the file:

/* Hacks */

#edit-bar li {
    display: inline;
}

#inner_container div#edit-bar,
#inner_container div#edit-bar div {
    padding-right: 0;
}

ul#content-views {
    margin: 0;
    padding: 0;
}

#top-menu {
    width: 71%;
}

form#searchGadget_form {
    margin-top: 1px;
}

/* One more from Sunburst: from @group Invisibles */

.hiddenStructure {
    display: block;
    background: transparent;
    background-image: none; /* safari bug */
    border: none;
    height: 0.1em;
    overflow: hidden;
    padding: 0;
    margin: -0.1em 0 0 -0.1em;
    width: 1px;
}

We need to do two things to activate this stylesheet. First we register it as a resource. Add to $HOME/Plone/zinstance/src/number10.demo/number10/demo/configure.zcml:

  <browser:resource
      name="demo.css"
      file="demo.css"
      />

Make sure to add the browser namespace to the <configure> element: xmlns:browser="http://namespaces.zope.org/browser"

Stop the instance with Ctrl-c then start it in the foreground again. You can now see your stylesheet in the browser at http://localhost:8080/Plone/++resource++demo.css

We need to register that URL with Plone's CSS resource registry. Create a file $HOME/Plone/zinstance/src/number10.demo/number10/demo/profiles/default/cssregistry.xml containing:

<?xml version="1.0"?>
<object name="portal_css">

 <stylesheet
     id="public.css"
     expression="not: request/HTTP_X_XDV | nothing" />

 <stylesheet
     id="++resource++demo.css" title=""
     media="screen" rel="stylesheet" rendering="import"
     cacheable="False" compression="safe" cookable="False"
     enabled="1" expression="request/HTTP_X_XDV | nothing"/>

</object>

With this file we apply a conditional expression to both our new stylesheet and Sunburst's public.css. We want public.css to render on the unXDVed site, demo.css to render on the XDVed site.

Reinstall number10.demo from the 'Add-ons' control panel (deactivate, then activate again). We also need a new line in rules.xml to pull through Plone's styles. (Note that this brings through styles from other Plone stylesheets like member.css).

<!-- Pull in Plone CSS -->
<append theme="/html/head" content="/html/head/link | /html/head/style " />

Ta da! Looking good. Our demo.css introduced the most relevant Sunburst styles, plus some pretty brutal "hacks" to workaround clashes with Number10's existing styles.

Moving up to a full site

So now we have a great-looking, working, demo site. Fingers crossed the client likes what they see and wants you to build their site on Plone. Hoorah! Where to from here?

• Read the guide to theming with collective.xdv on Plone.org, and the excellent XDV and collective.xdv PyPI pages. Everything I've talked about here is documented properly on those pages.
• client.demo is a throwaway package. Follow the collective.xdv worked example to create a proper client.theme package, including a static directory and all your theme's CSS/JavaScript files registered with Plone's resource registries.
• Take the opportunity to rework any nasty markup in the theme (looking at you Number10's global navigation). Or if you're writing the theme file fresh, make the CSS sympathetic to Plone's default markup. Though XDV gives you lots of freedom, you'll have less work to do the less you veer from Plone's non-XDV theming approach.
• If your theme has a reset.css but you want Plone's public.css styles for content editing, it might help to use cssregistry.xml to control the order as: reset.css, Plone's CSS, overrides.css, your theme's CSS. The overrides.css file goes in your theme package and contains styles to fix up any weirdness. Worked for me.
• Inspect the source of the Sunburst theme to see how a modern, non-XDV theme is done. That might seem counter-intuitive, but knowing a little about non-XDV Plone theming will allow you to override existing page elements cleanly.
• On all but the simplest sites you'll want to add custom dynamic page elements. To start with, learn what Plone's viewlets are and how to register your own.

Yes it's true that currently to get the best out of XDV you do need to be somewhat comfortable with the technologies it complements and replaces. The upcoming name change to Diazo shows that the dust hasn't settled yet! For Plone 5, expect Diazo to become the dominant approach and the Plone theming learning curve to mellow out nicely. In the meantime - for just a little effort - XDV can very quickly make your Plone site look great right now.