Current Python, Zope and Plone Testing frameworks

( ... a few things I picked up in Naples ... Ed Crewe ILRT Oct 2007)

Ed Crewe

ILRT - Internet Development Group

Plone 3 - Test and deploy

Testing in Plone - Tutorial on plone.org

• Plone's usage has grown in complexity and scale
• Hence the major new developments in zope / plone testing and deployment frameworks
• To aid that there has been an effort to bring the python, zope and plone methodologies for these more in alignment
• Regarding testing the attitude now is ... you must do it for released code ... and if you don't do it for your own sites and code as well ... you will not be seen as a professional.

Lots of New frameworks

When you create a new site with Plone, the least you should do is create a policy product.

Now zope uses eggs ... python jars which package up the Product(s) with their python dependencies and version compatibility. Use the egg paster utility to create a plone style egg ...

> $PYTHON_HOME/bin/paster create -t plone mynewsite.policy

- ready for installation via the zope buildout deployment mechanism (developed as part of the zope3 grok framework).

NB: These eggs are kept in a repository now rather boringly called the python package index which was called the cheeseshop ... and there is an emerging cheesecake system that is used to check if your code is good - no tests = inedible cake!

'I wanna tell you a story ...'

• The project specification - should become the stories for the eggs
• ... the story should become a series of high level functional docs
• ... the docs should be rewritten as doctests (docs with embedded python test code)
• ... each of these functional test lines can then break down into unit tests
  - along with interdependency testing via regression/integration tests
  
• ... oh and once you have all these tests - you can start writing some code ;-)

Lets test something

So the standard zope server now has all the testing framework for unit, functional, regression and doctests built in .... why not install Plone3 and have a look ...

• Change to the plone instance directory
• The test features in standard zope:
  > bin/zopectl test --help
• The 50 tests that come in one Plone product:
  > ls Products/Archetypes/tests
• An example of a doctest that uses the framework's browser:
  > cat Products/Archetypes/tests/editing.txt
• Run the Archetypes tests:
  > bin/zopectl test Products.Archetypes

How should I get started?

1. Run other peoples tests
   - to test your own customisations and any third party products are not incompatible or buggy
2. Be more formal with your own manual testing
   - we all test really, every time we edit a page template, we check to see it looks right.
   Before you embark on any coding at least try to write down the manual functional test you are going to do to check it works ... try to consider edge cases, and comment it properly, ie. why does it do what it does?
3. Document your overall functional tests - This should detail how a user makes the system do the original project spec. Now you should write or auto-generate these tests - the rest of the talk demos a tool to do this, Funkload, there are also browser based ones, eg. Selenium etc.

Well we would all like to be good developers and follow test driven development practice ... and I am sure we all will with our new sites ... which will be products in subversion with full coverage tests ... but even if you can't start writing tests tommorrow, the following are good starting points:

1. Run other peoples tests
   - to test your own customisations and any third party products you may have chosen to use have not broken things due to errors, or incompatiblity
2. Be more formal with your own manual testing
   - we all test really, every time we edit a page template, we check to see it looks right.
   Before you embark on any coding at least try to write down the manual functional test you are going to do to check it works ... try to consider edge cases, and comment it properly, so you know what its meant to do even if you haven't written the test for it yet.
3. Document your functional tests - Knitting together the individual functional test documentation should give you an overall set of functional tests you need to run through to check your code is working. These should also be derived from original functional specification you worked to.
   Generate functional tests - You can now use macro style tools to generate automated functional tests ... the rest of the talk details one of these, Funkload, there are also browser based ones, eg. Selenium etc.

   Write your own tests

   If you do any customisation to Plone, you have to write some stylesheets, code, etc. You should then put it into a versioning system ... unless you are a God who never needs to undo anything

   Of course even the most minimal customisation of Plone should be done as a product, so that it can be deployed.

   Once you are at this stage you should be writing tests to check the product code.

   .

Functional and Unit load testing tool

Funkload is a great utility for recording functional tests, but it is much more than that.

It can run any type of tests and hence be integrated with high level doctests, unit tests or use low level pdb

Along with running the tests in can do so as multiple users, to provide graphical benchmarking information out of all the tests mentioned above ... hence check whether your production site will actually cope with predicted usage.

NB: At Bristol we use MRTG and Munin to graph system load on the CMS server farm, to anticipate overall load issues, and perl scripts for peak page view load testing (~ 50 requests per sec) ... but with Funkload we could better check the overhead of new features, products and related usage patterns, before they are rolled out.

Funkload record functional test

The following demos how you would use it to record a functional test ... editing a page, adding another and publishing.

Installation is well documented but involved, my browser needs to be set to go via a TCPWatch proxy. Once done, I can just run the following:

> fl-record add_page
I go to the site I want to record my test from and do my edit, add and publish in Firefox. I can then replay this and get timing information
> fl-run-test -dV test_AddPage.py

Files fl-record generates: add_page-test.log | add_page-test.xml | test_AddPage.py

Funkload Benchmarking

Now lets do some functional load test benchmarking with my macro created test case ... using 1, 5, 10 and 20 concurrent users ...
> fl-run-bench -c 1:5:10:20 test_AddPage.py AddPage.test_add_page
Files fl-run-bench generates: add_page-bench.log | add_page-bench.xml

We want to generate nice HTML report from our benchmarking XML data file ...
> fl-build-report --html -o /var/www/htdocs/funkload add_page-bench.xml
So lets have a look at it ...
/testing/test_add_page-2007-10-19T12-59-08/index.html

Ooooh isn't it pretty ... but urrggh look how the successful requests drop between 10 and 20 concurrent users!

Now go and write some tests...

For a full step by step guide see Martin Aspeli's tutorial and his book is also based on test driven development.
I also attended the Naples talk on testing given by Untested code is Broken code by Philipp von Weitershausen and Martin Aspeli. Which evangelised test driven development.

PS: I should perhaps come clean here and say I have only written unit tests as part of java projects ... so I am a novice in the python world of unit testing myself. However in order to bring our code base back into line with Plone 3, and benefit from the new frameworks, plus start releasing Products, we have to get on board with testing. Which will hopefully have the many side benefits such as stopping nasty surprises and aiding productivity and consistency across our growing number of Plone developers.