Wednesday, April 24, 2013

The CPAN Testers game

For those who don't know, one of the greatest features of Perl programming language is the centralized repository of Perl modules known as CPAN (Comprehensive Perl Archive Network).

CPAN includes a vast number of different solutions for a vast range of applications. For almost everything you will be able to find a match (or more) of Perl modules (and maybe using different programming paradigms) for you.

CPAN is not just that: it is highly organized and this includes proper documentation about the standards to be used to create useful Perl modules and make them available.

Those standards includes how to include automated tests for your released modules. Those tests will be executed by:
  1. those people who wants to use your modules;
  2. people who created smokers to test everything at CPAN automatically.
This post is specially about the later.

Anyone can became a tester for CPAN and I'll try to illustrate the steps to become one.

Configuration to submit those distributions installed manually

In fact, this is so simple that anybody that uses CPAN should do it, including system administrators.

The wiki have almost all the details related.The tutorial is highly based on UNIX-like operation systems. You may need to do some adaptations (like, for example, using something else instead chmod to give execution permissions).

I also would add a step to upgrade the CPAN module (which is the default shell to interface with CPAN repository) before installing Task::CPAN::Reporter, following the steps below:

$ perl -MCPAN -e shell
cpan[1]> install CPAN
cpan[1]> reload CPAN
cpan[1]> exit

Usually reloading CPAN is enough, but I already have issues by reloading it only, so exiting CPAN and entering again is more an additional guarantee than a necessary step.

Installing Perl modules through CPAN is quite easy and it will make your life easier to install modules, specially because that are additional features that you can add to it (like colorized output and command history).

CPAN can work behind a HTTP/HTTPS proxy if you need to use one, as CPAN-Reporter does. CPAN-Reporter uses HTTPS access to send the tests results, but that can be changed to HTTP by editing the configuration file ~/.cpanreporter/config.ini, changing the protocol transport of the URL from HTTPS to HTTP.

Creating a smoker machine

For software testing practice, smoke tests is a term to designate the possibility to execute automated tests without need of human intervention.

CPAN::Reporter::Smoker will let you do that but the configuration of it is a lot more involving.

First of all, doing smoke tests can be a dangerous thing, specially because you are downloading code from people you don't know (probably) and don't have the slightest idea what the code does. I never heard about somebody that upload malicious code to CPAN, but while that could happen, it is usually ugly bugs that are a risk to damage the smoker machine or even your LAN.

So, to avoid that, the best is to create a Virtual Machine (VM). There are several options to create one. I myself found that using Virtual Box (from Oracle, bought from SUN) a good option (easy to setup and good enough performance). You can use any OS that supports Perl for that.

Which OS is better? Well, Perl runs better in UNIX-like OS's, that is not exactly a secret. Doing smoke tests are far easier at a UNIX-like box. Tests will not usually hang like using Microsoft Windows or Cygwin (I've tried both). Of course there are some modules that runs exclusively in some OS, and you can do some good to add more tests in those "weirdos" instead of a UNIX-like OS.

After that, consider limiting the smoker machine access to your LAN and to the Internet. Smoking tests are also possible by using an HTTP/HTTPS proxy.

The wiki has a detailed description about setting up the smoker, but again is highly based on UNIX-like OS.

There are some steps that I recommend doing before following the wiki instructions:
  1. Install a local perl to avoid root/administrator privileges: you can do it manually or use Perlbrew. I specially like using Perlbrew because I can get a local, customized perl installation without messing with the (probably) already perl setup available. You can get some performance improvement by compiling perl in your machine, specially if you setup it without some unnecessary features. Here there is a good tip about doing a faster Perlbrew setup.
  2. Install the CPAN::SQLite distribution: this will make CPAN to runs faster by using a SQLite database. Installing it may required you to install additional libraries depending on your OS of choice. Be sure to check CPAN::SQLite environment variables too.
  3. Install CPAN::Mini: be sure to read the related Pod to get the details, specially regarding the minicpan command line program.
Finally, after following the instruction in the wiki, be sure to check the hints documentation of CPAN::Reporter::Smoker.

What about the game (in the post title)?

Well, there is a little bit of game about who submit more tests to CPAN: it is just a funny way to incentive "competition" between the testers. There is even a website to give you the details here.

The last time that I checked I was at position 117 (my user id is ARFREITAS). To give you a hint how serious people get about creating smoker machines to submit their tests I created the two following charts below, being the Y axis the number of submissions and the X axis the ranking position:

Actually I would create a single chart, but the difference between the ten first submitters to the rest is so big that you wouldn't be able to see it nicely. As you can see, getting the first place from BINGOS is really a hard task. I hope you got a cluster for that. :-)


Becoming one of the greatest CPAN testers can be fun, but the more important thing is helping the Perl community doing all those tests, even if you're not a developer. Alas, you can learn a lot about tests automation with the experience (I did myself). Also, you can consider that TAP is a big reference about software testing automation, since other languages is implementing the concept.

Anybody that released a Perl distribution on CPAN will get an e-mail about your testing report, and thus have a change to correct bugs that would be totally unexpected when the distribution was initially tested.

After all, in the spirit of open source, everybody wins by getting better code from CPAN.


Thanks to Breno (breno at about giving the details about configuring protocol transport for CPAN::Reporter.

No comments:

Post a Comment