Wednesday, April 24, 2013

The CPAN Testers game

For those who don't know, one of the greatest features of Perl programming language is the centralized repository of Perl modules known as CPAN (Comprehensive Perl Archive Network).

CPAN includes a vast number of different solutions for a vast range of applications. For almost everything you will be able to find a match (or more) of Perl modules (and maybe using different programming paradigms) for you.

CPAN is not just that: it is highly organized and this includes proper documentation about the standards to be used to create useful Perl modules and make them available.

Those standards includes how to include automated tests for your released modules. Those tests will be executed by:
  1. those people who wants to use your modules;
  2. people who created smokers to test everything at CPAN automatically.
This post is specially about the later.

Anyone can became a tester for CPAN and I'll try to illustrate the steps to become one.

Configuration to submit those distributions installed manually


In fact, this is so simple that anybody that uses CPAN should do it, including system administrators.

The wiki http://wiki.cpantesters.org/wiki/TestDuringInstall have almost all the details related.The tutorial is highly based on UNIX-like operation systems. You may need to do some adaptations (like, for example, using something else instead chmod to give execution permissions).

I also would add a step to upgrade the CPAN module (which is the default shell to interface with CPAN repository) before installing Task::CPAN::Reporter, following the steps below:

$ perl -MCPAN -e shell
cpan[1]> install CPAN
cpan[1]> reload CPAN
cpan[1]> exit

Usually reloading CPAN is enough, but I already have issues by reloading it only, so exiting CPAN and entering again is more an additional guarantee than a necessary step.

Installing Perl modules through CPAN is quite easy and it will make your life easier to install modules, specially because that are additional features that you can add to it (like colorized output and command history).

CPAN can work behind a HTTP/HTTPS proxy if you need to use one, as CPAN-Reporter does. CPAN-Reporter uses HTTPS access to send the tests results, but that can be changed to HTTP by editing the configuration file ~/.cpanreporter/config.ini, changing the protocol transport of the URL metabase.cpantesters.org/api/v1/ from HTTPS to HTTP.

Creating a smoker machine

For software testing practice, smoke tests is a term to designate the possibility to execute automated tests without need of human intervention.

CPAN::Reporter::Smoker will let you do that but the configuration of it is a lot more involving.

First of all, doing smoke tests can be a dangerous thing, specially because you are downloading code from people you don't know (probably) and don't have the slightest idea what the code does. I never heard about somebody that upload malicious code to CPAN, but while that could happen, it is usually ugly bugs that are a risk to damage the smoker machine or even your LAN.

So, to avoid that, the best is to create a Virtual Machine (VM). There are several options to create one. I myself found that using Virtual Box (from Oracle, bought from SUN) a good option (easy to setup and good enough performance). You can use any OS that supports Perl for that.

Which OS is better? Well, Perl runs better in UNIX-like OS's, that is not exactly a secret. Doing smoke tests are far easier at a UNIX-like box. Tests will not usually hang like using Microsoft Windows or Cygwin (I've tried both). Of course there are some modules that runs exclusively in some OS, and you can do some good to add more tests in those "weirdos" instead of a UNIX-like OS.

After that, consider limiting the smoker machine access to your LAN and to the Internet. Smoking tests are also possible by using an HTTP/HTTPS proxy.

The wiki http://wiki.cpantesters.org/wiki/CPANReporterSmokerLong has a detailed description about setting up the smoker, but again is highly based on UNIX-like OS.

There are some steps that I recommend doing before following the wiki instructions:
  1. Install a local perl to avoid root/administrator privileges: you can do it manually or use Perlbrew. I specially like using Perlbrew because I can get a local, customized perl installation without messing with the (probably) already perl setup available. You can get some performance improvement by compiling perl in your machine, specially if you setup it without some unnecessary features. Here there is a good tip about doing a faster Perlbrew setup.
  2. Install the CPAN::SQLite distribution: this will make CPAN to runs faster by using a SQLite database. Installing it may required you to install additional libraries depending on your OS of choice. Be sure to check CPAN::SQLite environment variables too.
  3. Install CPAN::Mini: be sure to read the related Pod to get the details, specially regarding the minicpan command line program.
Finally, after following the instruction in the wiki, be sure to check the hints documentation of CPAN::Reporter::Smoker.

What about the game (in the post title)?

Well, there is a little bit of game about who submit more tests to CPAN: it is just a funny way to incentive "competition" between the testers. There is even a website to give you the details here.

The last time that I checked I was at position 117 (my user id is ARFREITAS). To give you a hint how serious people get about creating smoker machines to submit their tests I created the two following charts below, being the Y axis the number of submissions and the X axis the ranking position:




Actually I would create a single chart, but the difference between the ten first submitters to the rest is so big that you wouldn't be able to see it nicely. As you can see, getting the first place from BINGOS is really a hard task. I hope you got a cluster for that. :-)

Conclusion


Becoming one of the greatest CPAN testers can be fun, but the more important thing is helping the Perl community doing all those tests, even if you're not a developer. Alas, you can learn a lot about tests automation with the experience (I did myself). Also, you can consider that TAP is a big reference about software testing automation, since other languages is implementing the concept.

Anybody that released a Perl distribution on CPAN will get an e-mail about your testing report, and thus have a change to correct bugs that would be totally unexpected when the distribution was initially tested.

After all, in the spirit of open source, everybody wins by getting better code from CPAN.

Acknowledgments 


Thanks to Breno (breno at rio.pm.org) about giving the details about configuring protocol transport for CPAN::Reporter.

Wednesday, April 17, 2013

Reactivating the My Oracle Support autocomplete

This post is an updated version of the original one here (in Brazilian Portuguese).

I'm a Siebel CRM certified professional and I use My Oracle Support some couple of times during a week.

That said, it would be required that I type my login and password every time that I want to go in the website. Well, that is not very fun because even if you want your browser will not remember those values.

I strong disagree with such website configuration: if I want to keep my credentials saved within the browser, that's my problem, not Oracle's. A simple "remember me" check box would be far more polite.

In the meantime, Oracle's website is still HTML with steroids at the end of your HTTP request. Even if they disable the browser login reminder it is still possible to enable it. And that's the trick I'm posting here.

The restriction applied by Oracle is an HTML tag called autocomplete. What it does is tell to the browser if it can or cannot save values from form fields and automatically fill them in again when you come back. If the autocomplete=off, the web browser will not even try to save those values for you.

Nowadays several browsers have tools to access the HTML of the HTTP response and enable the user to change it in memory. Mozilla Firefox, my preferable web browser, has such tools.

There are some bookmarklets that would, theoretically, change those configurations automatically, but the ones that I tried didn't work. But it is still possible to change them manually.

First, install the Firebug plug-in into Firefox.

Then, after restarting Firefox, go to the My Oracle Support website login page and click the "sign-in" button.

The login form will be shown to you (mine is in Brazilian Portuguese, but that doesn't matter). Open the Firebug (use the F12 keyboard shortcut if you can't find the related menu option). You will see something like the screen-shot below:


Just ignore that the form shown has a value for the login field.

In the lower window the Firebug plug-in will show you the source of the HTML. Press CTRL+F and type "autocomplete" as shown in the screen-shot and press the next button. The value of the property will be "off", disabling Firefox for trying to save the login and password.

After that, select the value of the property by left clicking on it: Firebug will offer to change this value. Go ahead and replace "off" with "on".

Now it is just a matter to type your login and password and access the My Oracle Support as you would do normally.

Testing time: logout and go back to the login page.

In the form, double click with the left button in the login field. Firefox will offer to you the login typed previously even though the form from My Oracle Support has the autocomplete=off again.



Finally, select the suggested login and Firefox will fill in both form fields for you. Yes, it is that simple.

Friday, April 5, 2013

Get Siebel Tools 8.1 working with Windows 7

Alright boys and girls... all you know that Microsoft is always innovating: they keep creating new annoyances for each new MS Windows version they release. Today is the day to be annoyed by Windows 7 default file system permissions.

Some couple of days ago I got my old notebook replaced by a shinny new one. The downside is that I had to reinstall and reconfigure a lot of stuff, including Siebel Tools.

I know Siebel Tools 8.1 does not officially supports Windows 7, but there is no way I'll be able to go back to Windows XP (unfortunately in this case).

After running the Siebel Tools setup program as administrator, I got a non-working Siebel Tools: I could not even find out what was going on because I didn't have any permission to create log files under \Log directory.

After fixing that, it was also necessary to change permissions to the sse_data.dbf file since my user got no write rights on it. Obviously I got some permissions issue (maybe because I executed the setup program as Administrator?) even my user being part of the group Administrators of my notebook. I tried the not so fancy solution trying to give me full permissions for all files/directories under the Tools directory.

I didn't work... don't ask my why. Looks like the permission dialog does not work as expected. I tried to grab permissions several times, even tried to put my user as owner of all objects... no results.

I was able to open Tools but take a look of what I got after trying to "Edit Web Layout" of a applet:



Not cool. Probably more permissions issues. This is what I got in the respective log file:

2021 2013-03-25 17:07:28 0000-00-00 00:00:00 -0300 00000000 001 003f 0001 09 siebdev 9796 4412 C:\Siebel\8.1\Tools\log\siebdev.log 8.1.1.6 [21233] ENU
ObjMgrLog    Error    1    0000000251502644:0    2013-03-25 17:07:28    (filetransport.cpp (495)) SBL-GEN-10116: Error acquiring file lock
 

ObjMgrBusServiceLog    Error    1    0000000251502644:0    2013-03-25 17:07:28    (eaixmlprtsvc.cpp (348)) SBL-EAI-04261: Error getting XML from file 'standard.xml'.
 

ObjMgrLog    Error    1    0000000251502644:0    2013-03-25 17:08:45    (filetransport.cpp (495)) SBL-GEN-10116: Error acquiring file lock
ObjMgrBusServiceLog    Error    1    0000000251502644:0    2013-03-25 17:08:45    (eaixmlprtsvc.cpp (348)) SBL-EAI-04261: Error getting XML from file 'C:\Siebel\8.1\Tools\bin\expression_builder.xml'.


ObjMgrBusServiceLog    Error    1    0000000251502644:0    2013-03-25 17:08:45    (ebservice.cpp (3326)) SBL-DEV-61159: No valid Expression Builder could be found for the field Name on business component Repository Applet.


As always, going down to a terminal give me some more powerful options.

First I used the program Takeown from Microsoft itself: it just put everything under my user as proprietary by using it's recursive argument.

takeown /f C:\Siebel\8.1\Tools /r

After that, I went to a Windows Powershell prompt and did the following:

$newACL = Get-Acl C:\Siebel\8.1\Tools
Get-ChildItem C:\Siebel\8.1\Tools -Recurse -Force | Set-Acl -AclObject $newACL

If you know a bit of Powershell you will notice that I got a "ACL model" from the Tools directory itself: at least this directory I could put my desired permissions, the problem was to replicate this configuration for it's children objects. That's why I used it's ACL to replicate the configuration with the Get-ChildItem and Set-Acl cmdlets.

That's it: Siebel Tools was back to work. Of course I probably gave much more permissions to the folders than it was necessary, but anyway, life is too short to go looking for all necessary permissions on a not-available-for-users document from Oracle telling me what is required or not.

Thursday, April 4, 2013

Improving bulk inserts on SQLite

One of these days I decided that I needed to have a way to transport data from Siebel SQLAnywhere local database to anything that I could review information without having to install any Siebel application for it. My choice to go was SQLite and Perl to extract the data.

SQLite is far more "transportable" than SQL Anywhere, specially if you don't need all the fancy features it has when compared with SQLite.

First I checked if SQL::Translator wouldn't do the job, but it seems SQL Anywhere syntax is not supported by now. Then, after playing around with Perl DBI, DBD::ODBC (yes, Siebel local databases still uses ODBC, don't ask me why) and DBD::SQLite, I got something working that could check all SIEBEL schema tables DDL, create the tables into SQLite database and populate those tables with data. After all I got something like a "data dumper".

Unfortunately, performance was not that good. Specifically talking about the computer that I used for that (a old notebook with encrypted HD), the I/O was a performance hog: perl interpreter was not getting even 20% of all processor power available but the insert steps where really slow.

First thing when doing bulk inserts with DBI is to be sure to use binding parameters, which in fact was already applied:
my $dest_insert =
    'INSERT INTO '
  . $table . ' ('
  . join( ', ', @quoted_columns )
  . ') VALUES('
  . join( ', ', ( map { '?' } @quoted_columns ) ) . ')';

my $insert_sth = $sqlite_dbh->prepare($dest_insert);
$select_sth->execute();

while ( my $row = $select_sth->fetchrow_arrayref() ) {

    for ( my $i = 0 ; $i < scalar(@columns) ; $i++ ) {
        $insert_sth->bind_param( ( $i + 1 ),
        $row->[$i], $columns[$i]->sql_type() );
    }

    $insert_sth->execute();
    # and code goes on
}

With my homework done, I went to another basic step: check the storage back-end.

The thing is, the notebook was "plagued" with disk encryption: this means that everything in that HD that needs to be read/write must pass through the trouble to be encrypted/decrypted. While quite secure, doing I/O in these conditions are really bad for bulk inserts.

I decided then to go with a RAM disk: such solutions are uncommon in Microsoft realm, but after some configuration I got some good improvement. The problem is, the notebook didn't have enough RAM to have a RAM disk size that could hold the complete database file and keep everything else running... I ended exhausting all RAM memory available.

Next steps where taking a look about features of SQLite that could help on that... and the little database didn't let me down. Here is a quick list of the features that needed to be turned on:
By enabling those features I was able to get something in the middle of good performance and not exhausting the notebook resources.

One observation: COMMIT control in SQLite is really weird. I finished by using a combination to disabling Autocommit during database connecting and disabling it again after each COMMIT executed.