Finally a milestone as a CPAN tester!

Today I was excited to see that I finally got a second place at number the tests submissions from OpenBSD to CPAN:


It took a while for sure, but I was most of the time running a single smoker VM.

Of course, it would be more cool to get the first place - not much people are running smokers on OpenBSD, despite my efforts to make it easier - but for that I will need to get a working cluster because that's how BINGOS got that gigantic amount of submitted tests. Making plans to release some instances on AWS just to see what is what.

RabbitMQ on Docker with custom configuration

This is a quick tip.

I was thinking in a way of using RabbitMQ on a Docker container as part of an automatic integration tests for a application I was working with at my current job. That is easy, the Docker Hub has an official Docker image ready for use as rabbitmq:management-alpine.

My first thought was to automate the configuration of the RabbitMQ by using rabbitmqctl or rabbitmqadmin programs, but then I remembered that RabbitMQ can actually use a exported configuration backup in the form of a JSON file.

The problem was: how to load this configuration? I can't change the Dockerfile in order to do that because the broker must be running in order to load the configuration but then I would need to override the ENTRYPOINT, something that I was not willing to do.

Fortunately a good soul had already implemented a clever hack to the container configuration as you want, without doing many changes in the original Dockerfile.

So, at the end, the only thing I need to do is to write a Dockefile like the one below:

FROM rabbitmq:management-alpine
COPY rabbit_config.json /etc/rabbitmq/definitions.json

Dropping Archive::Tar::Wrapper support for Solaris

I recently adopt a Perl distribution: Archive::Tar::Wrapper.

Initially developed by Mike Schilli, this module is well know by the Perl community and it is being around since 2005. All started because I submitted a patch to Mike to make it work on OpenBSD.

From there, I tried to fix another known bugs (and introduced some!) and make it run on operational systems that were showing problems.

After fixing the distribution to run on MS Windows properly (with the help of Ingram Braun), the next step would be to fix it for Solaris. Carlos Guevara gave me some pointers and I decided to take a look.

First thing I thought was try to use a VM for it, of course. Vagrant VM's should be easier to use, and I was able to find one with, I think, a decent recent version.

That's is what I got after booting and logging in a shell:

Last login: Wed Jul 18 16:25:27 2018 from 10.0.2.2
Oracle Corporation      SunOS 5.11      11.3    September 2015
-bash-4.1$ perl -v

This is perl 5, version 12, subversion 5 (v5.12.5) built for i86pc-solaris-64int
(with 7 registered patches, see perl -V for more detail)

Copyright 1987-2012, Larry Wall

Perl may be copied only under the terms of either the Artistic License or the
GNU General Public License, which may be found in the Perl 5 source kit.

Complete documentation for Perl, including FAQ lists, should be found on
this system using "man perl" or "perldoc perl".  If you have access to the
Internet, point your browser at http://www.perl.org/, the Perl Home Page.

Oh, come on! perl 5.12?!? Really?

That gave me some thoughts about how old Perl 5.12 is and how little Oracle cares about Perl.

And brought back memories about the time I spent over there, when I was able to see that Perl was used everywhere inside the company, from day-to-day automation to specifically helping their own products. I was even able to find that in the past (and I don't remember how long it lasted) there was a group of people specialized in using Perl to automate anything, or even develop web applications using Catalyst. I wasn't able to contact any member of the group about that time.

Not long after that, I got a blog entry from Perl Weekly about a new release of DBD::Oracle. To my surprise, even after many years Oracle still makes the life of Perl programmers unnecessary miserable when they just need to connect their code to a Oracle database and have to do all sorts of configurations to just install a Perl module due binary dependencies.

Long story short: I gave up trying to support Solaris. Just put

if ( $^O eq 'solaris' ) {
    die "OS unsupported\n";
}

in the Makefile.PL and life goes on.

I won't reject a patch to make Archive::Tar::Wrapper on Solaris, as long it passes on the unit tests and don't break on other OSes, but from my perspective, the Perl community doesn't really need anything from Oracle.

I have the utmost respect about Sun Microsystems and all the contributions it gave to the community but for Oracle? Not really. At all.

Quickly making a DEB from a Perl module

Recently I had to quickly create a DEB package for Ubuntu for a Perl module available on CPAN (not sure why REST::Client is not available anymore).

It has been a while since the last time I did, and honestly I couldn't remember what was required to.

After some frustating minutes trying to go over the documentation for dh-make-perl, I was able to reach those two simple lines in a shell:

$ export DEBFULLNAME='Alceu Rodrigues de Freitas Junior'
$ dh-make-perl --cpan REST::Client --arch all --depends libjson-xs-perl, liblwp-protocol-https-perl, liburi-perl libwww-perl --email glasswalk3r@yahoo.com.br --build

Documentation of dh-make-perl does have a lot of space of improvement... and for my dismay, it is not even easy to do make contributions for it: no available access to the repository as you would expect to do nowadays (git pull request), first you got to be accepted as a Debian contribuitor... which is not that easy too.

Perl on Windows... still not that good

Today I had to use Perl from Windows... it was a couple of years ago that I needed to do it so.

Frankly, it was never the best experience from a end user point of view. Strawberry Perl made it a lot easier, but to this day it is still an incomplete solution.

I don't want just to complain about it... if you don't like a opensource project, don't use it. Or try to improve it yourself.

Here are some notes for those willing to do the later:

1. Install Git for Windows. You will get some other nice programs as tar and gzip. You will need those and what the heck, everybody uses git nowadays.
2. Configure the cpan program. You can use the default configuration offered and set some items later.
3. Configure cpan to use the tar and gzip programs from step 1. In the case you don't remember, type o conf tar and o conf gzip and setup the complete path to those guys. If you were unwise and setup Git for Windows use the suggested path (C:\Program Files\Git), you will have to do some quoting and escaping the backslash (like "C:\\Program Files\\Git\\usr\\bin\\tar.exe").
4. Configure cpan to use the gmake.exe program (don't really know why it wasn't configured by default, considering that gmake.exe is part of Strawberry Perl): o conf make C:\\Strawberry\\c\\bin\\gmake.exe
5. Configure cmd.exe to use Unicode by default: fire up the command REG ADD HKCU\Console /v CodePage /t REG_DWORD /d 0xfde9 with administrator privileges in order to do that. It avoids you having warnings all over because of Unicode characters.
6. Configure cpan to use a SQLite database (which speeds up the process): o conf use_sqlite 1
7. Install YAML::XS, which is faster than YAML.
8. Be smart and add C:\Program Files\Git\usr\bin to your PATH variable.

cron versus Crom

Do you know cron? If you ever worked on a UNIX-like OS, it's very likely you do!

But what do you know about Crom?

What have they in common besides the (almost) identical names?

Let's see how the venerable cron compares to the imortal Crom!

	cron	Crom
What is it?	Software time-based job scheduler for UNIX-like OSes.	Fictional Cimmerian deity of Hyborian Ages.
Uses	Schedule tasks to be executed, per user, with granularity of one minute.	Basically nothing. Some believe that he bestow men and women at birth with the courage to survive, persevere, and vanquish adversity.
Creation	1979 (Version 7 UNIX).	1932. Crom is the most venerable indeed.
Versions	Several (proprietary, open source and free software) implementations.	There is only one Crom, you hound dog son of a thousand fathers!
How to use it	Executing crontab -e, or editing the several files that comprises it's configuration	Invoke his name during an oath or curse. That's the only know safe way to mention him.
Known problems	It will issue error messages if you messed up with the expected syntax	He has little patience for weakness, which means asking him for help will make him to ignore you (at best) or really f@#$%&! you (at worst).

Getting Ubuntu, Python 2.7, Jedi and Vim playing nice to each other

This took me some hours to figure out: how to finally set Jedi with Vim to have a good editor for Python 2.7 code.

Ubuntu 16.04 by default install Vim compiled against Python 3.5 interpreter, which is not immediate obvious but becomes a pain in the ass after you discover that even after installing Jedi and python-jedi, you're still getting the error message:

Please install Jedi if you want to use jedi-vim.

When opening a file with Python code.

To get things working for your Python 2.7 code, you might need to uninstall your current Vim related packages and install the following:

• vim-addon-manager
• vim-common
• vim-gnome-py2
• vim-gtk3-py2
• vim-gui-common
• vim-nox-py2
• vim-python-jedi
• vim-runtime

You can check your current configuration with:

$ dpkg-query -l 'vim*'

Moving away from Dist::Zilla @Basic

Today a got an issue registered at my distribution Linux::Info and decided to take a quickly look at it since it was quite a while since the last release.

The issue itself was not that interesting (looks like the distribution Pod could receive some more attention, which indeed I added), but also gave me a chance to try and add a cpanfile to it.

Some months ago I was notified that my distributions were not providing a cpanfile. I wasn't even aware what was it about and what I was missing.

The thing is that a cpanfile is just another way to declare the distribution dependencies. Not really interesting from this point of view.

The good part of it is to allow the developer to install the distribution without having to install Dist::Zilla (and it's quite large number of dependencies) before even being able to build the distribution (well, assuming that the distribution uses Dist::Zilla).

That means you can skip some minutes and download bandwitdh, ignoring whatever Dist::Zilla depends on. Which also means you can just git clone a Perl project and install the requisites automatically with cpanm with two steps:

1. git clone
2. cpanm --installdeps .

That's it! Well, almost, you might need to install cpanm first. And be sure to include the dot in the step 2, it matters.

If your distribution uses Dist::Zilla already, you can add the cpanfile automatically by just adding [CpanFile] to the dist.ini file. The plug-in does the rest for you. Almost.

In order to have this cool stuff available from your code repository, you will need to have the cpanfile available over there... that means you need to build the distribution and copy the file to the repository, where it is going to be source controlled as everything else.

To cover that, you can use Dist::Zilla::Plugin::CopyFilesFromBuild, another plug-in to make your life easier.

But if you have the [@Basic] declared at the dist.ini (as I did) to load the most basic Dist::Zilla plug-ins you need, you will have a nasty surprise:

[DZ] attempt to add cpanfile multiple times; added by: GatherDir (Dist::Zilla::Plugin::GatherDir line 100); CPANFile (Dist::Zilla::Plugin::CPANFile line 65)

And the distribution generation is aborted. I was able to even find a bug registered about that, but it is not really a bug after all: it happens because the GatherDir, automatically registered by [@Basic], is not configured to ignore the cpanfile over there.

I tried to to that, but instead of getting rid of the error, I got a lot more of them, all of them related to GatherDir. Don't know why, but I was able to fix by removing [@Basic] and putting only the plug-ins I was indeed using and the over automatically included by the bundle and adding the following two lines to configured GatherDir:

[GatherDir]

exclude_filename = cpanfile

So, fixing the errors were a nice side effect of review what I indeed need in my distribution in terms of plug-ins.

Bottom line: Dist::Zilla::PluginBundle::Basic is great to start from scratch, but keep in mind that once your dist.ini is not that "basic" anymore, you will need to start cutting-off plug-ins!

Stop pestering me Yahoo!

I'm a user from Yahoo! from a long time ago already... something between 12 to 15 years.

For the most of the time, I can tell you I was a happy user... Yahoo! web interface was modern and I used it too with POP3 (later moved to IMAP of course) and SMTP with different e-mail clients.

From the last few years, well... we all know things are not good with Yahoo! anymore... Google took from it a good amount of the payed advertisement with it's (much more powerful) search engine. But from the past few months, the e-mail system is becoming a pain in the ass.

First, the SPAM filter doesn't work well. And I can't disabled it. I tried to refine it with some training but it seems it will be forever stupid because it keeps putting e-mails from mailing list that I'm subscribed for years in the SPAM folder.

Second, Yahoo! thinks my e-mail client is not secure anymore. Now and then, when I got check my e-mail on Thunderbird, I got a error message stating that the authentication process failed and asks me to try a different password. There is no password change, it is Yahoo! pestering me to stop using a e-mail client (yes, I tested with Claws too).

I didn't change my password, so let's go back to the web interface and see what is going on.

OK, it asks me my same old user and password. And this "nice" Captcha. And here is the reason for not being able to use Thunderbird:

For non Brazilian Portuguese speakers, it says:

"Alceu... disable the less secure access to your Inbox"
"Disable the applications that are not from Yahoo and that use methods of entering less secure to access the Yahoo Mail. If you don't do that, your account will be vulnerable"

Well... that's just bullshit. I use the same authentication method (Yahoo! doesn't asks me for a Captcha unless I got blocked first on my e-mail client) with Thunderbird, and also the connection uses TLS the same way that my browser uses.

Let's also add that I access it from my home network, that is behind a router/firewall, using a cabled connection (not WiFi) and that my PC uses a firewall as well.

What is considered secure as well? Their mobile application... that I should install on a Android, which I have little to no control (unless I got root on it) of how it works and the software that is installed and have to use WiFi or the shitty 4G available on Brazil.

Yeah, it doesn't look very secure too me so... STOP PESTERING ME Yahoo!

How to disable all logging from a Siebel component

You can change parameter to TRUE from the UI:

Here is default output from srvrmgr program:

srvrmgr:ssia811> list hidden param BypassHandler for comp EAIObjMgr_enu

PA_ALIAS PA_VALUE PA_DATATYPE PA_SCOPE PA_SUBSYSTEM PA_SETLEVEL PA_DISP_SETLEVEL PA_EFF_NEXT_TASK PA_EFF_CMP_RSTRT PA_EFF_SRVR_RSTRT PA_REQ_COMP_RCFG PA_NAME

------------- -------- ----------- --------- ------------- ------------- ---------------- ---------------- ---------------- ----------------- ---------------- ----------
-------------------
BypassHandler False Boolean Subsystem Event Logging Default value Default value N N N N Do not send events to logging

1 row returned.

Here is the output after I changed from UI:

srvrmgr:ssia811> list hidden param BypassHandler for comp EAIObjMgr_enu

PA_ALIAS PA_VALUE PA_DATATYPE PA_SCOPE PA_SUBSYSTEM PA_SETLEVEL PA_DISP_SETLEVEL PA_EFF_NEXT_TASK PA_EFF_CMP_RSTRT PA_EFF_SRVR_RSTRT PA_REQ_COMP_RCFG PA_NAME

------------- -------- ----------- --------- ------------- ----------- ------------------- ---------------- ---------------- ----------------- ---------------- ---------
--------------------
BypassHandler True Boolean Subsystem Event Logging Comp level Component level set N N N N Do not send events to logging
1 row returned.

srvrmgr:ssia811>

FAQ

• This is will disable logging completely? Or the component log file will be created empty?

Setting BypassHandler to true will disable logging on component level there should be no logs.

• This parameter can be enabled to other components? What are the restrictions?

You can enable same parameter for another components – no restrictions.

• Is the output from srvrmgr "list hidden param" a bug?

It is not a bug, work this way as designed.

Playing around with Vagrant

I finally finished preparing a Virtualbox image for OpenBSD to execute the CPAN::Reporter::Smoker, all provisioned by Vagrant.

Most of the provisioning was implemented with shell script (Bash) and honestly took me much more time I initially thought it would take.

Although I had almost all configuration steps already documented, the development cycle was long, specially because I had to wait for perl compilation as well all the required Perl modules. Disabling tests when provisioning did reduce the time (for the trade off regarding stability) , but that is still a long way to go.

Having to create a local mirror (CPAN::Mini) of CPAN inside the VM was not fun either: after finishing my first Vagrant box, I realized that shipping the VM with the mirror pre-initialized was not a good idea: the box ended with 3.2GB.

Worst... the mirror on the box might get considerable outdated pretty quickly.

Even though I was able to reduce the total size of the mirror in about 1Gb, it is still a lot, even to download. To my surprise, Vagrant doesn't seem to be able to resume downloads at all, which surprised me. This post confirms that too. It turns out that downloading boxes with wget on internet connections that sucks (like mine) is a better idea... At least I can download the box and still keep watching Netflix by using --limit-rate with it.

Lessons learned, I think I'll just cut out the pre-initialized CPAN mirror. At least this should give people a opportunity to maybe reuse their own CPAN on their local networks, although I do think that having a independent VM running the mirror is not a bad idea... disk is cheap anyway these days.

Next step should take Collectd installed on the OpenBSD VM... and try to figure out the best set of CPUs to be configured on it.

Poor's man test notification

Somewhere 2014, my manager asked me to write down test cases for the application the team was developing. He didn't care the programming language I was going to use as long as I could implement it quickly.

Those tests should run regularly, and an e-mail should be sent in the event things go south.

Being used to write tests with TAP (Test::More), I went with Perl and Linux crontab for the job. Besides the tests itself (not really interesting for this post anyway), I end up with a generic script to run over the tests, analyze the results and send an e-mail with a nice HTML report attached. Here is the script:

use warnings;

use strict;

use TAP::Harness 3.30;

use TAP::Formatter::HTML 0.11;

use Email::Stuffer 0.009;

use Email::Sender::Transport::SMTP 1.300010;

use MIME::Base64 3.14;

use Authen::SASL 2.16;

use Config::Tiny 2.20;

use lib './lib';

my @tests = glob('Tests/*.t');

foreach my $test (@tests) {

   my $output_filename = 'results.html';
   my $fmt = TAP::Formatter::HTML->new();

   $fmt->output_file($output_filename);

   $fmt->verbosity(-2);

   my $harness = TAP::Harness->new(

       {

           formatter => $fmt,

           merge     => 1,

           lib       => ['lib']

       }

   );

   my $aggregator = $harness->runtests($test);

   if ( $aggregator->has_problems ) {

       send_email( $test, $output_filename );
   }

   else {

       remove_report($output_filename);
   }

}

sub remove_report {

   my $report_filename = shift;
   unlink $report_filename

     or warn "Could not remove unused report file $report_filename: $!";

}

sub send_email {

   my ($test_name, $attachment) = @_;

   my $cfg = Config::Tiny->read('xyz.ini');

   my $body = <<BLOCK;

Greetings,

Sadly a test ($test_name) executed against XYZ REST API failed.

Please check the attachment for details.

BLOCK

   my $transport = Email::Sender::Transport::SMTP->new(

       {

           host          => $cfg->{SMTP}->{'smtp.host'},

           ssl           => 1,

           sasl_username => $cfg->{SMTP}->{'smtp.username'},

           sasl_password => $cfg->{SMTP}->{'smtp.password'},

       }

   );

   Email::Stuffer->from( $cfg->{SMTP}->{'smtp.default.from.email'} )

     ->to('afreitas@foobar.com')->text_body($body)->transport($transport)

     ->subject('XYZ REST API failed')->attach_file($attachment)->send;

   remove_report($attachment);
}

Being generic enough, I could just point the script to different folders containing the tests and hopefully got nothing because all tests passed. In fact, it was so simple I wondered why something like this was not available on CPAN.

Well, not anymore... Test::Health is available for download already, so you can install it and run the script health_check.pl to run your tests. Since this script was created initially to health check the application, the name makes sense... but it seems to be generic enough to be reused in different situations that I didn't foresee when I wrote it.

The only thing that is missing from my original script is the SMTP authentication... hopefully I should implement subclasses to make it in the future (but contributions in that sense are welcome!).

Happy testing!

OpenBSD CPAN smoker: mission accomplished

I finally reached my personal set milestone regarding configuring and running a CPAN smoker on OpenBSD OS and grabbed me a honorable third place in number of submissions:

That was quite a ride! Despite my short experience with OpenBSD, I naively though it wouldn't be that hard... well, it was hard enough to generate some debate on CPAN Testers discussion list because  looks like nobody is doing much test on OpenBSD (or at least is not submitting results about it) nowadays.

All things considered, I decided to go a little further, so I grabbed all my lessons learned and created a document about the process. This work result is available at http://wiki.cpantesters.org/wiki/SmokerOnOpenBSD.

Painless (native) load balancing your Siebel Enterprise

So, how do you setup the native load balancer for your AOM components in a Siebel Enterprise?

You can do it manually, of course. That's the way people is doing for years already. Just go through the documentation, plan and execute all the search-and-edit steps (although you actually don't have actually to think too much after the planning phase since those tasks are quite repetitive).

Or, you can jump the traditional last steps by:

1. Installing Siebel::Lbconfig (available in your nearest CPAN mirror)
2. Reading how to create a INI file.
3. Execute in your shell:

lbconfig -e </my/directory/to/eapps*.cfg> -p <SCBroker port number> -c <my INI file>

The lbconfig program will connect to your Siebel Enterprise (using the information provided in the INI file) and create a lbconfig.txt in your current directory, with all required configuration to use the native Siebel Load Balancer. And, in the directory defined with the -e parameter, you're have the all the eapps*.cfg files copied and updated with the definitions from lbconfig.txt file (they will have a ".new" extension added).

Before you get too excited, I cheated a bit about the whole process because you're going to have install first the Siebel::Srvrmgr Perl distribution (also available in your nearest CPAN mirror). But you already have it, haven't you? :-)

Just kidding, be sure to use the last release of Siebel::Srvrmgr (0.27), some important updates were made over there, specially to allow Siebel::Lbconfig do it's stuff:

• Support the Siebel components' start mode column from the "list comp" command output.
• Support the Siebel Server Id from the "list server" command output.

Since this is the first release of Siebel::Lbconfig, I wouldn't be much surprised if there is a bug or two, even though there are unit tests in place (actually this is my first run with TDD and Perl). Be careful and please let me know if you find something wrong.

Comparing Siebel component parameters

Sometimes I get some ideas for a new post on this blog but usually I don't have the time to write it down to completion. To not loose completely the opportunity, usually I start a post with only some basic notes and leave it for future review.

That's exactly how this post started. :-)

I was asked some time ago to provide performance information about EAI components deployed in a specific Siebel Enterprise. A very basic step-by-step receipt to recover those details are listed below:

• Find which servers have the EAI components deployed
• Get those parameters for each server
• Generate a (hopefully) nice to read report of it.

If I want just to compare values, Siebel already has the cfgmerge tool for that. Here is a nice reference about it: http://siebeltechworld.blogspot.com/2013/04/component-parameter-export-and.html.

Since I'm a fan of srvrmgr program (in detriment of using IE + Siebel Web Client + CRTL C + CRTL V) and was on a Linux environment, that is how I did it (after connecting to the Siebel Enterprise to check which servers had the component):

srvrmgr> list comp EAI%

SV_NAME|CC_ALIAS|CC_NAME|CT_ALIAS|CG_ALIAS|CC_RUNMODE|CP_DISP_RUN_STATE|CP_STARTMODE|CP_NUM_RUN_TASKS|CP_MAX_TASKS|CP_ACTV_MTS_PROCS|CP_MAX_MTS_PROCS|CP_START_TIME|CP_END_TIME|CC_INCARN_NO|CC_DESC_TEXT|
someserver01|EAIObjMgr_enu|EAI Object Manager (ESN)|EAIObjMgr|EAI     |Interactive|Activated         |Auto        |0               |20          |1                |1               |2014-10-23 01:37:37|           |            |            |
someserver02|EAIObjMgr_enu|EAI Object Manager (ESN)|EAIObjMgr|EAI     |Interactive|Activated         |Auto        |0               |100         |1                |5               |2014-10-23 01:15:09|           |            |            |
someserver03|EAIObjMgr_enu|EAI Object Manager (ESN)|EAIObjMgr|EAI     |Interactive|Activated         |Auto        |0               |100         |3                |5               |2014-10-23 01:24:14|           |            |            |

I could also used the Server Group feature (if implemented) for that but that was not the case. And then I added following commands to a text file named "params.txt":

list param MaxTasks for component EAIObjMgr_enu
list advanced param MaxSharedDbConns for component EAIObjMgr_enu
list advanced param MinSharedDbConns for component EAIObjMgr_enu
list param MaxMTServers for component EAIObjMgr_enu
list param MinMTServers for component EAIObjMgr_enu
list param BusObjCacheSize for component EAIObjMgr_enu

And applying a bit of Bash scripting:

$ for i in $(seq 1 3); do srvrmgr -e ENTERPRISE -g $SIEBEL_GATEWAY -u sadmin -p PASS -s "someserver0${i}" -i $HOME/params.txt -o $HOME/params.out -b; cat $HOME/params.out >> "$HOME/params_result_${i}.txt"; done; rm $HOME/params.out

And done! Right? No... some more editing will be required here to get a nice table with the information per server. Boring.

When I wrote Siebel::Srvrmgr I thought it would be a better idea to make something generic to be reused later. That's a nice example of that, enter Siebel::Params::Checker.

Built upon Siebel::Srvrmgr, the Siebel::Params::Checker distribution provides the command line program scpc.pl (not a very creative name, I know) that does exactly the steps I gave above plus generating a nice to read (hopefully) HTML report.

And you should get something like that after the program finishes (you're on your own to open the HTML file):

Enjoy it! Release 0.001 is already available on CPAN, 0.002 should follow in the next few days.