Wednesday, January 28, 2015

Faster imports of Siebel Repositories


So, it is just another day in your SADMIN life. The development team just finished cooking their configurations and now they want to test their doing in another Siebel Enterprise, which implicates that you need to import a new Siebel Repository.

Well, all you need is to launch Database Server Configuration Utility, go over all those dialog boxes filling the fields as necessary and choosing the correct options to import a Siebel Repository.

Lame try... as you probably did that on MS Windows, didn't you ever pay attention to the command line application that is invoked when you finish entering the details? That guy over there execute "repimexp", which is a command line program that you can actually use on your own.

Well, by using this program you can select different options that the Database Server Configuration Utility will not expose to you. And two of those options will allow you to dramatically improve the speed of importing a new Siebel repository.

Don't bother looking around a documentation that gives you all those details... they are only at command line help message. Let's review it:

bash-3.2$ repimexp
Siebel Enterprise Applications Repository Import/Export Utility, Version 8.1.1.7 [21238] LANG_INDEPENDENT
Copyright (c) 1990-2008, Oracle. All rights reserved.


The Programs (which include both the software and documentation) contain
proprietary information; they are provided under a license agreement containing
restrictions on use and disclosure and are also protected by copyright, patent,
and other intellectual and industrial property laws. Reverse engineering,
disassembly, or decompilation of the Programs, except to the extent required to
obtain interoperability with other independently created software or as specified
by law, is prohibited.


Oracle, JD Edwards, PeopleSoft, and Siebel are registered trademarks of
Oracle Corporation and/or its affiliates. Other names may be trademarks
of their respective owners.


If you have received this software in error, please notify Oracle Corporation
immediately at 1.800.ORACLE1.


repimexp


Error: Value missing for argument 'Rows Per Commit'.
Usage:
-------------- required parameters -----------------


/C <ODBC data source>   ODBC data source.
                          Default: SIEBEL_DATA_SOURCE environment variable.
/U <userName>           User name
/P <password>           Password
/D <table owner>        Siebel database table owner.
                          Default: SIEBEL_TABLE_OWNER environment variable.
/R <repository>         Repository name, default: Siebel Repository
/F <dataFile>           Import/Export/Dump repository data file's name


------------- choose which action you want to take by /A -------------
------------- and select the sub-parameters accordingly --------------


/A <D|I|X|E>            Dump/Import/Import_INTL/Export actions


/A D                    Action: Dump basic data info from the datafile


/A I                    Action: Import base tables and (optional) INTL tables
  /G <languages>       (Optional) Specify what languages to import, e.g. ENU,FRA,ITA
                                  ALL for all languages. Default: no language import
  /X <Codepage#>       (Optional) Verify (not really import) the data file against
                                  a codepage character set, e.g. CP1252,CP874 ...
  /H <number>          (Optional) Number of rows per commit
  /8 <moduleFile>      (Optional) Module list filename
  /K <Y|N>             (Optional) Preserve DB system column values (Default: N)
  /J <string>          (Optional) DB system column source (DB_LAST_UPD_SRC) (Default: repimexp)
  /Z <number>          (Optional) Array Insert Size (Default: 5)


  /A X                    Action: Import INTL tables only
  /G <languages>       Specify what languages to import, e.g. ENU,FRA,ITA
                          ALL for all languages
  /O <Y|N>             (Optional) Abort INTL import if unable to resolve parent row
                                  in server repository, i.e. orphans
                                  Default: N
  /I <Y|N>             (Optional) Abort INTL table import if insert fails
                                  Default: Y
  /A E                    Action: Export
  /1 <exp rep userName>       (Optional) Default: to same as User name
  /2 <exp rep password>       (Optional) Default: to same as Password
  /3 <exp rep ODBC dt src>    (Optional) Default: to same as ODBC data source
  /4 <exp rep table owner>    (Optional) Default: to same as Siebel database table owner
  /5 <exp rep repository>     (Optional) Default: to same as Repository name
  /E <Y|N>                    (Optional) Export prototype data. Default:N
  /S <Signature>              (Optional) Repository signature
  /N <0|1|2>                  (Optional)
                                  0: no change.
                                  1: change CREATED_BY, UPDATED_BY, OWNER_BRANCH
                                  2: change CREATED_BY, UPDATED_BY, dates columns, OWNER_BRANCH
                                  Default: 1


------------ other optional parameters ---------------


/L <logFile>            (Optional) Log output messages to this file as well.
/W <lang code>          (Optional) The language env where this program is running.
                                  Default: SIEBEL_LANGUAGE, if not set ENU
/B <appServer root>     (Optional) Siebel server installation directory to
                                  override SIEBEL_HOME environment variable.
/T <Y|N>                (Optional) Test/Debug use only, do not import into database
/V <Y|N>                (Optional) Verify data.  For import, default: Y
                                  For export, default: N
/M <Y|N>                (Optional) Commit changes even if verification failed.
                                  Default: N

I highlighted the options that will help you get better performance.

The /H is the one that you will use in most cases, and probably the one that gives you the best improvement of speed. It's is basically allowing you to set the number of rows imported before doing a single commit. If you don't set this option, the program will commit every line inserted, which is a very secure operation... but also a lot slower.

It is all about trading off data consistence with I/O speed. The point is, the worst thing that might happen with a repository import that went bad is that you will need to erase it and repeat the operation. Not a big deal in most of cases.

That said, talk to your DBA about how much lines he considers reasonable inserting before a single commit to avoid using all your UNDO (or whatever other term is used by the DBMS Siebel is installed over) space and set the /H option with it.


The /G options allows you to select which languages you want to import. If you have a multilingual repository but do not need all those languages, just select the one you need it. That will reduce the number of rows to be imported. This parameter is not as good as /H but it will help anyway.

Another tip to improve Siebel Repository maintenance speed is to keep only the most recent ones imported (see Doc Id 761894.1). By my experience, keeping the current and the previous one should be enough. You don't need to try doing "big data" with Siebel Repositories. If you are worried about backups, just use the .dat files for that and erase all of those older repositories. Your DBA will thank you later for that.

Installing openSSL inside your Perlbrew

A while ago I had to install some monitoring scripts in reasonable old Linux servers. The script would require a substantial number of Perl modules from CPAN and I was not interested in running the issue to convince the administration responsible team (I didn't have root access) to install those modules for me.

The easiest way to solve that was installing Perlbrew. Perlbrew allows anyone to download perl source code, compile and install anywhere, including your own home directory (Perlbrew is available only for UNIX-like operational systems at the moment I'm writing this post).

Since I didn't have direct access to the internet on those servers, what I did was create a VM with the same Linux distribution and version and install Perlbrew from there, connecting to CPAN and installing all dependencies that I need too.

After that I just had to generate a tarball from everything and move to those Linux servers. Another wonder from Perl community made my day.

After some days of having those test scripts working, I had receive a new testing request: validate if some webpages were working fine. Perl has a lot of tools to help with that, so no issue here. The problem is that those webpages where available only through TLS and the version available of openSSL was so old that newer versions of IO::Socket::SSL would simple not compile with it.

Old libs and no play makes Jack a dull boy

IO::Socket::SSL depends on Net::SSLeay, that will then look for shared libraries of openSSL to compile C/XS code.If there are changes in the expected interface, this code will not compile at all.

Some how I would need to avoid using the present openSSL binaries and download and install a newer one inside my Perlbrew setup. I spent an unpleasant time trying to do that (I'm not a C programmer) because first I thought that I should use static libraries, than I tried to force shared libraries... and during all this time I was being stomped by SELinux (yes, it was installed in all those servers).

It turned out at the end that a much more simple was available: just compile openSSL with standard options. And some changes regarding system variables. Let's review my recipe on that:
  1. install Perlbrew: you probably already know how to do that. If not, just visit it's website for instructions.
  2. Install openSSL: no magic here, the simplest
  3. Installing Perl modules that depends on openSSL

Step 2 is pretty easy. I suggest to set up the option --prefix as a sub directory under the Perlbrew install. For instance, that might be something like:

./config shared --prefix=$PERLBREW_ROOT/openssl
make
make test
make install

For Step3, some previous setup is necessary.

Create a test file named "openssl_env" in your $HOME with the following content:

export OPENSSL=$PERLBREW_ROOT/openssl
export C_INCLUDE_PATH=$OPENSSL/include
export LIBRARY_PATH=$OPENSSL/lib
export LD_LIBRARY_PATH=$LIBRARY_PATH:$LD_LIBRARY_PATH
export OPENSSL_PREFIX=$PERLBREW_ROOT/openssl

Now go back to the shell and type:

perl -MCPAN -e shell
get Net::SSLeay
look Net::SSLeay

perl Makefile.PL
make
make test
make install
exit

Make sure all tests from "make test" worked fine. If not, check if the declared variables paths are correct.

Now edit again the openssl_env file and comment the C_INCLUDE_PATH variable. Should be unnecessary to have it setup unless you need to compile more stuff.

Now add the following to the end of your .bashrc (or whatever other shell configuration file you're using):

source $HOME/openssl_env


Now, go back to the CPAN shell to install IO::Socket::SSL:

perl -MCPAN -e shell install IO::Socket::SSL

If you did everything right, now you should have a plenty functional customized Perl with openSSL working!

Wednesday, November 26, 2014

Better checking your Java version

A couple of days ago I received a task to check installed applications in a bunch or servers. One of the things to check was the Java VM installed into those and see which versions was there.

It looks a simple task, but it is not: some applications requires different versions of Java, so there is none "one JVM to rule them all".

It should a be simple thing to do it automatically, but actually Java doesn't help as much I as would like. This is the output from Java in my MS Windows box:

C:\>java -version
java version "1.7.0_71"
Java(TM) SE Runtime Environment (build 1.7.0_71-b14)
Java HotSpot(TM) Client VM (build 24.71-b01, mixed mode, sharing)

C:\>


The version is there, right? Well, it is, but can you tell me if this JVM is 64 or 32 bits version?

The truth is, if the JVM is 64 bits it should let me know... but for 32 I'm just guessing. That's OK when I'm doing it manually, but guessing from script is not so.

After some searching over the Internet I got this ridiculous simple code, actually I could say this could be the "Hello World" example of Java, and here it goes:

public class JavaArch {
 

    public static void main(String[] args) {
                          System.out.println(System.getProperty("sun.arch.data.model"));


    }

}


Compile that code with javac and ship it with your script and you're good to go. No guessing required, just the bare information.

C:\>javac JavaArch.java

C:\>java JavaArch
32

C:\>


Can't be easier than that.

See more information about here.

If you need even more properties available for the environment from the JVM point of view, go checking http://docs.oracle.com/javase/7/docs/api/java/lang/System.html#getProperties%28%29.

Here is an example from my environment:

C:\>java JavaArch
java.runtime.name = Java(TM) SE Runtime Environment
sun.boot.library.path = C:\Program Files (x86)\Java\jre7\bin
java.vm.version = 24.71-b01
user.country.format = BR
java.vm.vendor = Oracle Corporation
java.vendor.url = http://java.oracle.com/
path.separator = ;
java.vm.name = Java HotSpot(TM) Client VM
file.encoding.pkg = sun.io
user.script =
user.country = US
sun.java.launcher = SUN_STANDARD
sun.os.patch.level = Service Pack 1
java.vm.specification.name = Java Virtual Machine Specification
user.dir = C:\Users\foobar\Documents\temp
java.runtime.version = 1.7.0_71-b14
java.awt.graphicsenv = sun.awt.Win32GraphicsEnvironment
java.endorsed.dirs = C:\Program Files (x86)\Java\jre7\lib\endorsed
os.arch = x86
java.io.tmpdir = C:\Users\foobar\AppData\Local\Temp\
line.separator =

java.vm.specification.vendor = Oracle Corporation
user.variant =
os.name = Windows 7
sun.jnu.encoding = Cp1252
java.library.path = C:\Program Files (x86)\Java\jre7\bin;C:\windows\Sun\Java\bin;C:\windows\system32;C:\windows;C:\oracle\product\11.2.0\client_1\bin;C:\windows\system32;C:\windows;C:\windows\System32\Wbem;C:\windows\System32\WindowsPowerShell\v1.0\;C:\Program Files\Lenov
o\Fingerprint Manager Pro\;C:\strawberry\c\bin;C:\strawberry\perl\site\bin;C:\strawberry\perl\bin;C:\oracle\product\11.2.0\client_1\bin;C:\windows\system32;C:\windows;C:\windows\System32\Wbem;C:\windows\System32\WindowsPowerShell\v1.0\;C:\Program Files\Lenovo\Fingerprint
Manager Pro\;C:\strawberry\c\bin;C:\strawberry\perl\site\bin;C:\strawberry\perl\bin;C:\Program Files (x86)\GnuWin32\bin;C:\Program Files (x86)\Java\jre7\bin;C:\Program Files (x86)\PuTTY;C:\Users\foobar\Downloads\sqldeveloper\jdk\bin;.
java.specification.name = Java Platform API Specification
java.class.version = 51.0
sun.management.compiler = HotSpot Client Compiler
os.version = 6.1
user.home = C:\Users\foobar
user.timezone =
java.awt.printerjob = sun.awt.windows.WPrinterJob
file.encoding = Cp1252
java.specification.version = 1.7
user.name = foobar
java.class.path = .
java.vm.specification.version = 1.7
sun.arch.data.model = 32
java.home = C:\Program Files (x86)\Java\jre7
sun.java.command = JavaArch
java.specification.vendor = Oracle Corporation
user.language = en
user.language.format = pt
awt.toolkit = sun.awt.windows.WToolkit
java.vm.info = mixed mode, sharing
java.version = 1.7.0_71
java.ext.dirs = C:\Program Files (x86)\Java\jre7\lib\ext;C:\windows\Sun\Java\lib\ext
sun.boot.class.path = C:\Program Files (x86)\Java\jre7\lib\resources.jar;C:\Program Files (x86)\Java\jre7\lib\rt.jar;C:\Program Files (x86)\Java\jre7\lib\sunrsasign.jar;C:\Program Files (x86)\Java\jre7\lib\jsse.jar;C:\Program Files (x86)\Java\jre7\lib\jce.jar;C:\Program Files (x86)\Java\jre7\lib\charsets.jar;C:\Program Files (x86)\Java\jre7\lib\jfr.jar;C:\Program Files (x86)\Java\jre7\classes
java.vendor = Oracle Corporation
file.separator = \
java.vendor.url.bug = http://bugreport.sun.com/bugreport/
sun.cpu.endian = little
sun.io.unicode.encoding = UnicodeLittle
sun.desktop = windows
sun.cpu.isalist = pentium_pro+mmx pentium_pro pentium+mmx pentium i486 i386 i86


And the updated code to retrieve that:

import java.util.*;

public class JavaArch {

    public static void main(String[] args) {
   
        Properties props = System.getProperties();
        Enumeration names = props.propertyNames();
       
        while (names.hasMoreElements()) {
       
            String current = names.nextElement().toString();
       
            System.out.println(current + " = " + System.getProperty(current));
       
        }
   
    }

}

Monday, July 21, 2014

Some notes about srvrmgr usage

The command line program srvrmgr is not that popular in Siebel, but any reasonably seasoned sadmin will soon discover how useful it is considering the GUI limitations.

One of the most obvious reasons to use is when the AOM is out-of-service for any reason and you don't have a Siebel Client Dedicated configured. Other usage is for tasks automation.

This posts is about some tips that I found here and there about using srvrmgr.

Note: this post was updated at October 6, 2015 to include additional content.

Hide your password

Let's consider for a moment the password usage with the program: you may have notice that doesn't matter if you use the "-p" option or not, the password is always echoed to the standard output!

There are different forms to solve this, but the better one is to disable the echo when typing the password. Several programming languages may help you in this. With Perl, for example, this is very straightforward in a Microsoft Windows environment:

use Term::ReadKey qw(ReadMode);
use Win32::Process;

my $server     = '';
my $gateway    = '';
my $enterprise = '';
my $user       = '';
my $srvrmgr    = 'sea752\\client\\BIN\\srvrmgr.exe';

system('chcp 1252');

print 'Type your password to connect to Siebel Server: ';
ReadMode(2);
my $password = ;
ReadMode(0);

chomp($password);

print "\n\n";

my $process;

Win32::Process::Create( $process, $srvrmgr,
    "srvrmgr /e $enterprise /g $gateway /u $user /p $password",
    0, NORMAL_PRIORITY_CLASS, '.' )
  or die errorReport();

$process->Wait(10);

sub errorReport {

    print Win32::FormatMessage( Win32::GetLastError() );

}

Of course, any modern programming language will help you with that.

If you have a MS Windows environment and is particularly interested in better hiding the password, take a look at the Server Manager 2 program that is part of the Siebel GNU Tools project.

Configuring the output

Character encoding

First, let's talk about configuring character encoding. Let's suppose that your system is configured in PTB and you run a simple list comp command in the srvrmgr prompt:

srvrmgr> list comp

CC_ALIAS           CC_NAME                               CG_ALIAS      CC_RUNMODE   CP_DISP_RUN_STATE
-----------------  ------------------------------------  ------------  -----------  -----------------
ClientAdmin        Client Administration                 System        Background   Ativado
CommConfigMgr      Communications Configuration Manager  CommMgmt      Batch        Ativado
CommOutboundMgr    Communications Outbound Manager       CommMgmt      Batch        Ativado
CommSessionMgr     Communications Session Manager        CommMgmt      Batch        Ativado
DbXtract           Database Extract                      Remote        Batch        Ativado
EAIObjMgr_enu      EAI Object Manager (ENU)              EAI           Interactive  Ativado
MailMgr            Email Manager                         CommMgmt      Background   Ativado
EIM                Enterprise Integration Mgr            EAI           Batch        Ativado
FSMSrvr            File System Manager                   System        Batch        Ativado
GenNewDb           Generate New Database                 Remote        Batch        Ativado
GenTrig            Generate Triggers                     Workflow      Batch        Ativado
PageMgr            Page Manager                          CommMgmt      Background   Ativado
PDbXtract          Parallel Database Extract             Remote        Batch        Em execuþÃ’o
RepAgent           Replication Agent                     Remote        Background   Ativado
ServerMgr          Server Manager                        System        Interactive  Em execuþÃ’o
SRBroker           Server Request Broker                 System        Interactive  Em execuþÃ’o
SRProc             Server Request Processor              System        Interactive  Em execuþÃ’o
SynchMgr           Synchronization Manager               Remote        Interactive  Ativado
TxnMerge           Transaction Merger                    Remote        Background   Em execuþÃ’o
TxnProc            Transaction Processor                 Remote        Background   Em execuþÃ’o
TxnRoute           Transaction Router                    Remote        Background   Em execuþÃ’o
UpgKitBldr         Upgrade Kit Builder                   SiebAnywhere  Batch        Ativado
WorkActn           Workflow Action Agent                 Workflow      Background   Ativado
WorkMon            Workflow Monitor Agent                Workflow      Background   Ativado
WfProcBatchMgr     Workflow Process Batch Manager        Workflow      Batch        Ativado
WfProcMgr          Workflow Process Manager              Workflow      Batch        Ativado
ePharmaObjMgr_enu  ePharma Object Manager (ENU)          LifeSciences  Interactive  Ativado
ePharmaObjMgr_esn  ePharma Object Manager (ESN)          LifeSciences  Interactive  Ativado
ePharmaObjMgr_ptb  ePharma Object Manager (PTB)          LifeSciences  Interactive  Ativado  
  
Pretty standard output, except by the weird characters. Output from srvrmgr is usually Unicode (UTF-8) but your OS terminal must be set to use it.

If your are in a Microsoft Windows environment, you will have to change your current code page. Your default configuration may vary, but to get all the characters corrected printed (assuming the output is in UTF-8) you will need to set it to 65001:

I:\> chcp 65001
Active code page: 65001

That will do. In Linux, most probably your terminal is already configured to UTF-8. If not, you may want to take a look at exporting the LC_ALL variable to your language. For Brazilian Portuguese, that would be the desired value:

export LC_ALL=pt_BR.UTF-8

That may vary depending in your Linux distribution. If you are on UNIX, you may need to consult the OS documentation for that.
It is also good to remember that the spool command may also follow your database encoding. At least when I used databases in UTF-8, the output of srvrmgr followed the same.

Manipulating the commands output

It is possible to change the way that srvrmgr will show the output from commands. For example, the default output of list comp is this:

srvrmgr> list comp
SV_NAME     CC_ALIAS                       CC_NAME                                   CT_ALIAS  CG_ALIAS      CC_RUNMODE   CP_DISP_RUN_STATE  CP_NUM_RUN_  CP_MAX_TASK  CP_ACTV_MTS  CP_MAX_MTS_ CP_START_TIME        CP_END_TIME  CP_STATUS  CC_INCARN_NO  CC_DESC_TEXT
----------  -----------------------------  ----------------------------------------  --------  ------------  -----------  -----------------  -----------  -----------  -----------  -----------  -------------------  -----------  ---------  ------------  ------------
siebsrvdev  BusIntBatchMgr                 Business Integration Batch Manager                  EAI           Batch        Online             0            20           1            1            2012-03-01 14:59:11
siebsrvdev  BusIntMgr                      Business Integration Manager                        EAI           Batch        Online             0            20           1            1            2012-03-01 14:59:11
siebsrvdev  ClientAdmin                    Client Administration                               System        Background   Online             0            1                                      2012-03-01 14:59:11
        
The output is a bit difficult to read and I did it intentionally: this is exactly how the data is displayed in my terminal. If you use this frequently, it may be a pain.
This can be changed with the command configure list. Let's see it's on line help:

srvrmgr> help configure list
  configure list  
       [show  [ AS ]]
       [,     [ AS ]] ....

The command allows you to configure the columns that will be shown, the size of it and even creating a different alias for them.

You can also check the current configuration of list comp:

srvrmgr> configure list comp
    SV_NAME (31):  Server name
    CC_ALIAS (31):  Component alias
    CC_NAME (76):  Component name
    CT_ALIAS (31):  Component type alias
    CG_ALIAS (31):  Component GRoup Alias
    CC_RUNMODE (31):  Component run mode (enum)
    CP_DISP_RUN_STATE (61):  Component display run state
    CP_NUM_RUN_TASKS (11):  Current number of running tasks
    CP_MAX_TASKS (11):  Maximum tasks configured
    CP_ACTV_MTS_PROCS (11):  Active MTS control processes
    CP_MAX_MTS_PROCS (11):  Maximum MTS control processes
    CP_START_TIME (21):  Component start time
    CP_END_TIME (21):  Component end time
    CP_STATUS (251):  Component-reported status
    CC_INCARN_NO (23):  Incarnation Number
    CC_DESC_TEXT (251):  Component description

With the show option you be able to see which columns you want and in the desired sequence:

srvrmgr> configure list comp show SV_NAME(31),CC_ALIAS(21),CC_NAME(76)
srvrmgr> configure list comp
        SV_NAME (31):  Server name
        CC_ALIAS (21):  Component alias
        CC_NAME (76):  Component name
srvrmgr> list comp
SV_NAME     CC_ALIAS              CC_NAME
----------  --------------------  ----------------------------------------
siebsrvdev  BusIntBatchMgr        Business Integration Batch Manager
siebsrvdev  BusIntMgr             Business Integration Manager
siebsrvdev  ClientAdmin           Client Administration
siebsrvdev  CommConfigMgr         Communications Configuration Manager
siebsrvdev  CommInboundMgr        Communications Inbound Manager
...
siebsrvdev  eSitesClinicalObjMgr  eSitesClinical Object Manager (PTB)
52 rows returned.

Now it is easier on the eyes! But what if you need all the columns to be displayed?
There is a trick for that and as far as I know, it's undocumented. I present you the set command. With it you can change or (show current) configuration. Let's see the options available:

The parameter we want here is ColumnWidth. If you enable it will be possible to define the width of the columns output using the configure list command:

srvrmgr> set ColumnWidth true
srvrmgr> set
username            sadmin*
service             (null)
enterprise          (null)
app server          (null)
compression         FALSE*
encryption          FALSE*
protocol            TCP/IP*
port                (null)*
cluster             (null)*
name server         (null)*
log filter          (null)
verbose             FALSE
batch               FALSE
header              TRUE
footer              TRUE
ColumnWidth         1
delimiter


All done! If you paid attention, you saw that the configure list command uses the column name followed by a parenthesis with a integer between: this is the width in characters of the column. With the command below you can change the columns width to have longer (or narrower) output. Beware that you may truncate the output or even the columns names if the width is not larger enough:

srvrmgr> configure list comp show
SV_NAME(31),CC_ALIAS(21),CC_NAME(76),CT_ALIAS(31),CG_ALIAS(31),CC_RUNMODE(31),CP_DISP_RUN_STATE(61),CP_NUM_RUN_TASKS(16),CP_MAX_TASKS(12),CP_ACTV_MTS_PROCS(17),CP_MAX_MTS_PROCS(16),CP_START_TIME(21),CP_END_TIME(21),CP_STATUS(251),CC_INCARN_NO(23),CC_DESC_TEXT(251)

Time saving tips

Avoid some typing

After configuring your optimal output, you may wonder that is a bit boring to repeat this every time. That's right, it would be boring.
Here are some commands to save your time and patience:
  • alias
  • save preferences
  • load preferences
  • read
alias is for creating aliases for your commands. You type:

alias [name] [command]

And that's it.

save and load preferences will keep your current environment configuration to a text file and load those preferences, respectively. The default location for this file will always be the bin directory where the srvrmgr program is available and the name of the file will always be ".Siebel_svrmgr.pref".
Beware that any set you used won't be save to the preferences file. Of course, being a text file is just a matter to open it and repeat the commands there.
Then you may be in trouble if the srvrmgr is shared between different users. You may want to have your own copy of preferences. That's where read commands helps:

read c:\temp\create_custom_compdefs.txt

Use the wild card

You can filter further the output from any command by using the wild card "%" (without the parenthesis) for some of the values that are part of the output. Depending on the command, this is be applicable to different columns but not all of them. For example, if you want to list components that have the alias starting with "wf":
list comp wf%
The wild card only matches what starts or ends with the given pattern. You cannot use both at the same time.

Configure and use Server Group feature

Since Siebel version 8 you can use the feature of Server Group with srvrmgr. This feature allow you to create... servers groups. This will allow you to connect to a defined group and send a single command to all of them at once. When you want to connect to the server group, include the /z command line option to the server, followed by the server group name you had defined previously.
To create a server group, use the command below to each one of the server that you want to be part of the group:
  
change attribute groupname=[server group name] for server [siebel server name]

This example is assuming that you had connected to the Enterprise instead of specifying the Siebel server name.
To remove a Siebel Server from a Server Group use:

change attribute groupname=" " for server [siebel server name]

For any reason that escapes my imagination, it is necessary to have a space between the double quotes.

Performance tips

And here are some tips related to performance when using srvrmgr:
  • srvrmgr program instantiates new processes of Server Manager component, which is not a multi-thread component, that means that each connection creates a new process, which is a expensive task for any OS.
  • use the "/s" options whenever possible to retrieve data: instead of connecting to the enterprise and submitting a command to all servers, connect individually to each server and retrieve the data you want: this will avoid serializing the output and creating additional processes in the other servers instead of a single one.
  • if you have a lot of commands to send to the Server Manager component, don't use several srvrmgr executions with the "/c" option. Instead, write all the commands to a text file (one command per line) and use the "/i". This will use a single connection to run all the commands. You most probably will want to the check the "/o" and "/b" options too.

Batch mode tips

srvrmgr  can also work in batch mode. That means you can connect only once and get out only after all the commands were executed. There are several situations when this feature is interesting. One is executing a large amount of commands that you might need to repeat several times. Another is executing some commands in a scheduled task. These are some tips for batch mode:
  • use the /b option whenever possible. This options will required that all commands executed before the current command were executed successfully. Any command that raised an error will cause the program to abort execution. This will help you to generate more errors or putting the system into a invalid state.
  • use the sleep command to put a time between a command and other. Useful if you starting a server or are submitting a lot of new tasks.
  • specify a value for the parameter TaskTag when starting a new task. This text will appear in the list tasks command if you include the TK_TASKTAG column with the configure output command or the show parameter.
  • you can omit the header when connecting with the Siebel Server by using the -q command line option of srvrmgr (also undocumented).

Caveats

There are some caveats using srvrmgr. I'll give some workarounds when possible.

Your settings with output width is not working

If you are fetching information from several servers, you will notice that the output may be larger than the width you set. This seems to happen when one of the servers have larger output for a column than the other servers. If you get output from a single servers you will be able to control that.

Another possibility is to set the delimiter variable to a single character to use as delimiter. Once you do this you will be able to fetch correctly the values from one column to another. In fact, you can totally ignore column width output unless some columns are having their name truncated due being larger than their respective output (for example, columns with Boolean values).

To set the delimiter, you can use the set delimiter command inside the srvrmgr prompt or use the command line option -k for it. Both expects as parameter a single character to be used as the delimiter.

Advanced parameters output configuration is not working

When setting list parameters output, all works. For any reason, the same is not applied to the list advanced parameters command.

You can't set the output of list advance parameters directly with configure. Also, you can't rely on the list parameters to get the same formated output.

This looks like a bug that I saw only with Siebel 8.1. So far, the workaround is not setting the list parameters output anyhow before running list advanced parameters: if you do it, the respective columns that shows when the parameters starts to take effect will be truncated.

set command is not saved to preferences files

If you need to keep set commands to the preferences files, know that save preferences will not help you. Instead, go there and write the command manually. The load preferences will do the expected thing.

The non-documented commands

Despite the fact that you can always use the on-line help command, there are some commands that are not documented there.

list advanced parameters

To see advanced parameters for any one of the available items of list parameters command.

list hidden parameters

To see hidden parameters for for any one of the available items of list parameters command.

List threads id's

When analyzing Siebel component crashes, having the associated threads id's will help you identifying the root cause of the errors.

srvrmgr> list tasks show CC_ALIAS, TK_TASKID, TK_TID, TK_PID

This will generate a list of tasks and include the component alias (CC_ALIAS), the task id (TK_TASKID), the OS thread id (TK_TID), and OS process id (TK_PID).

Note the TK_TID and TK_PID values for the appropriate task id that you have flushed the FDR buffer for. This will help you find the appropriate FDR file (the last part of the file name should map to the TK_PID value), and after decoding the file, will help you identify the entries relevant to the task id you are interested in. Each entry should have a ThreadID value equal to the TK_TID value. This is especially important when considering that a single process may have many threads.

Listing servers id's

The command list server show SBLSRVR_NAME, SV_SRVRID allows you to show the server Id created when the server was installed in the Siebel Enterprise. This information is required when configuring the native load balance of AOM's. The configure command for list server will not show the column SV_SRVRID either.

list procs

This command will show the process associated with the Siebel components with some additional data.

srvrmgr:foobar_srv01> configure list procs
        SV_NAME (31):  Server name
        CC_ALIAS (31):  Component alias
        TK_PID (11):  Task process id
        TK_SISPROC (11):  Task sisproc number
        TK_NUM_NORMAL_TASKS (11):  Number of normal tasks in this process
        TK_NUM_SUBTASK_TASKS (11):  Number of subtask tasks in this process
        TK_NUM_HIDDEN_TASKS (11):  Number of hidden tasks in this process
        PROC_VM_FREE_PAGES (11):  Process virtual memory free pages
        PROC_VM_USED_PAGES (11):  Process virtual memory used pages
        PROC_PH_USED_PAGES (11):  Process physical memory used pages
        TK_IS_PROC_ENABLED (31):  Is the process enabled for tasks
        TK_DISP_RUNSTATE (31):  Process run state
        TK_SOCKETS (11):  Sockets Received

Some sample data:

srvrmgr:foobar_srv01> list procs

SV_NAME       CC_ALIAS                   TK_PID  TK_SISPROC  TK_NUM_NORMAL_TASKS  TK_NUM_SUBTASK_TASKS  TK_NUM_HIDDEN_TASKS  PROC_VM_FREE_PAGES  PROC_VM_USED_PAGES  PROC_PH_USED_PAGES  TK_IS_PROC_ENABLED  TK_DISP_RUNSTATE  TK_SOCKETS
------------  -------------------------  ------  ----------  -------------------  --------------------  -------------------  ------------------  ------------------  ------------------  ------------------  ----------------  ----------
foobar_srv01  CommInboundRcvr            5504    29          0                    0                     35                   947650              100925              40205               True                Running           0
foobar_srv01  ServerMgr                  2153    119         1                    0                     2                    1020680             27895               7538                True                Running           0
foobar_srv01  EAIObjMgr_enu              5394    24          0                    0                     7                    914435              134140              74499               True                Running           0
foobar_srv01  EAIObjMgr_enu              5371    23          0                    0                     7                    930429              118146              60284               True                Running           0
foobar_srv01  EAIObjMgr_enu              5353    22          0                    0                     7                    916442              132133              73079               True                Running           0
foobar_srv01  eCommunicationsObjMgr_enu  5558    34          0                    0                     7                    749560              299015              183322              True                Running           0
foobar_srv01  ServerMgr                  2910    120         1                    0                     2                    1020804             27771               7523                True                Running           0
foobar_srv01  eCommunicationsObjMgr_enu  5702    39          0                    0                     7                    714186              334389              216924              True                Running           0
foobar_srv01  SiebSrvr                   5024    1           2                    0                     2                    1039160             9415                6633                True                Running           0
foobar_srv01  eCommunicationsObjMgr_enu  5571    32          1                    0                     13                   712222              336353              217281              True                Running           0
foobar_srv01  WfRecvMgr                  5184    13          0                    0                     26                   954054              94521               38468               True                Running           0
foobar_srv01  ServerMgr                  15100   121         1                    0                     2                    1020315             28260               7299                True                Running           0
foobar_srv01  WfProcMgr                  5203    15          0                    0                     27                   887391              161184              99693               True                Running           0
foobar_srv01  AdminNotify                5094    7           0                    0                     17                   1035909             12666               2607                True                Running           0
foobar_srv01  SvrTaskPersist             5056    8           1                    0                     3                    961872              86703               40503               True                Running           0
foobar_srv01  SRProc                     5093    6           2                    0                     6                    995037              53538               30798               True                Running           0
foobar_srv01  CommConfigMgr              5487    28          0                    0                     26                   954182              94393               38165               True                Running           0
foobar_srv01  eSalesObjMgr_enu           5433    25          0                    0                     7                    1018825             29750               6389                True                Running           0
foobar_srv01  EAIObjMgr_enu              5313    20          0                    0                     7                    866626              181949              124903              True                Running           0
foobar_srv01  eSalesCMEObjMgr_enu        5724    40          0                    0                     7                    1018834             29741               6451                True                Running           0
foobar_srv01  SCBroker                   5048    4           1                    0                     2                    1039526             9049                2375                True                Running           0
foobar_srv01  ServerMgr                  14885   100         1                    0                     2                    1020803             27772               7788                True                Running           0
foobar_srv01  SRBroker                   5039    2           43                   0                     10                   1007397             41178               8950                True                Running           0
foobar_srv01  EAIObjMgr_enu              5291    19          0                    0                     7                    904051              144524              84758               True                Running           0
foobar_srv01  SCCObjMgr_enu              5227    16          0                    0                     7                    1018808             29767               6381                True                Running           0
foobar_srv01  WfProcBatchMgr             5162    11          0                    0                     26                   954304              94271               38448               True                Running           0
foobar_srv01  eCommunicationsObjMgr_enu  5656    38          1                    0                     9                    730965              317610              200334              True                Running           0
foobar_srv01  CommInboundProcessor       5530    30          0                    0                     56                   947578              100997              37553               True                Running           0
foobar_srv01  SCBroker                   5040    3           1                    0                     2                    1039423             9152                2376                True                Running           0
foobar_srv01  ServerMgr                  16328   101         1                    0                     2                    1020586             27989               7906                True                Running           0
foobar_srv01  ServerMgr                  7735    45          1                    0                     2                    1020032             28543               8112                True                Running           0
foobar_srv01  FSMSrvr                    5053    5           0                    0                     26                   1033215             15360               4564                True                Running           0
foobar_srv01  CommOutboundMgr            5516    31          0                    0                     56                   943817              104758              38734               True                Running           0
foobar_srv01  AsgnSrvr                   5224    14          0                    0                     26                   964218              84357               57768               True                Running           0
foobar_srv01  eCommunicationsObjMgr_enu  5670    37          1                    0                     10                   692726              355849              238679              True                Running           0
foobar_srv01  eCommunicationsObjMgr_enu  16116   61          0                    0                     7                    745602              302973              193532              True                Running           0
foobar_srv01  eCommunicationsObjMgr_enu  5619    35          0                    0                     7                    742789              305786              187294              True                Running           0
foobar_srv01  ServerMgr                  458     118         1                    0                     2                    1020686             27889               7537                True                Running           0
foobar_srv01  eProdCfgObjMgr_enu         5462    27          0                    0                     8                    957354              91221               38498               True                Running           0
foobar_srv01  XMLPReportServer           5245    18          0                    0                     26                   956796              91779               37351               True                Running           0
foobar_srv01  eCustomerObjMgr_enu        5446    26          0                    0                     7                    1018852             29723               6423                True                Running           0
foobar_srv01  eChannelObjMgr_enu         5256    17          0                    0                     7                    1018732             29843               6356                True                Running           0
foobar_srv01  EAIObjMgr_enu              5332    21          0                    0                     7                    907893              140682              83965               True                Running           0
foobar_srv01  eCommunicationsObjMgr_enu  5640    36          2                    0                     11                   738753              309822              191577              True                Running           0

As you can see, the presence of PID, component alias and current number of executing tasks are quite useful to have in a single command output. Unfortunately, there is no documentation about the other details.

The command can be used with the additional options for server and for comp as well. For example, to monitor when a server has completely started up, you can either run list procs for server <servername> command and verify all processes are in running state.

Tuesday, March 18, 2014

Load test Siebel 8.1 with Jmeter

The problem

Last year I was asked to help with an issue regarding Siebel File System in the production environment of Siebel 8.1. The environment was installed within a Red Hat Linux, and the Siebel File System was showing issues of locking when the system started receiving a higher volume of requisitions than the normal.
In short, after help of Red Hat administrators, we had identified that the read/write of Siebel Preferences Files (*.spf) were generating a great number of locking that were not being released, provoking the related Siebel Component to stop responding new requests or even crashing. During some crises, it was even necessary to restart the Siebel Server to get things working again.
During initial setup of all environments, initial configuration went with NFS as choice for the Siebel File System and soon it showed poor performance. Due time constraints, it was chosen to go with the Red Hat GFS file system, which initially shown better performance. Of course, since Oracle didn't support GFS about that time, it was not clear that this was a bad design decision or not.

The proposal

It was clear too that we should go back with NFS since it is supported by Siebel on Linux. The problem was, of course, it was having poor performance: in fact the default configuration was so poor that is was not acceptable.
The group decided to give another try to NFS, but we would need to be sure that NFS wouldn't show the same problem. We need a way to test and guarantee that and a load test/stress test was the way to go. Now the problem was to simulate usage of Siebel Preferences Files in the Siebel File System.

Enter Jmeter

Since the company didn't have any tool to simulate web clients, I suggested to go with Jmeter, and the group accepted the proposal.
I started using Jmeter's proxy feature to see how the HTTP requests were send and received from a regular Siebel Web Client in High Interactive mode. We assumed that only doing a log in/log out of the application would be enough to force the Siebel Component to read the spf file, but I was also aware that the result could be messy considering that the High Interactive client is a complicated mix of HTTP, HTML, ActiveX and Java applets.
As you can see below, I was not that wrong:


And that was for only requesting the HTML form, entering with user and password and submitting it. After that, a log out request was made. At total, 20 requests for a simple log in the application.
The major issue was understanding the parameters and their meaning. Some of then were pretty cryptographic:


Since I could not guess their meaning and what should I use as a value for each one of them, creating this test required a series of try-and-error sessions, including monitoring the Siebel Server to see if the log in/log out was actually working as expected.
Once this was successfully created, we also decided that only reading the spf files was not enough: we should force it to be changed. I turned on the proxy feature of Jmeter and tried again.
An easy thing to do was to go to a list applet and swapping positions of two list columns in the Siebel Web Client. Implementing that in Jmeter was even more painful, because the amount of parameters were higher and number of requests too. In total, it was 30 request just to swapping the columns of position after log in and log out of the application! That required even more try-and-error tests, including checking for modification in the related spf and in the application itself.

Lessons learned

Well, Jmeter is a great tool for automating load test/stress test of web applications (and other things too), but for Siebel I would not recommended it.
When I saw that it would be even harder to implement as I initially thought, I opened a ticket for Oracle support asking for documentation about those parameters. Unfortunately, the answer was that those parameters were not documented for end user utilization.
After some researching, I found this article from Alex Amat that shows clearly why is so hard to simulate a Siebel High Interactive client by simply looking at the HTTP request exchanged. The secret for that seems to be the correlation library (represented by the ssdtcorr.dll in Windows), which enables you to simulate the HI client without doing yourself a brain damage. While this also helps with the matter of not having documented the usage of each HTTP parameter it's open to debate and I honestly don't have a opinion about that because I haven't tested the application either (Oracle has a product called Oracle Application Testing Suite which hopefully makes such tasks much easier).
Maybe when using Jmeter with Open UI will be easier to write such tests, but again, this is something to checkin the future but I would not bet on that.

And what about the tests results?

Fortunately, the tests went OK. After reproducing the issue on GFS, the team started setting up the NFSv4 in test environment and a lot of settings were tried on the Red Hat side. The OS specialists were able to find good settings and performance with NFSv4 was not an issue again. Of course, the Jmeter's tests were there to confirm that the same locking issue would not happen with the new Siebel File System.

Saturday, February 8, 2014

REST::Client and invalid TLS certificates

A couple of days ago I had the need to consume data from a REST application to complete a test automation job that I was creating.
While I could do it by simple using curl in a shell, the response was also in JSON, which I would need to process anyway, so I decided to do it with a Perl script using REST::Client and JSON::XS.
Since I was working in a development environment, the TLS certificates were self-signed and thus, not valid.
Looking for an option with REST::Client Pod I was disappointed to find nothing. That should be an easy enough thing to do.
REST::Client is built over LWP::UserAgent, so after some researching I found an option with it that does the job:

my $client = REST::Client->new();
$client->getUseragent()->ssl_opts( verify_hostname => 0 ); 


Simple enough. Also a shame to find nothing about it in Google (well, at least until now :-) ).

Friday, November 1, 2013

Ubuntu, Firefox and all those useless dictionaries

So, you got Ubuntu installed in your computer, made your personal customizations and everything seems to be fine.

Well, that until you start using spell checking with Firefox and/or Thunderbird and discover that you haven't exactly tamed those animals.

For default, Ubuntu seems to install lots of dictionaries for both programs. If it's language settings include, for example, Portuguese and English, you will got spell checking with both languages in Firefox and Thunderbird, but probably not exactly what you want: when you click if your right mouse button in your text in progress, you will discover that English spell checking will take you four different English options, one for a different country. If you're coming from a upgrade or restore a previously used backup, then probably you'll get repeated options.

Fortunately, you can get rid of those country-specific dictionaries, but configuration for it will not be available in Firefox or Thunderbird.

Open up a terminal session and check out the commands output below:

youruser@yourhost:~$ ls /usr/lib/firefox/dictionaries
en_AU.aff  en_AU.dic  en_GB.aff  en_GB.dic  en_US.aff  en_US.dic  en_ZA.aff  en_ZA.dic  pt.aff  pt_BR.aff  pt_BR.dic  pt.dic  pt_PT.aff  pt_PT.dic

This list shows all dictionaries installed to Firefox. If you look for the same thing in the thunderbird folder of Thunderbird e-mail program, you'll see a similar output.

Let's remove them and configure dpkg to not install new files then, but move then to other location instead, which is a nice way to avoid the files to be restored every time you upgrade Firefox and/or Thunderbird.

youruser@yourhost:~$ sudo rm -f /usr/lib/firefox/dictionaries
[sudo] password for youruser:


youruser@yourhost:~$ sudo dpkg-divert --add /usr/lib/firefox/dictionaries 
Creating 'local diversion from /usr/lib/firefox/dictionaries to /usr/lib/firefox/dictionaries.distrib'

Restart Firefox and/or Thunderbird. If you got no dictionary after that, don't worry: just go to Tools -> Complements menu options and install your desired dictionaries. After that, you probably won't need to worry about those settings again until you got a fresh setup of Ubuntu.