The problem
Last year I was asked to help with an issue regarding Siebel File System in the production environment of Siebel 8.1. The environment was installed within a Red Hat Linux, and the Siebel File System was showing issues of locking when the system started receiving a higher volume of requisitions than the normal.
In short, after help of Red Hat administrators, we had identified that the read/write of Siebel Preferences Files (*.spf) were generating a great number of locking that were not being released, provoking the related Siebel Component to stop responding new requests or even crashing. During some crises, it was even necessary to restart the Siebel Server to get things working again.
During initial setup of all environments, initial configuration went with NFS as choice for the Siebel File System and soon it showed poor performance. Due time constraints, it was chosen to go with the Red Hat GFS file system, which initially shown better performance. Of course, since Oracle didn't support GFS about that time, it was not clear that this was a bad design decision or not.
The proposal
It was clear too that we should go back with NFS since it is supported by Siebel on Linux. The problem was, of course, it was having poor performance: in fact the default configuration was so poor that is was not acceptable.
The group decided to give another try to NFS, but we would need to be sure that NFS wouldn't show the same problem. We need a way to test and guarantee that and a load test/stress test was the way to go. Now the problem was to simulate usage of Siebel Preferences Files in the Siebel File System.
Enter Jmeter
Since the company didn't have any tool to simulate web clients, I suggested to go with Jmeter, and the group accepted the proposal.
I started using Jmeter's proxy feature to see how the HTTP requests were send and received from a regular Siebel Web Client in High Interactive mode. We assumed that only doing a log in/log out of the application would be enough to force the Siebel Component to read the spf file, but I was also aware that the result could be messy considering that the High Interactive client is a complicated mix of HTTP, HTML, ActiveX and Java applets.
As you can see below, I was not that wrong:
And that was for only requesting the HTML form, entering with user and password and submitting it. After that, a log out request was made. At total, 20 requests for a simple log in the application.
The major issue was understanding the parameters and their meaning. Some of then were pretty cryptographic:
Since I could not guess their meaning and what should I use as a value for each one of them, creating this test required a series of try-and-error sessions, including monitoring the Siebel Server to see if the log in/log out was actually working as expected.
Once this was successfully created, we also decided that only reading the spf files was not enough: we should force it to be changed. I turned on the proxy feature of Jmeter and tried again.
An easy thing to do was to go to a list applet and swapping positions of two list columns in the Siebel Web Client. Implementing that in Jmeter was even more painful, because the amount of parameters were higher and number of requests too. In total, it was 30 request just to swapping the columns of position after log in and log out of the application! That required even more try-and-error tests, including checking for modification in the related spf and in the application itself.
Lessons learned
Well, Jmeter is a great tool for automating load test/stress test of web applications (and other things too), but for Siebel I would not recommended it.
When I saw that it would be even harder to implement as I initially thought, I opened a ticket for Oracle support asking for documentation about those parameters. Unfortunately, the answer was that those parameters were not documented for end user utilization.
After some researching, I found this article from Alex Amat that shows clearly why is so hard to simulate a Siebel High Interactive client by simply looking at the HTTP request exchanged. The secret for that seems to be the correlation library (represented by the ssdtcorr.dll in Windows), which enables you to simulate the HI client without doing yourself a brain damage. While this also helps with the matter of not having documented the usage of each HTTP parameter it's open to debate and I honestly don't have a opinion about that because I haven't tested the application either (Oracle has a product called Oracle Application Testing Suite which hopefully makes such tasks much easier).
Maybe when using Jmeter with Open UI will be easier to write such tests, but again, this is something to checkin the future but I would not bet on that.
And what about the tests results?
Fortunately, the tests went OK. After reproducing the issue on GFS, the team started setting up the NFSv4 in test environment and a lot of settings were tried on the Red Hat side. The OS specialists were able to find good settings and performance with NFSv4 was not an issue again. Of course, the Jmeter's tests were there to confirm that the same locking issue would not happen with the new Siebel File System.
