Sunday, January 22, 2006

A Painful Learning Experience

Over the last week, I had to help an Orion Implementation Consultant fix a problem she was having with one of the products that I look after. Everytime she tried to access the application, an exception was thrown which produced a message "Authorisation Failed. Access Denied".

A bit of background........

One of our important clients is currently in the process of upgrading their IT infrastructure, this also means an upgrade of Concerto from 5.2 to 6.2.3. That was done and she installed Soprano Medical Documents and Soprano Medical Records as well. Uh-oh, it seemed that the install of Concerto 6.2.3 may have broken the Soprano products.

This was brought to my attention the Thursday before last. She showed me the problem, and sure enough it was quite clear that the products failed to work. I checked the code where the exception was thrown, and the code was correctly written but the cause of the problem in this scenario was unknown. After much thinking I could only conclude that the problem was with Concerto, so I advised her that the upgrade from Concerto 5.2 to 6.2.3 was the problem. She took this onboard and approached the Concerto team. For two days she did not get back to me regarding the problem, so I assumed the problem lay with Concerto.

On Monday, the team leader for Concerto emailed back his findings and said that the exception was generated by the Soprano products, so the problem lay with Soprano instead of Concerto. D'oh, back to square one. Without any clue as to how to proceed, I advised her to do a full clean install of both products on the client's site and on a VM used by PSG. Both environments were approximately the same, with the same products versions installed and configurations applied. Strangely enough, the problem still persisted on the client's enviroment, yet the PSG environment worked. This clearly indicated that there was no problem with the compatibility between Concerto 6.2 and the Soprano products.

Wednesday, the Concerto team leader and myself used Fiddler to check the HTTP traffic between Concerto and Soprano. We didn't find anything odd and the Concerto team were unable to offer any insight to the problem. Resigned in defeat, I went back to double check the Concerto configurations between the client and the PSG VM. No differences came up, especially in the security settings.

Thursday, we noticed that the Java VM versions were different. On the PSG VM, the version was 1.3.1_08. On the client site, the version was 1.3.1_16. Based on past experience, discrepencies like that may break products. First of all we tried installing 1.3.1_16 on the PSG VM. This didn't break the product. On the client site, we rolled back to 1.3.1_08. This did not fix the original problem. So it seemed that Java VM versions was not the cause of the problem.

Friday, I went through the Fiddler traces again on the client's enviroment and noticed something very odd. Apparently the session id's assigned to the requests were all different for every request sent to the server. Then I ran Fiddler traces on the PSG VM. Sure enough, the session id's were identical. BINGO. The different session id's were causing the problem. The question was: why are there different session id's for every request?

We figured that the client must have a special security setup with their browser. Maybe the browser was not set to accept cookies, or the security level was too high. We compared browser settings, and they were different; the security setting on the client site was too high. But when we lowered it to be identical to the PSG VM browser, the problem still remained. Finally with all avenues exhausted, I tried typing a different URL for Concerto in the browser address bar.

What followed was a realisation that hit me like a kick to the teeth. Apparently we were all accessing Concerto through the URL: http://localhost/concerto. However the server base for the Soprano products was set to http://servername/concerto. When Concerto sends a request, the server cannot determine where the request came from. It treats the servername and localhost as different machines, even if the servername is the same as the localhost machines. This explains the change in session id's previously. What I typed in the browser address bar to access Concerto was http://servername/concerto. I tried accessing the Soprano products, and voila, the problem was fixed.

After all that pain and futile exercise, the problem was caused by the wrong address in the browser!!!!!!!! So the lesson is: do not use localhost, it is evil!!!!! Use the servername or an ip address instead!

No comments: