What to do if a bug has leaked into production?

The application which your team had tested was released into production. A day after the release, you get a mid-night call, that a priority 1 issue has been reported in the production environment. What do you do in such a scenario?

First of all do not panic. You need to remain calm and composed. There could be different reasons due to which an issue can occur:

• Application is not deployed or configured correctly
• Incorrect data in production database is causing it
• Due to incorrect understanding by end users, the feature is reported as a bug
• If none of the above, then it could be a bug in the application.
  
  Broadly, your response to such a situation can be divided in two broad categories. One is the immediate corrective action which needs to be taken to minimize the impact of this issue, and second is the root cause analysis for identifying that why it happened and how we can ensure that similar issue does not happen again.
  
  a) Immediate corrective action:
  Try to collect all the required information regarding the issue found in production.
  
  - In which particular scenario/condition is the problem happening?
  - Is the problem happening consistently in production or is it intermittent?
  - What percentage of end users is getting impacted by this issue?
  - Is this problem corrupting or losing data which is not recoverable? If yes, what can be done to minimize the data loss?
  - Can this issue be replicated in QA environment?
  
  b) Root cause analysis and fixing the process:
  One the issue is identified, we need take the following steps so that similar issue does not happen again:
  - The QA team tries to simulate the problematic scenario in QA environment. It works closely with Business team, development team and production support team to recreate the scenario.
  - If the resolution of the issue requires a patch release, then start planning to test the patch release.
  - To identify the root cause of the issue, perform causal analysis.
  - Identify corrective actions required so that similar issue does not happen again.
  - Implement corrective action
  - Be proactive and keep all stakeholders informed at each stage.

The above steps will help in making sure that you do not loose the confidence of the stakeholders.

Value addition using Freeware automated tools

Many testing consultants are proud of using and recommending expensive licensed automated testing tools which are provided by big software vendors. The IT companies may get impressed with such testing tools provided by the big software vendors. However, in this approach a major chunk of the Testing budget of these IT companies will be exhausted in procuring the expensive testing tool and very little budget would remain for actual automation of the test cases.

As testing consultants, we need to ensure that we recommend a testing strategy, in which the total cost of the automation solution (including the cost of testing tool) is minimal. This can be achieved by evaluating and recommending freeware testing tools. There are several freeware testing tools which are available for functional, performance, security testing etc.

Many testers are sometimes reluctant to use the freeware tools as they feel that there is a risk associated with the freeware tool as it has limited features. A good approach to mitigate this risk is to create a Proof Of Concept (POC) by automating few test cases using the freeware tool. This will mitigate the risk associated with using the freeware tool. The POC should be demonstrated to the stakeholders to get the required approval.

Another observation is that freeware tools have almost the same set of features as licensed tools, but the freeware tools supports limited platforms or protocols. So, depending upon the Application Under Test (AUT) requirement, we should evaluate and select an appropriate freeware testing tool.

By using the above approach, you can make direct saving in the IT budget of your client.

Reliability testing – Detecting and fixing memory leaks

Last month, I encountered an interesting situation regarding memory leak in production. I would like to share my experience of how we were able to diagnose and fix the problem.

While working for a client, we released a new version of the application. A strange behavior was observed. On every Monday, the application used to give “Out of Memory exception”. On discussion with the users it was clear that the load on the system was 50 – 60 % higher on Monday. So, definitely there was a memory leak in the system which was causing the out of memory exception.

The client had not invested in any dedicated QA environment. So, it was not possible to simulate or diagnose the problem. We started looking for a freeware tool which could help us diagnose the system in the production environment.

We finally used JConsole which comes along with JDK 5.0. This is a freeware which can be used for monitoring, the memory utilization with time. It clearly shows how many threads are running. It shows at which time the Garbage collection is happening. In fact, it has a provision where you can use it to simulate Garbage Collection.

Using this tool, we could locate the probable portion of the code where the problem could be. Then we did a thorough code walk thru of that portion of the code. We found that the calls to certain functions were being made by starting a new session. This problem was there at multiple places in the application. After making the changes, we could observe the difference in JConsole tool.

More details of JConsole are available at the following location:
http://java.sun.com/developer/technicalArticles/J2SE/jconsole.html

Handling pressure on Testing team

Apart from keeping up to date with the latest tools and technologies, a tester has to bear tremendous pressure from other groups. When a tester is testing an application or product, there is constant pressure from the business users to release it at the earliest. Even if the there is slippage in schedule by the development team, the overall release date is not changed which puts extra pressure on the testing team. The development team, the business users, the senior management, everyone wants to push the release to production. It seems that the whole world except the testing team wants to push the release out to production at the earliest.

What should the testing manager do in such a scenario? The testing manager is the only person who is standing alone to stop/delay the release until the release is of acceptable quality. A testing manager should act as a gatekeeper of the release. If he/she succumbs to the pressure from other groups, then later on the accountability of any post-release defects comes on the testing team.

How can the testing manager handle such pressure situation? The testing manager should rely on well defined testing process. During the test planning stage itself, the testing plan and process should be clearly documented. The test plan and process should be got approved from all stakeholders. This helps in handling the pressure situation. The test manager is then viewed as the person who is implementing the agreed upon process, rather than an individual person who is delaying/stopping the release.

I as a testing manager have faced several such high pressure scenarios, and this technique has helped a lot to disperse the pressure to other stakeholders.