add

IIB/Message Broker – Debugging and problem Determination

Trace and Debug Mechanisms.

IIB Debug

Most of the times, we run into various issues either it could be in Production or while QA Defects.

For a Quick and proper root cause analysis,we might need to run various debugging steps to generate the trace/log/intermediary status when problem occurred. Also, For PMR cases, we need to provide the good amount of Debug and problem evidence for a quick resolution from the vendor.

Below are the step by step process to collect the Debugging trace of various Broker run-time processes :

 

 Capturing User and Service level Trace :

use below steps to capture a user or service level trace of a IIB/WebSphere Message Broker message flows at an integration server/execution group level.

Gathering this information before reaching vendor Support will help familiarize you with the troubleshooting process and also could save your time.

 

  • Service Level Trace : 

Service level trace provides more detailed information than that provided by the entries that are written to the O/S  level Syslogs, Event Logs, Trace Node Outputs, or User Trace.

Typically, IBM support uses service trace for debugging problems as it can trace internal calls in addition to the Node/brokers,   Integration Server/execution groups, and all the deployed message flows.

Step : 1    Start trace using below MQSI command :

              mqsichangetrace <NodeName> -t -e <Integration Server> -l debug  -r    -c 100000

step : 2  Trigger the message flow/application to reproduce the problem in Test Environment/Prod environment.

step :3  Stop trace by issuing below command :

               mqsichangetrace <NodeName> -t -e <integration Server> -l  none

Step : 4 Retrieve the trace log for the intended component using the mqsireadlog MQSI command :

               mqsireadlog <NodeName> -t -e <integration Server> -f -o  <fileName>.xml
                  Example : mqsireadlog <NodeName> -t -e <Integration server> -f -o sampleFlowTrace.xml

Step : 5  Format the XML trace file to the readable format using mqsiformat log MQSI command :

 mqsiformatlog -i </../../<fileName>.xml  -o   </../../<fileName><.txt/.log/nothing if unix>
                 Example : mqsiformatlog -i sampleFlowTrace.xml  -o  serviceSampletrace.txt

 

  • User Level Trace :

User trace provides more information than that provided by the entries that are written to the logs. Typically, you use user trace for         debugging your applications, as it can trace brokers, execution groups, and deployed message flows.

Step : 1  Start trace using below command ( this time with switch  -u )

mqsichangetrace <NodeName> -u -e <Integration Server> -f  <MessageFlowName> -l debug -r -c 50000

Step 2 :  reproduce the problem by triggering the message flow.

step 3 :  Stop the trace using the below MQSI command

mqsichangetrace <brokername> -u -e <egroup> -f "<Message Flow Name>" -l none

You can also stop tracing, by going to Operations tab, and right click Message Flow, and click Stop Trace in GUI.

Step :4  Retrieve the trace log for the specified component using below command :

mqsireadlog <brokername> -u -e <egroup> -f -o flowtrace.xml

step :5 Format XML tracefile in user readable format.

mqsiformatlog -i flowtrace.xml -o userflowtrace.txt

 

 startup trace of biphttplistener in IIB/WebSphere Message Broker (WMB) :

Below is the process to collect startup trace of HTTP listener ( biphttplistener component) in IIB.

To trace the startup of the biphttplistener, run the following commands:

1. Make sure the run-time ( broker/node)  is running before enabling the HTTPListener trace override property.

2. run the following command to enable the trace :

mqsichangeproperties brokerName -b httplistener -o HTTPListener -n traceOverrideLevel -v debug

3. Shutdown and restart the IIB node

4. reproduce the problem for trace capture initiation

5. run the command to dump all the component level trace to an XML file.

mqsireadlog NodeName-t -b httplistener -f -o sampleListenerTrace.xml

6. run the following commannd on XML file that was generated in previous step :

mqsiformatlog -i sampleListenerTrace.xml -o sampleListenerTrace.txt

7.  revert back the debug to None to stop trace ( if you forget, it could cause performance overhead/increase in file system)

mqsichangeproperties NodeName-b httplistener -o HTTPListener -n traceOverrideLevel -v none

 

Process to generate Trace of a MQSI command

 

Use this procedure to capture a trace of a IIB/WebSphere Message Broker command.
1.Enable tracing of utility commands by Setting up below UNIX/Windows Environment variable :

SET/EXPORT MQSI_UTILITY_TRACE=debug

2. Set the maximum size of the trace as it could generate vast amount of log data based on command type.

SET/EXPORT MQSI_UTILITY_TRACESIZE=51200  The value is set in kilobytes (recommended is 50MB).

3. Run the mqsi* command(s) that are failing.

4. Disable the trace by reverting the Environment variable by setting its value to “none”

SET/EXPORT MQSI_UTILITY_TRACE=none

5. Retrieve the trace log for the mqsi* command we ran before.
This can be issued against all commands executed in step 3.

mqsireadlog <component> -t -b <mqsicommand> -f -o <mqsicommand>trace.xml

6. Format the xml trace log(s) into text.

mqsiformatlog -i <mqsicommand>trace.xml -o <mqsicommand>trace.txt

The value of ‘<component>’ will changed based on the object of the failing mqsi* command.

You can find more information in the Infocenter/ allReportableEntities object on mqsireportproperties command on which component   name to use.

 

Capturing service level trace for a broker’s HTTP listener

 

1. Start the IIB Node/broker.

 mqsistart <NodeName>

2. Start service level trace.

mqsichangetrace <NodeName> -t -b -l debug

3. Recreate the problem (check for errors in the syslog or event viewer log).

4.  Format the XML log.

mqsireadlog <NodeName> -t -b httplistener -f -o listenertrace.xml

5. Retrieve the trace log for the specified component.

mqsiformatlog -i listenertrace.xml -o listenertrace.log

6. Stop service level trace.

mqsichangetrace <NodeName> -t -b -l none

7.After a PMR is open, submit the file listenertrace.log to IBM Support for problem determination.
( reference : IBM DevWorks tech questions).

 

Procedure in IIB/WMB service level trace for HTTP Listener

Use this procedure in WebSphere Message Broker (WMB) to start a service level trace of the broker’s HTTP Listener for the following scenario:

Your message flow works fine, but the HTTP listener does not seem to be getting the messages from the SYSTEM.BROKER.WS.REPLY queue.

Gathering this information before calling IBM Support will help familiarize you with the troubleshooting process and save you time.

 

Process to collect Trace of Java Classes

Use the following procedure to collect a trace of the Java classes that are loaded by IIB/WebSphere Message Broker at startup.

To collect a trace of the Java classes that are loaded by WebSphere Message Broker at startup, complete the following steps:

    1. 1. Stop your  Node/broker.

 

    1. 2.In the shell environment of your Node/broker, enable the trace by exporting the following environment variable:
       export IBM_JAVA_OPTIONS=-Dibm.cl.verbose=*

 

  1. 3. Start your Node/broker.

The classloading information is written to the broker’s standard output (STDOUT) and standard error (STDERR) files, in the directory $MQSI_WORKPATH$/components/<broker name>/<eg uuid>.

 

Process to Deployment failure trace

Use this procedure to capture a WebSphere Message Broker trace of a failing deploy to a broker.

Gathering this information before calling IBM support will help familiarize you with the troubleshooting process and save you time.

Use the following procedure to capture WMB V7 or V8 deploy traces:
On the broker server:

1. Start tracing of broker agent and execution groups

mqsichangetrace <brokername> -t -b -l debug
mqsichangetrace <brokername> -t -e <egroup> -l debug -r

where <brokername> is the name of your broker while <egroup> is the name of your execution group

2. Deploy and monitor for errors.

3. Stop all tracing when deploy errors occur.

mqsichangetrace <brokername> -t -b -l none
mqsichangetrace <brokername> -t -e <egroup> -l none

Format the traces using the commands below:

4. Retrieve the trace log for the specified component.

mqsireadlog <brokername> -t -b agent -f -o agenttrc.xml

5. Retrieve and format the XML log

mqsiformatlog -i agenttrc.xml -o agenttrc.txt

6. Retrieve the trace log for the specified component.

mqsireadlog <brokername> -t -e <egroup> -f -o <egroup>.xml

7. Format the XML log file

mqsiformatlog -i <egroup>.xml -o <egroup>.txt

8. After a PMR is open, submit the files agenttrc.txt and <egroup>.txt to IBM Support for problem determination.

 

Process to capture DB2 trace

This procedure is used to help determine problems with DB2 access and updates in IIB/WebSphere Message Broker.

Gathering this information before calling IBM Support will help familiarize you with the troubleshooting process and save you time.
Resolving the problem
Use the following procedure to capture a DB2 CLI trace:

1. Append the following stanza to db2cli.ini file under /db2 Install directory/sqllib/cfg

[COMMON]
TRACE=1
TRACEFILENAME=C:\DB2TRC.TXT
TRACEFLUSH=1

2. Reproduce the error.

3. To turn the Trace Off, set Trace=0.

4. After a PMR is open, submit the file DB2TRC.TXT to IBM Support for problem determination.

reference :

IBM Infocenter | IBM Developer Works | IBM tech answers

Note : Please kindly let me know for any correction/change.I’ll be happy to do that.

Written by Ramesh Metta


Leave a Reply

Your email address will not be published. Required fields are marked *

*
*