add

PROD Support Hacks

Productivity Hacks

productivity hacks

Imagine a post major release days where you have pushed lots of new code into the prod servers and when we are on a development support for the release and being flooded with lot of issues.

 

Some times due to a minor or major bugs ( specially when you systems are of full Pub/Subs), then there could be slew of messages to all the systems.
Or may be your integration interface rejected so many payload messages and they are ended in an error queue or a dead letter queue, you may be in a situation to fix the bug and reprocess
all the payloads again.

 

Below are some of the hacks i use normally in my day to day issues. Majority of interfaces talk in XML Messages or fixed format layout.

 

NotePad++ - an Iron-Man suite for a developer

 

60% of the things could be done in NotePad++ ( along with its supported plugins like MIME converters,XML plugins and Data converters) but some times we still are left with few things where
we may need to take the extra help of Unix.

 

Unix - A cockpit with everything

Tip: Most of the people use windows machine for their work and if you need the power of Unix, you can install git bash.  Git Bash is a package that installs Bash, some common bash utilities, and Git on a Windows operating system.

If you have more than 5 files, then you may need perform same set of commands on multiple files. I throw almost everything into below for loop construct and do some other Tasks.

for f in *.xml;  do

  ....

  ...

done

For example, you missed a RfhUtil SaveQ sequence and would like to recreate a sequnce to perform LoadQ operation in RfhUtil or may be you want to load all xml payloads into a queue and saving it back.

throwing the amqsput/amqsputc command inside a for loop does everything for us.

 

for f in *.xml;  do    amqsput TESTQ IB9QMGR < $f ; done

 

iconv command in Unix:

 

if your enterprise ESB has heterogeneous systems like mainframes that has EBCDIC CharSet, Utf-8, cp-1252 etc, then this command is of great help to do the transformation from one CharSet Encoding to another CharSet encoding.

iconv -f utf-8 -t cp1252 < utf-8_file.xml > cp1252_file.xml

Sometimes you may need to convert Utf-16 Little Endian to utf-8 :

iconv -f utf-16LE -t utf-8 < utf-16_file.xml > utf-8_file.xml

 

Load Bunch of payloads from a file where each payload data ends with newline

 

while IFS=  read -r msg
do
   echo $msg | amqsput LOCALQ IB9QMGR
done < xml_payloads.dat

 

Search and Fix Data files

 

There could be scneario's where the end system does not support the utf-8 chars but source system has the default charset as utf-8.

for example : emdash character which is a multi-byte char ( refer this utf-8 char here for more details) looks similar to normal ASCII char hyphen (-)

To remove such chars and replace it with normal hyphens, i loop through each file and do look for hex codes of this multi-byte char ( 0xE2 0x80 0x94 (e28094) or html entity : &#x2014; ) .

if found, i'll use sed command to search and replace with normal hyphen in whole file ( -g switch for global) and then copy the new output to new file.

( Note: we can apply search and replace opertaion on same file with sed -i option but i would like to retain original payload and fixed payload for validation at later point)

 

$ for f in `grep -l $'\xe2\x80\x94' *.xml`; do echo "found emdash char in file $f"; grep -l $'\xe2\x80\x94' $f | xargs sed "s/\xe2\x80\x94/-/g" > "`basename -s .xml $f`_fixed.xml"; done

 

Another example:

 

for f in `grep -l $'\xa0' *.xml`; do echo "found no-break-space char in $f"; grep -l $'\xa0' $f | xargs sed "s/\xa0//g" > "`basename -s .xml $f`_fixed.xml"; done

 

 

CRLF/LF Operations between Windows/Unix Files

 

 

Sometimes, we may need to perform Line operations. following commands will be handy for CRLF/LF switch line operations to avoid cntrl-M ( ^M) chars when we transfer files to unix machines from windows machines.

 sed command to delete a carriage Return (CR) - Unix do not recognize the CR ( Carriage Return). Below command helps in deleting the CR from a file.

sed 's/\r//' input.txt  > output.txt

 sed command to replace a linefeed(LF)

sed ':a;N;$!ba;s/\n//g' input.txt  > output.txt

 

Find UTF-8 multibyte Characters

 

Below grep with regex expression helps you detect any multi-byte data (utf-8 char thats out of range between hexcodes 00-7f) in your file.

grep -lP '[^x\00-\x7f]+' *.xml    

grep -loP '[^x\00-\x7f]+' *.xml

Even we can grep for a specific multi byte char.

For Example:

grep -l $'\xe2\x80\x94' *.xml --- emdash char grep -lP '\xa0' --- No-Break space character (C2 A0)

 

 

Identifying the Utf-8 code point with Utf-8 char

 

Lets say, you received a file on a system ( that has locale of cp1252/US-ASCII/ISO-8859-1) and when broker tries to read the file with 819 charset, it might convert the multibyte char to normal Ascii charset.

For example, emDash character '—'  ( which looks like normal hyphen (-)

if you want to get its Utf-8 code point, you can use unix printf command that gives you the code point value.

$ printf "%x" "'—"
2014

This is \x2014 ( since Windows supports utf-16 by default, it returns the utf-16 hexcode value 2014)

$ printf "%x" "' " ----- 'no-break space' character.
a0

 

XML file manipulations using xmlstarlet

 

xmlstarlet is one useful too to play with complex xml files. Below are some of the most frequent hacks we may need to edit our xml payloads.

To use this tool, download the xmlstarlet binary and add the executable to the PATH variable so that you can run the command from any location.

I have clubbed all the command extractions into one single script that demonstrates the recursive traversing the file.

 

Sample File contents:

 

File:1

<?xml version="1.0" encoding="UTF-8" standalone="yes" ?>
<Stock>
<Record Name="ItemsInv" >
<Items record_id="9999-7777-6666" RecordType="itemdtls" status="husbandary" >
<Item orgtype="drawings" ItemID="111111" Product="pencil" type="HB" >
<ItemDtl>
<Status avail="YES" Quantity="100" SupplyType="Truck" ></Status>
</ItemDtl>
</Item>
<Item orgtype="drawings" ItemID="222222" Product="eraser" type="duster" >
<ItemDtl>
<Status avail="YES" Quantity="101" SupplyType="bag" ></Status>
</ItemDtl>
</Item>
<Item orgtype="Books" ItemID="333333" Product="notebook" type="hardcover" >
<ItemDtl>
<Status avail="YES" Quantity="102" SupplyType="paper" ></Status>
</ItemDtl>
</Item>
<Item orgtype="Books" ItemID="444444" Product="notebook" type="hardcover" >
<ItemDtl>
<Status avail="YES" Quantity="103" SupplyType="paper" ></Status>
</ItemDtl>
</Item>
<Item orgtype="drawings" ItemID="555555" Product="eraser" type="duster" >
<ItemDtl>
<Status avail="YES" Quantity="104" SupplyType="bag" ></Status>
</ItemDtl>
</Item>
</Items>
</Record>
</Stock>

 

 

File 2:

<?xml version="1.0" encoding="UTF-8" standalone="yes" ?>
<Stock>
<Record Name="ItemsInv" >
<Items record_id="5555-4444-3333" RecordType="itemdtls" status="husbandary" >
<Item orgtype="drawings" ItemID="666666" Product="pencil" type="HB" >
<ItemDtl>
<Status avail="YES" Quantity="100" SupplyType="Truck" ></Status>
</ItemDtl>
</Item>
<Item orgtype="drawings" ItemID="777777" Product="eraser" type="duster" >
<ItemDtl>
<Status avail="YES" Quantity="101" SupplyType="bag" ></Status>
</ItemDtl>
</Item>
<Item orgtype="Books" ItemID="888888" Product="notebook" type="hardcover" >
<ItemDtl>
<Status avail="YES" Quantity="102" SupplyType="paper" ></Status>
</ItemDtl>
</Item>
<Item orgtype="Books" ItemID="999999" Product="notebook" type="hardcover" >
<ItemDtl>
<Status avail="YES" Quantity="103" SupplyType="paper" ></Status>
</ItemDtl>
</Item>
<Item orgtype="drawings" ItemID="000000" Product="eraser" type="duster" >
<ItemDtl>
<Status avail="YES" Quantity="104" SupplyType="bag" ></Status>
</ItemDtl>
</Item>
</Items>
</Record>
</Stock>


File 3:

 

 

<?xml version="1.0" encoding="UTF-8" standalone="yes" ?>
<Stock>
<Record Name="ItemsInv" >
<Items record_id="2222-1111-0000" RecordType="itemdtls" status="husbandary" >
<Item orgtype="drawings" ItemID="101010" Product="pencil" type="HB" >
<ItemDtl>
<Status avail="YES" Quantity="100" SupplyType="Truck" ></Status>
</ItemDtl>
</Item>
<Item orgtype="drawings" ItemID="202020" Product="eraser" type="duster" >
<ItemDtl>
<Status avail="YES" Quantity="101" SupplyType="bag" ></Status>
</ItemDtl>
</Item>
<Item orgtype="Books" ItemID="303030" Product="notebook" type="hardcover" >
<ItemDtl>
<Status avail="YES" Quantity="102" SupplyType="paper" ></Status>
</ItemDtl>
</Item>
<Item orgtype="Books" ItemID="404040" Product="notebook" type="hardcover" >
<ItemDtl>
<Status avail="YES" Quantity="103" SupplyType="paper" ></Status>
</ItemDtl>
</Item>
<Item orgtype="drawings" ItemID="505050" Product="eraser" type="duster" >
<ItemDtl>
<Status avail="YES" Quantity="104" SupplyType="bag" ></Status>
</ItemDtl>
</Item>
</Items>
</Record>
</Stock>

 

Sample script that uses xmlstarlet utility to traverse the xml payloads.

 

for f in *.xml ; do 
fileName=$f;
RecordID=$(xml sel -t -v "//Stock/Record/Items/@record_id" $f) ; 
if [ -z "$RecordID" ]; then 
RecordID="--empty--" ; 
fi;

ItemID=$(xml sel -t -v "//Stock/Record/Items/Item/@ItemID" $f | awk 'NF>0 {if (i++) printf ","; printf "%s", $0 } END { printf "" }') ; 
if [ -z "$ItemID" ]; then 
ItemID="--empty--" ; 
fi;
echo ""
echo "file Name: $fileName"
echo "Record ID: $RecordID"

for i in $(echo $ItemID | tr ',' '\n') 
do
itm="$(echo -e "${i}" | tr -d '[:space:]' | tr -d '\n')"
#echo "$itm"
qty=$(xml sel -t -v "/Stock/Record//Items/Item[@ItemID="$itm"]/ItemDtl/Status/@Quantity" $f) ;
if [ -z "$qty" ]; then 
qty="--empty--" ; 
fi ; 
echo "ItemID: $itm - Quantity: $qty"
done
echo ""
done

$ ./report.sh

file Name: file1.xml
Record ID: 9999-7777-6666
ItemID: 111111 - Quantity: 100
ItemID: 222222 - Quantity: 101
ItemID: 333333 - Quantity: 102
ItemID: 444444 - Quantity: 103
ItemID: 555555 - Quantity: 104


file Name: file2.xml
Record ID: 5555-4444-3333
ItemID: 666666 - Quantity: 100
ItemID: 777777 - Quantity: 101
ItemID: 888888 - Quantity: 102
ItemID: 999999 - Quantity: 103
ItemID: 000000 - Quantity: 104


file Name: file3.xml
Record ID: 2222-1111-0000
ItemID: 101010 - Quantity: 100
ItemID: 202020 - Quantity: 101
ItemID: 303030 - Quantity: 102
ItemID: 404040 - Quantity: 103
ItemID: 505050 - Quantity: 104

 

Note if you have an xml payload with namespace, you can use -N switch to specify as many namespaces you need to retrieve data from an xml payload.

 

For example :

 

xml sel -N ns=http://www.dummyns.com/test/namespace/v1 -t -v "//ns:Stock/ns:Items/ns:Item" namespace_sample.xml | awk 'NF>0 {if (i++) printf ","; printf "%s", $0 } END { printf "" }'

 

Sometimes, in interfaces, we may need to extract a certain section of xml

 

Example Payload:

 

<?xml version="1.0" encoding="UTF-8" standalone="yes" ?>
<Stock>
<Record Name="ItemsInv" >
<Items record_id="9999-7777-6666" RecordType="itemdtls" status="husbandary" >
<Item orgtype="drawings" ItemID="111111" Product="pencil" type="HB" >
<ItemDtl>
<Status avail="YES" Quantity="100" SupplyType="Truck" ></Status>
</ItemDtl>
</Item>
<Item orgtype="drawings" ItemID="222222" Product="eraser" type="duster" >
<ItemDtl>
<Status avail="YES" Quantity="101" SupplyType="bag" ></Status>
</ItemDtl>
</Item>
</Items>
</Record>
<totalItems>
<records count="15" amount="3000$"/>
</totalItems>
</Stock>

 

We can extract using the -c Option or we can use an xpath file as shown below :

 

Using Xpath File

Create a file with name :  test.xpath and paste below contents.

<XPath>
(//. | //@* | //namespace::*)[ancestor-or-self::totalItems]
</XPath>

 

run the xmlstarlet command as below :

$ xml c14n --exc-with-comments file1.xml  test.xpath


<totalItems>
<records amount="3000$" count="15"></records>
</totalItems>

 

Using -c Option

 

$ xml sel -t -c //Stock/totalItems file1.xml <totalItems> <records count="15" amount="3000$"/> </totalItems>

These are not the exact ways you can do these things but can be done in many alternate ways.

 

 

Final Caution

 

Please remember to always backup your original payloads before playing with these commands.

Credits: Always indebted to Stackoverflow.

Written by Ramesh Metta


2 thoughts on “PROD Support Hacks”

  • BG Shirdi

    July 6, 2020 at 09:39

    Very useful tips. Thanks.

    Reply
    • Ramesh Metta

      July 6, 2020 at 10:33

      Thanks Shirdi 🙂

      Reply

Leave a Reply

Your email address will not be published. Required fields are marked *

*
*