multiple ORACLE Documaker instances reading single jms/hornetq issue.

Documaker's snail waxed zero size PDF's on our production environment, and it took me a day to figure it out, whats wrong with it! :(

below is the setup we had in our production environment.

whats happening here is, that two Documaker instances are reading from single/same jms/hornetq queue, and as Documaker does not have any identifier mechanism for soap messages, as it seems both tries to hold the message and somehow destroys the soap message arrived from ETPM in jms queue, so both Documaker instances endup with generating ZERO size PDF's.

the workaround is to run single Documaker instance per jms/hornetq queue. But this way you d'nt have real time fail-over for Documaker, you have to do manual switching in case of failure!

cheers.

hornetQ high available cluster, step by step

well, unlike jboss cluster, hornetq cluster work on shared/common storage space that could be a SAN shared disc or even NFS mount, though best option is SAN disc, you must keep in mind that both cluster members will not write simultaneously on shared disc as oracle RAC do, but they will use it to validate master/live server health, soon master/live server die, slave/backup server will replace it and from passive started state will move to active started state and all persistent messages of queues will be served from slave/backup server. This failover process is automatic, even when master/live server will come back online, the slave/backup server will again set itself automatically to passive started mode and will leave master/slave to take the control of queues and client requests/responses.

hornetQ offer two type of cluster solution i.e. load-balance plus failover and just failover not load-balance, in our case we is used just failover not load-balance. i have little edited the official hornetq cluster diagram, below, because for no good reason they made shared storage in two separate places and then the storage show no link among each other, while actually it must be single storage with link to both Live and Backup server.

for more details you must consult official guide or check below my step by step for hornetQ cluster installation.

the startup script was made by our development team as they like to pass JVM arguments inside script, though yo can use default run.sh which is more easy to understand.

find below step by step for hornetQ high available cluster.

Download

Cheers

ETPM, Documaker, JMS, LDAP, Database, Conceptual and abstract workflow diagram

ladies, if you are unable to understand how ORACLE ETPM, ORACLE Documaker, JMS/HornetQ, LDAP/OpenDS and Database/Exadata should work together, here is the lazy drawn diagram, i hope you will get an idea about what i have been implementing recently :)

you may consider it as ETPM application data center conceptual design as well;

Cheers

step by step ORACLE Documaker 12.0 installation & configuration on Linux 64x

believe me or not, but this one is gem, it is almost impossible to understand installation steps from ORACLE's bleeding ass Documaker installation manuals, i took the courage to prepare it, you must contact me if you get confused at some step!

here we go:



step 01: download V29335-01 & V29336-01 from edelivery.oracle.com

step 02: download jre1.6_0_32 from oracle.com

step 03: copy all downloads to /u01/app/documaker/documaker-source

step 04: install JRE 32bit

[cissys@documaker-host documaker-source]$ cd /u01/app/documaker/documaker-source
[cissys@documaker-host documaker-source]$ chmod +x jre-6u32-linux-i586.bin
[cissys@documaker-host documaker-source]$ ./jre-6u32-linux-i586.bin
[cissys@documaker-host documaker-source]$ mv jre1.6.0_32/ ../

step 05: install Documaker engine

[cissys@documaker-host documaker]$ mkdir rel120p01
[cissys@documaker-host documaker]$ mkdir dm-env
[cissys@documaker-host documaker]$ cd documaker-source
[cissys@documaker-host V29335-01]$ chmod +x * -R
[cissys@documaker-host V29335-01]$ ./setuprterp120p01b15380.lnx

* * * Documaker RP 12.0 (Turnover Level p01b15380) (Type RTE) Installation/Update * * *
Loading, Please Wait ...




Please select an Installation/Update Directory:
Press to accept default:
?>/u01/app/documaker/rel120p01


Install Documaker RP 12.0 (RTE) to ?
{y or n}?>y


Updated Base Components ...


...
RTE Installation Complete.

[cissys@documaker-host documaker-source]$ cd V29336-01
[cissys@documaker-host V29336-01]$ chmod +x *
[cissys@documaker-host V29336-01]$ ./setuprteids023p00b713.lnx

* * * Docupresentment 2.3 (Patch Level p00b713) (Type RTE) Installation/Update * * *
Loading, Please Wait ...




Please select an Installation/Update Directory:
Press to accept default:
?>/u01/app/documaker/dm-env


Install Docupresentment 2.3 (RTE) to ?
{y or n}?>y


Install/Upgrade in progress.  Please wait...
Updated Base Components ...


Install/Upgrade Base Components Complete.


Where is your 32-bit Java Run-time (JRE) Location?
 ( Example:  /opt/jdk1.6.0_24_x86/jre )
?>/u01/app/documaker/jre1.6.0_32
JRE Version=java version "1.6.0_32"


Option for WatchDog Error SMTP Email Notifications:
Do you want to configure this option now?
  (Note: You can manually configure it later in the log4j logconf.xml file)
{y or n}?>n
Continuing...
Installing Setup Files ...
Please Wait ...
New Installation Components Installed ...


RTE Installation Complete.

[cissys@documaker-host V29336-01]$ ./sharedobjs120p01b15380.lnx

* * * Documaker RP Shared Objects Bridge 12.0 (Turnover Level p01b15380) Installation/Update * * *
Loading, Please Wait ...




Please identify your Docupresentment Installation/Update Directory:
Note: Docupresentment directory must already exist.
Press to accept default:
?>/u01/app/documaker/dm-env/docserv


Install Docupresentment Documaker RP Shared Objects Bridge 12.0 (Turnover Level p01b15380) to ?
{y or n}?>y


Installation/Update in progress.  Please wait...
Updated Base Components ...


Installing Setup Files ...
Setup files detected (docserv.ini|docserv.xml)
Overwrite setup files?{y or n}?>y
Overwrite in progress...
Please Wait ...
New Documaker RP Shared Objects Bridge Installation Components Installed ...


Running logconfconvert.sh
Results:


logconf.xml.new
Setting up docserv.xml to use jre located at ...


Installation Complete.
step 06: start/stop documaker

[cissys@documaker-host docserv]$ cd /u01/app/documaker/dm-env/docserv
[cissys@documaker-host docserv]$ ./docserver.sh status/start/stop

step 07: configuration of Documaker environment.

[cissys@documaker-host docserv]$ cd /u01/app/documaker/dm-env/docserv
[cissys@documaker-host docserv]$ cp ../../documaker-source/dm-env_orig/dm-env/docserv/docserv.xml 
[cissys@documaker-host docserv]$ cp ../../documaker-source/dm-env_orig/dm-env/docserv/logconf.xml 
[cissys@documaker-host docserv]$ cp ../../documaker-source/uat_orig/dm-env/docserv/dap.ini .
[cissys@documaker-host docserv]$ cp ../../documaker-source/uat_orig/dm-env/docserv/app.ini .
[cissys@documaker-host uat]$ vi /u01/app/documaker/dm-env/mstrres/appdir/app.ini


cheers

step by step ROCKEYE configuration to test JMS (HornetQ) and sending message to Documaker

yumm yumm, after installing JMS/HornetQ and ORACLE's Documaker you would like to test the queues and want to see if messaging between Documaker and HornetQ is working?

Chris introduce me to ROCKEYE tool which is pretty cool tiny desktop application, it helps you to validate your HornetQ and Documaker configuration...though on Documaker part the JMS queues are not properly implemented so you might get some strange harmless warnings and errors :) but you can keep your eyes close on them!

you can check below step by step how to configure ROCKEYE on your desktop.

Download

Cheers

step by step tomcat on ETPM App Server

ok, i know it has been overwritten, but i had to dump it somewhere and blog is the best place to leave my ETPM snail wax! :)


step 01: [amadm@etpm-hostname ~]$ cd /opt/tomapp  --- tomcat installation base directory

step 02: download latest tomcat release from http://tomcat.apache.org/download-70.cgi

step 03: copy apache-tomcat-7.0.27.tar.gz to /opt/tomapp

step 04: verify if downloaded file is correct.

[amadm@etpm-hostname  ~]$ md5sum apache-tomcat-7.0.27.tar.gz
486769ec1094fb45f000d30f237b3643  apache-tomcat-7.0.27.tar.gz

now compare the value with http://www.apache.org/dist/tomcat/tomcat-7/v7.0.27/bin/apache-tomcat-7.0.27.tar.gz.md5
if both values are same, it mean download is correct.

step 05: [amadm@etpm-hostname  ~]$ tar xvf apache-tomcat-7.0.27.tar.gz

step 06: create link:

[amadm@etpm-hostname  ~]$ ln -s apache-tomcat-7.0.27 apache-tomcat

step 07: start & stop tomcat

[amadm@etpm-hostname bin]$ cd /opt/tomapp/apache-tomcat/bin/
[amadm@etpm-hostname bin]$ ./startup.sh
[amadm@etpm-hostname bin]$ ./shutdown.sh

Cheers

ETPM & external LDAP conceptual diagram and configuration's step by step.

so, does it burns your ass when you reset/initalSetup.sh your ETPM (Enterprise Taxation and Policy Management) and all your newly added external LDAP configurations disappears!

hmm, and you need some sort of solution? you want it to be permanently stored in ETPM configs, rather then you have to reconfigure every time you reset/refresh/build your environment? ok, here is the ice-assing step by step and a pathetic conceptual diagram.

step 01: [cissys@YOUR-ETPM-HOST ~]$ cd /u01/app/etpm/TEST/templates

step 02: [cissys@YOUR-ETPM-HOST templates]$ cp config.xml.template config.xml.template.orginal

step 03: [cissys@YOUR-ETPM-HOST templates]$ vi config.xml.template

[ADD FOLLOWING LINES ACCORDING TO LDAP SERVER CONFIGURATION PROVIDED]

step 04: [OPTIONAL] run initialsetup.sh to confirm if LDAP configuration remains after envrionment rest.

 Cheers

Error in invoking target 'links proc gen_pcscfg procob rtsora' of makefile

boy o boy, if you are getting error during ORACLE client installation especially on Redhat Linux 5.5, and the error is somewhat like:

"Error in invoking target 'links proc gen_pcscfg procob rtsora' of makefile '/soft/oracle/app/cliente/product/11.2.0/precomp/lib/ins_precomp.mk'."


or by any means you are seeing something "rtsora" in error message it means you must unset your environment variables especially regarding to COBOL, because ORACLE does mind these variables setting before actually ORACLE's client installation.

we had the same issue at our ORACLE ETPM (Enterprise Taxation and Policy Management) server, because we actually set our ~/.bash_profile and declared $COBDIR before even actually installing COBOL OR ORACLE client.

Solution:

Prior to doing the Client install:
1.  Unset the following variables:
COBDIR, COBLIB, COBPATH, LD_LIBRARY_PATH, ORACLE_PATH

2.  Remove the Cobol and Oracle client bin directories from your PATH variable:

Reinstall on clean Client Home.

Cheers

The root LoggedException was: Error setting oracle NLS Date format

Ok baby! you are receiving NLS Format error in ETPM or CC&B logs? and you want to know whats going on with your Application Server? ok ok... here we go!

ORACLE Enterprise Taxation and Policy Management has bug i.e. once it lost connection to database server, unable to reset it seemingly less, and during this time when it tries to recover the connection between Application Server & Database server, it start giving HTTP-500 error on user's browser.


As you will continue to receive NLS Format error in logs, while it has nothing directly related to connectivity issue you will remain confuse, in our case we had the same issue, but later ORACLE informed us that they are working on this bug. I hope it will be fixed pretty soon!


in our case the connect drop was caused due to our CISCO PIX/Firewall server, it had some rules which were causing connection drop between Application Server & Database server.


After eliminating this connection drop issue, we were able to work smoothly and waiting for ORACLE to produce bug fix!


cheers