Sunday, December 19, 2010

Tomcat Application Vertical Cluster

Application Cluster

Packages Needed

1. 1. jdk-1_5_0_19-linux-i586.bin

(http://www.oracle.com/technetwork/java/javase/downloads/index-jdk5-jsp-142662.html)

2. 2.. apache-tomcat-5.5.27.tar.gz

(http://archive.apache.org/dist/tomcat/tomcat-5/v5.5.27/bin/apache-tomcat-5.5.27.tar.gz)

3. 3. httpd-2.2.3-11.el5_1.3 (RHEL DVD)

4. 4. httpd-devel-2.2.3-11.el5_1.3.i386.rpm (RHEL DVD)

5. 5.tomcat-connectors-1.2.28-src.tar.gz (http://archive.apache.org/dist/tomcat/tomcat-connectors/jk/source/jk-1.2.28/tomcat-connectors-1.2.28-src.tar.gz)

1. Install JDK


1.Copy jdk file to /tmp

2.[root@localhost]# chmod 755 jdk-1_5_0_19-linux-i586.bin

3.[root@localhost]#./ jdk-1_5_0_19-linux-i586.bin

Accept license agreement by typing “yes

A new directory will be created on /tmp directory with name jdk1.5.0_19

Copy the jdk1.5.0_19 directory to /usr/local

4. .[root@localhost]# cp -rv jdk1.5.0_19 /usr/local

Create a symlink of /usr/local/jdk1.5.0_19 to /usr/loca/java

5.[root@localhost]#ln –s /usr/local/jdk1.5.0_19 /usr/loca/java

Open /etc/profile add these lines

5.[root@localhost]#vi /etc/profile

PATH=/usr/local/java/bin:$PATH

JAVA_HOME=/usr/local/java

export PATH USER LOGNAME MAIL HOSTNAME HISTSIZE INPUTRC JAVA_HOME

Log out the session and login again

Type echo $JAVA_HOME ,will show /usr/local/java

2. Install Tomcat

Create a system user with name “tomcat”

1.[root@localhost]#useradd –r tomcat

Create four directories /app1,/app2 ,/app3 and /applog

2.[root@localhost]#mkdir /app1 /app2 /app3 /applog

Change ownership of those directories to “tomcat”

3.[root@localhost]#chown tomcat:tomcat /app1 /app2 /app3 /applog -R

We will install three instances of tomcat for load balanced cluster, for that we need three directories /app1,/app2 and /app3

All log files related to application goes to /applog directory

Unzip and untar the package

4.[root@localhost]#tar -xzvf apache-tomcat-5.5.27.tar.gz

apache-tomcat-5.5.27 directory will be created after executing the command

Copy all the contents from apache-tomcat-5.5.27 to application directories

5.[root@localhost]#cd apache-tomcat-5.5.27

6.[root@localhost]#cp –rv * /app1

7.[root@localhost]#cp –rv * /app2

8.[root@localhost]#cp –rv * /app3

*.Change tomcat default log location from /app1/logs/ to /applog/app1

Create three directories inside /applog

9.[root@localhost]#cd /applog

10.[root@localhost]#mkdir app1 app2 app3

11.[root@localhost]#chown –R tomcat.tomcat /applog

12.Edit /app1/bin/catalina.sh

Change "$CATALINABASE"/logs/catalina.out path to /applog/app1/catalina.out

Do the same for remaining two instances

*.Disable all logging for application to increase tomcat performence

Rename /app1/conf/logging.properties to /app1/conf/logging.properties.org

[root@localhost]#mv /app1/conf/logging.properties /app1/conf/logging.properties.org

Do the same for remaining two instances

3. Install Apache

Install packages for apache

[root@localhost]#rpm -ivh apr-devel-1.2.7-11.i386.rpm

[root@localhost]#rpm -ivh apr-util-devel-1.2.7-7.el5.i386.rpm

[root@localhost]#rpm -ivh httpd-2.2.3-11.el5_1.3.i386.rpm

[root@localhost]#rpm -ivh httpd-devel-2.2.3-11.el5_1.3.i386.rpm

4. Install Tomcat JK connector

we have to give apache extension binary file path before compiling JK connector

[root@localhost]#which apxs

The command will give path of apxs

/usr/sbin/apxs

[root@localhost]#tar -zxvf tomcat-connectors-1.2.28-src.tar.gz

[root@localhost]#cd tomcat-connectors-1.2.28-src/native

[root@localhost]#./configure --with-apxs=/usr/sbin/apxs

[root@localhost]#make

After make the mod_jk.so will be created under apache-2.0 directory

Copy mod_jk.so module to /etc/httpd/modules directory

[root@localhost]#cd apache-2.0/

[root@localhost]#cp mod_jk.so /etc/httpd/modules/

4. Install Application and Configure Cluster

Copy application directories to /app1/webapps , /app2/webapps and /app3/webapps

For example ADMINAPP and USERAPP

[root@localhost]# cp –rv ADMINAPP USERAPP /app1/webapps

[root@localhost]#cp –rv ADMINLIC LICUSER /app2/webapps

[root@localhost]#cp –rv ADMINLIC LICUSER /app3/webapps

Copy necessary jar files for connection pooling to tomcat instances

[root@localhost]#cp commons-pool-1.4.jar mysql-connector-java-3.0.11-stable-bin.jar commons-dbcp-1.2.2.jar /app2/common/lib

Edit configuration files for tomcat

[root@localhost]#vi /app1/conf/server.xml

1. Comment HTTP connector tag

Do the same for remaining two instances

2. Change tomcat shutdown port

vi /app1/conf/server.xml

<Server port=""8005" shutdown="SHUTDOWN">

vi /app2/conf/server.xml

<Server port="8105" shutdown="SHUTDOWN">

vi /app3/conf/server.xml

<Server port="8205" shutdown="SHUTDOWN">

3.Add JVM route to Catalina engine

vi /app1/conf/server.xml

<Engine name="Catalina" defaultHost="localhost"jvmRoute="app1">

vi /app2/conf/server.xml

<Engine name="Catalina" defaultHost="localhost" jvmRoute="app2">

vi /app3/conf/server.xml

<Engine name="Catalina" defaultHost="localhost"jvmRoute="app3">

4.Add Context and Resources to tomcat

vi /app1/conf/server.xml

Add the lines indicated bold in Host tag

<Host name="localhost" appBase="webapps"
unpackWARs="true" autoDeploy="true"
xmlValidation="false" xmlNamespaceAware="false">

<Context path="" docBase="USERAPP" debug="0" reloadable="true" crossContext="true">
<Resource
type="javax.sql.DataSource"
auth="Container"
name="jdbc/MyDB"
factory="org.apache.commons.dbcp.BasicDataSourceFactory"
maxActive="20"
maxIdle="5"
maxWait="1000"
username="db_username"
password="db_password"
driverClassName="com.mysql.jdbc.Driver"
url="jdbc:mysql://127.0.0.1:3306/db-name?autoReconnect=true"
removeAbandoned="true"
removeAbandonedTimeout="60"
logAbandoned="true"
/>
</Context>
<Context path="USERAPP" docBase="USERAPP" debug="0" reloadable="true" crossContext="true">
<Resource
type="javax.sql.DataSource"
auth="Container"
name="jdbc/MyDB"
factory="org.apache.commons.dbcp.BasicDataSourceFactory"
maxActive="20"
maxIdle="5"
maxWait="1000"
username="db_username"
password="db_password"
driverClassName="com.mysql.jdbc.Driver"
url="jdbc:mysql://127.0.0.1:3306/db-name?autoReconnect=true"
removeAbandoned="true"
removeAbandonedTimeout="60"
logAbandoned="true"
/>
</Context>

<Context path="ADMINAPP" docBase="ADMINAPP" debug="0" reloadable="true" crossContext="true">
<Resource
type="javax.sql.DataSource"
auth="Container"
name="jdbc/TestDB"
factory="org.apache.commons.dbcp.BasicDataSourceFactory"
maxActive="20"
maxIdle="5"
maxWait="1000"
username="db_username"
password="db_password"
driverClassName="com.mysql.jdbc.Driver"
url="jdbc:mysql://127.0.0.1:3306/db-name?autoReconnect=true"
removeAbandoned="true"
removeAbandonedTimeout="60"
logAbandoned="true"
/>
</Context>

Do the same for remaining two instances

5. Enable Session replication and Clustering

vi /app1/conf/server.xml

Uncomment Cluster tag

<Cluster className="org.apache.catalina.cluster.tcp.SimpleTcpCluster"

managerClassName="org.apache.catalina.cluster.session.DeltaManager"

expireSessionsOnShutdown="false"

useDirtyFlag="true"

notifyListenersOnReplication="true">

……….striped…….

<ClusterListener className="org.apache.catalina.cluster.session.ClusterSessionListener"/>

</Cluster>

Do the same for remaining two instances

6.Change Multicast port for Cluster listener

Edit Receiver tag in Cluster and change tcpListenPort

vi /app1/conf/server.xml

<Receiver className="org.apache.catalina.cluster.tcp.ReplicationListener"

tcpListenAddress="auto"

tcpListenPort="4001"

tcpSelectorTimeout="100"

tcpThreadCount="6"/>

vi /app2/conf/server.xml

<Receiver
className="org.apache.catalina.cluster.tcp.ReplicationListener"

tcpListenAddress="auto"

tcpListenPort="4002"

tcpSelectorTimeout="100"

tcpThreadCount="6"/>

vi /app3/conf/server.xml

<Receiver
className="org.apache.catalina.cluster.tcp.ReplicationListener"

tcpListenAddress="auto"

tcpListenPort="4003"

tcpSelectorTimeout="100"

tcpThreadCount="6"/>

7. Change AJP 13 Protocol port for tomcat

vi /app1/conf/server.xml


<!-- Define an AJP 1.3 Connector on port 8009 -->

<Connector port="8009"

enableLookups="false" redirectPort="8443" protocol="AJP/1.3" />

vi /app2/conf/server.xml

<!-- Define an AJP 1.3 Connector on port 8009 -->

<Connector port="8109"

enableLookups="false" redirectPort="8443" protocol="AJP/1.3" />

vi /app3/conf/server.xml

<!-- Define an AJP 1.3 Connector on port 8009 -->

<Connector port="8209"

enableLookups="false" redirectPort="8443" protocol="AJP/1.3" />

7. Add cluster parameter to Application’s web.xml

/app1/webapps/USERAPP/WEB-INF/web.xml

/app2/webapps/USERAPP/WEB-INF/web.xml

/app3/webapps/USERAPP/WEB-INF/web.xml

/app1/webapps/ADMINAPP/WEB-INF/web.xml

/app2/webapps/ADMINAPP/WEB-INF/web.xml

/app3/webapps/ADMINAPP/WEB-INF/web.xml

add at the end of the file.

Install mysql jdbc driver

cp /app1/common/lib/mysql-connector-java-3.0.11-stable-bin.jar /usr/local/java/jre/lib/ext/

Create a worker file for tomcat instances

vi workers.properties

workers.tomcat_home=/app1

workers.java_home=/usr/local/java

ps=/

worker.list=loadbalancer,jkstatus

worker.app1.port=8009

worker.app1.host=127.0.0.1

worker.app1.type=ajp13

worker.app1.lbfactor=1

worker.app1.socket_timeout=1800

worker.app1.socket_keepalive=1

worker.app2.port=8109

worker.app2.host=127.0.0.1

worker.app2.type=ajp13

worker.app2.lbfactor=1

worker.app2.socket_timeout=1800

worker.app2.socket_keepalive=1

worker.app3.port=8209

worker.app3.host=127.0.0.1

worker.app3.type=ajp13

worker.app3.lbfactor=1

worker.app3.socket_timeout=1800

worker.app3.socket_keepalive=1

worker.loadbalancer.type=lb

worker.loadbalancer.balance_workers=app1,app2,app3

worker.jkstatus.type=status

Copy the worker file to /etc/httpd/conf.d/

cp workers.properties /etc/httpd/conf.d/

Configure Apache

vi /etc/httpd/conf/httpd.conf

In Main Server configuration add these lines

LoadModule jk_module modules/mod_jk.so

JkWorkersFile "/etc/httpd/conf.d/workers.properties"

JkLogFile "/etc/httpd/logs/mod_jk.log"

JkLogStampFormat "[%a %b %d %H:%M:%S %Y]"

JkRequestLogFormat "%w %V %T"

JkLogLevel info


JkShmFile logs/jk.shm

Create a virtual Host listening on all interfaces in 80 port

ServerAdmin root@lms.icaps.com

#DocumentRoot /var/www/html

ServerName lms.icaps.com

ErrorLog logs/lms.icaps.com-error_log

CustomLog logs/lms.icaps.com-access_log common

JkMount jkstatus

Order deny,allow

Deny from all

Allow from 127.0.0.1

Order deny,allow

Deny from all

Order deny,allow

Deny from all

Order deny,allow

Deny from all

Order deny,allow

Deny from all

Order deny,allow

Deny from all

Order deny,allow

Deny from all

JkMount / loadbalancer

JkMount /* loadbalancer

JkMount /ICMSUSER loadbalancer

JkMount /USERAPP/* loadbalancer

JkMount /ADMINAPP loadbalancer

JkMount /ADMINAPP/* loadbalancer

Create a Script for starting Application Cluster with name “tomcat”

#!/bin/sh

############################################

# Startup script for Tomcat Servlet Engine #

# Author : Ansil Hameed #

# Last Modified : 28/10/2010 #

############################################

# User under which tomcat will run

TOMCAT_USER=tomcat

APP1_DIR="/app1"

APP2_DIR="/app2"

APP3_DIR="/app3"

RETVAL=0

. /etc/init.d/functions

# start, debug, stop, status and flush-log functions

flush(){

LOG_DATE=`date +%d-%m-%y`

cp /applog/app1/catalina.out /applog/app1/catalina.out-${LOG_DATE} 2>>/applog/tomcat-flush.log

if [ $? -ne 0 ]

then

echo "Log flush for App1 failed .Check Log file for details"

fi

>/applog/app1/catalina.out

cp /applog/app2/catalina.out /applog/app2/catalina.out-${LOG_DATE} 2>>/applog/tomcat-flush.log

if [ $? -ne 0 ]

then

echo "Log flush for App1 failed .Check Log file for details"

fi

>/applog/app2/catalina.out

cp /applog/app3/catalina.out /applog/app3/catalina.out-${LOG_DATE} 2>>/applog/tomcat-flush.log

if [ $? -ne 0 ]

then

echo "Log flush for App1 failed .Check Log file for details"

fi

>/applog/app3/catalina.out

}

start() {

SHUTDOWN_PORT=`netstat -vatn|grep LISTEN|grep 8005|wc -l`

if [ $SHUTDOWN_PORT -ne 0 ]; then

echo -e "\033[31mAPP1 already started\033[0m"

else

echo -e "\033[31m....... \033[32mStarting APPLICATON CLUSTER\033[31m ......\033[0m"

echo -e "\033[32mStarting APP1...\033[0m"

echo ""

chown -R $TOMCAT_USER:$TOMCAT_USER ${APP1_DIR}/*



LVM thin provisioning - file system usage and pool usage dosn't match

When I was demonstrating LVM thin provisioning to new batch of campus hires ; they pointed out an important mismatch between thin pool usag...