Jboss AS 7.x cluster setup

Overview

This blog will help you understand the steps required to configure Jboss AS 7.x in clustered mode. Jboss can be configured in the 'standalone' or 'domain' mode. For this we would have to modify from both side Apache as well as JBoss AS7 side, hence let’s see the configuration one at a time.

Apache side configuration

Let’s see what all configurations has to be made from Apache side

Download the required binaries for your OS from below link, example for RHEL x64 bit its [dynamic libraries linux2-x64] http://www.jboss.org/mod_cluster/downloads/1-1-0.html

Copy following .so files to your “<Apache_Home>/modules” folder.

• mod_proxy.so
• mod_proxy_ajp.so
• mod_slotmem.so
• mod_manager.so
• mod_proxy_cluster.so
• mod_advertise.so

Edit your httpd.conf (i.e. /conf/http.conf) and add following lines at the bottom of the file. 

############### mod_cluster Setting - STARTED ###############
LoadModule slotmem_module modules/mod_slotmem.so
LoadModule manager_module modules/mod_manager.so
LoadModule proxy_cluster_module modules/mod_proxy_cluster.so
LoadModule advertise_module modules/mod_advertise.so
<VirtualHost 127.0.0.1:80>
<Directory />
Order deny,allow
Allow from all
</Directory>
<Location /mod_cluster_manager>
SetHandler mod_cluster-manager
Order deny,allow
Allow from all
</Location>
KeepAliveTimeout 60
MaxKeepAliveRequests 0
ManagerBalancerName testcluster
AdvertiseFrequency 5
</VirtualHost>
############### mod_cluster Setting - ENDED ###############

You can just add the IP_ADDRESS of the box on which Apache is running instead of “*”

NOTE:

You have to COMMENT the follwoing module (i.e. mod_proxy_balancer.so) in “httpd.conf” file, or else you would get error while starting your Apache, this is been done because we are now using mod_proxy_cluster.so instead of mod_proxy_balancer.so

#LoadModule proxy_balancer_module modules/mod_proxy_balancer.so

You have to do the following things before starting the APACHE

$ setenforce 0

The above command will disable SE Linux for your current running session. However, if you want to disable it permanently then follow the below steps.

$ vi /etc/selinux/config

#SELINUX=enforcing (comment this line and add the below line)

SELINUX=disabled

May be this would need to restart your system, this way you will be sure that the changes have taken place.

JBoss side configuration

Standalone mode

In JBoss AS 7 we have by default two modes which are domain mode and standalone mode. Here, we would be using standalone mode. However in standalone mode also we have different xml files under the configuration folder from which cluster is enabled in standalone-ha.xml and standalone-full-ha.xml, thus make sure you would be using them and not other xml files

We would be seeing two scenarios here one would be creating a cluster on the same box and second when creating a cluster between different boxes.

Scenario 1: Cluster on same box

Once you have unzipped jboss-as-7.1.1.Final.zip , you would have to create two copies ofstandalone folder and rename them as standalone-node1 and standalone-node2 as shown below

/home/user/jboss-as-7.1.1.Final/standalone-node1
/home/user/jboss-as-7.1.1.Final/standalone-node2

Note: Make sure you keep the original copy for standalone folder as it is for future usage.
Give a unique name in the server element, as shown below .

standalone-node1

<server name="standalone-node1" xmlns="urn:jboss:domain:1.2">

standalone-node2

<server name="standalone-node2" xmlns="urn:jboss:domain:1.2">


We have to add the instance-id attribute in web subsystem as shown below in both the standalone nodes.

<subsystem xmlns="urn:jboss:domain:web:1.1" default-virtual-server="default-host" instance-id="${jboss.node.name}" native="false">
<connector name="http" protocol="HTTP/1.1" scheme="http" socket-binding="http"/>
<connector name="ajp" protocol="AJP/1.3" scheme="http" socket-binding="ajp"/>
.
.
.
</subsystem>

Last you just have to add the proxy-list in the attribute in mod-cluster-config of modcluster subsystem, which would be having IP Address and Port on which your Apache server is running so that JBoss server can communicate with it, as shown below in both the standalone nodes.

<subsystem xmlns="urn:jboss:domain:modcluster:1.0">
<mod-cluster-config advertise-socket="modcluster" proxy-list="1.1.1.1:80">
.
.
.
</mod-cluster-config>
</subsystem>

Now you would have to run the below command to start both the JBoss node in a cluster

Node1

./standalone.sh -c standalone-ha.xml -b 127.0.0.1 -u 230.0.0.4 -Djboss.server.base.dir=../standalone-node1 -Djboss.node.name=node1 -Djboss.socket.binding.port-offset=100

Node2

./standalone.sh -c standalone-ha.xml -b 127.0.0.1 -u 230.0.0.4 -Djboss.server.base.dir=../standalone-node2 -Djboss.node.name=node2 -Djboss.socket.binding.port-offset=200

Where:

-c = is for server configuration file to be used

-b = is for binding address

-u = is for multicast address

-Djboss.server.base.dir = is for the path from where node is present

-Djboss.node.name = is for the name of the node

-Djboss.socket.binding.port-offset = is for the port offset on which node would be running

Note: However we need to keep in mind the following things

• Both the nodes should have same multicast address
• Both the nodes should have different node names
• Both the nodes should have different socket binding port-offsets

Once both the node comes up properly you would not see them in cluster, hence to make sure if both of the nodes are in a cluster then you would need to deploy the an application which has the distributable tag in web.xml . You can download one of our sample clustered application by  clicking here

After downloading the ClusterWebApp.war you just have to keep it in (/home/user/jboss-as-7.1.1.Final/standalone-nodeX/deployments) both nodes deployments folder, just after that you would see similar below messages in both the nodes prompt, having both node names in there cluster view.

17:31:06,600 INFO  [stdout] (pool-14-thread-1)
17:31:06,601 INFO  [stdout] (pool-14-thread-1) -------------------------------------------------------------------
17:31:06,601 INFO  [stdout] (pool-14-thread-1) GMS: address=standalone-node1/web, cluster=web, physical address=127.0.0.1:55300
17:31:06,602 INFO  [stdout] (pool-14-thread-1) -------------------------------------------------------------------
17:31:08,791 INFO  [org.infinispan.configuration.cache.EvictionConfigurationBuilder] (MSC service thread 1-2) ISPN000152: Passivation configured without an eviction policy being selected. Only manually evicted entities will be pasivated.
17:31:08,796 INFO  [org.infinispan.configuration.cache.EvictionConfigurationBuilder] (MSC service thread 1-5) ISPN000152: Passivation configured without an eviction policy being selected. Only manually evicted entities will be pasivated.
17:31:08,839 INFO  [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (pool-15-thread-1) ISPN000078: Starting JGroups Channel
17:31:08,844 INFO  [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (pool-15-thread-1) ISPN000094: Received new cluster view: [standalone-node1/web|0] [standalone-node1/web]
17:31:08,845 INFO  [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (pool-15-thread-1) ISPN000079: Cache local address is standalone-node1/web, physical addresses are [127.0.0.1:55300]

Considering everything went well you should be able to view clusters by hitting URL http://localhost/mod_cluster_manager . Below image is showing when cluster is been made in on the same box using standalone mode

Scenario 2: Cluster on different boxes

After unzipping JBoss AS 7 in both the boxes [i.e. box-1=10.10.10.10 and box-2=20.20.20.20 ] then you can create just a single copies of standalone folder in respective boxes

Box-1 : 10.10.10.10

/home/user/jboss-as-7.1.1.Final/standalone-node1

Box-2 : 20.20.20.20

/home/user/jboss-as-7.1.1.Final/standalone-node2

Now you would have to run the below command to start both the JBoss node in a cluster

Note: However we need to keep in mind the following things

Both the nodes should have same multicast address

Both the nodes should have different node names

Both the nodes should be running on the IP_ADDRESS or HOST_NAME of the box

Node1 on Box-1 [10.10.10.10]

./standalone.sh -c standalone-ha.xml -b 10.10.10.10 -u 230.0.0.4 -Djboss.server.base.dir=../standalone-node1 -Djboss.node.name=node1

Node2 on Box-2 [20.20.20.20]

./standalone.sh -c standalone-ha.xml -b 20.20.20.20 -u 230.0.0.4 -Djboss.server.base.dir=../standalone-node2 -Djboss.node.name=node2

Here we would not have to worry about the port conflicts as we are running both the nodes on different boxes having different binding address.

Repeat the same step-3 and step-4 of Scenario-1 and you would then see the same cluster view in each running nodes prompts.

If you are looking for running multiple clusters, then you would have to make sure you give a different set of multicast address (i.e.  -u option) for each cluster.

Domain mode

Scenario 1: Cluster on same box

Now in “/home/user/jboss-as-7.1.1.Final/domain/configuration/domain.xml” file make the below changes, which is just adding a new server-group (i.e. ha-server-group) which would be using haprofile and ha-sockets socket binding group, where ha is for cluster enabled.

<server-groups>
<server-group name="ha-server-group" profile="ha">
<jvm name="default">
<heap size="64m" max-size="512m"/>
</jvm>
<socket-binding-group ref="ha-sockets"/>
</server-group>
.
.
</server-groups>

Where:

profile: tells which type of profile is been used (i.e. web, messaging, cluster, full)

• socket-binding-group: tells which all type of protocols is been used (i.e. web [http,ajp], messaging, jgroups [udp, tcp], full)
• server-group : tells what profile is been used and what type of sockets is been used

You just have to add the proxy-list in the attribute in mod-cluster-config of modcluster subsystem, which would be having IP Address and Port on which your Apache server is running so that JBoss server can communicate with it, as shown below in profile element that are using in domain.xml.

<subsystem xmlns="urn:jboss:domain:modcluster:1.0">
<mod-cluster-config advertise-socket="modcluster" proxy-list="1.1.1.1:80">
.
.
.
</mod-cluster-config>
</subsystem>

After that you would have to make the below changes in “/home/user/jboss-as-7.1.1.Final/domain/configuration/host.xml” file which is just adding two new JBoss nodes with the name ha-server-1 and ha-server-2 which are using the ha-server-group server group created in the pervious setp and making this servers clusterd enabled.

<servers> <server name="ha-server-1" group="ha-server-group" auto-start="true"> <socket-bindings port-offset="100"/> </server> <server name="ha-server-2" group="ha-server-group" auto-start="true"> <socket-bindings port-offset="200"/> </server>..</servers>

Note: You are giving unique name and port offset for these servers, as both the servers are running on the same box.

Create a Management User using the add-user.sh script as shown below. This is done so that we can access admin console.

bin]$ ./add-user.sh 

What type of user do you wish to add?

 a) Management User (mgmt-users.properties)

 b) Application User (application-users.properties)

(a): a

Enter the details of the new user to add.

Realm (ManagementRealm) :

Username : testuser

Password : testpassword

Re-enter Password : testpassword

About to add user 'testuser' for realm 'ManagementRealm'

Is this correct yes/no? yes

Added user 'testuser' to file '/home/user/jboss-as-7.1.1.Final/standalone/configuration/mgmt-users.properties'

Added user 'testuser' to file '/home/user/jboss-as-7.1.1.Final/domain/configuration/mgmt-users.properties'

Once everything is done start your server by using the below command, however you would not see that the nodes ha-server-1 and ha-server-2 are in a cluster for that you would have to deploy an application which has the distributable tag in web.xml .

bin]$ ./domain.sh

Now you can download one of our sample clustered application by : clicking here and deploy it from admin console from the URL “http://localhost:9990/console”

Just after deploying application and adding it to ha-server-group you would see the below cluster view in the prompt in which the domain is running.

[Server:ha-server-2] 15:12:33,971 INFO  [org.jboss.web] (MSC service thread 1-1) JBAS018210: Registering web context: /ClusterWebApp
[Server:ha-server-2] 15:12:34,239 INFO  [org.jboss.as.clustering.impl.CoreGroupCommunicationService.lifecycle.web] (Incoming-1,null) JBAS010247: New cluster view for partition web (id: 1, delta: 1, merge: false) : [master:ha-server-2/web, master:ha-server-1/web]
[Server:ha-server-2] 15:12:34,242 INFO  [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (Incoming-1,null) ISPN000094: Received new cluster view: [master:ha-server-2/web|1] [master:ha-server-2/web, master:ha-server-1/web]
.
.
[Server:ha-server-1] 15:12:34,377 INFO  [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (pool-14-thread-3) ISPN000078: Starting JGroups Channel
[Server:ha-server-1] 15:12:34,378 INFO  [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (pool-14-thread-3) ISPN000094: Received new cluster view: [master:ha-server-2/web|1] [master:ha-server-2/web, master:ha-server-1/web]

Troubleshooting for SELinux

Adding policy for mod_cluster under SELinux

Mod_cluster needs to open port and create shared memory and files, therefore some permission have to be added, you need to configure something like:

policy_module(mod_cluster, 1.0)
require {
        type unconfined_java_t;
        type httpd_log_t;
        type httpd_t;
        type http_port_t;
        class udp_socket node_bind;
        class file write;
}
#============= httpd_t ==============
allow httpd_t httpd_log_t:file write;
corenet_tcp_bind_generic_port(httpd_t)
corenet_tcp_bind_soundd_port(httpd_t)
corenet_udp_bind_generic_port(httpd_t)
corenet_udp_bind_http_port(httpd_t)
#============= unconfined_java_t ==============
allow unconfined_java_t http_port_t:udp_socket node_bind;

Put the above in a file for example mod_cluster.te and generate the mod_cluster.pp file.

# make -f /usr/share/selinux/devel/Makefile

Compiling targeted mod_cluster module

/usr/bin/checkmodule:  loading policy configuration from tmp/mod_cluster.tmp

/usr/bin/checkmodule:  policy configuration loaded

/usr/bin/checkmodule:  writing binary representation (version 14) to tmp/mod_cluster.mod

Creating targeted mod_cluster.pp policy package

rm tmp/mod_cluster.mod.fc tmp/mod_cluster.mod

The mod_cluster.pp file should be proceeded by semodule as root:

# semodule -i mod_cluster.pp

You can confirm policy added

# semodule -l

Adding AJP port using semanage

Default port for AJP is 8009. When we configure # of clusters using mod_cluster in JBoss each cluster has its specific AJP port usually configured as per offset value. For example if you are running cluster node 1 with offset 100 then your port will be 8109.

SElinux will not allow apache to connect to AJP port, we need to add this port using following command.

Port contexts

Allow Apache to listen on tcp port 8109

#  semanage port -a -t http_port_t -p tcp 8109

See newly added port by following command

#  semanage port –l | grep http

You will need to add all the AJP ports based on # of clusters nodes you have configured.

Reference 

http://middlewaremagic.com/jboss/?cat=8

Nagios monitoring for Jboss AS 7.x - jboss2nagios

jboss2nagios

Integrate JBoss into Nagios monitoring through a small MBean and a perl based Nagios plugin. Lets you read and monitor JMX values from JBoss servers very efficiently. On the Nagios server no JDK or JBoss installation is needed. This Jboss SAR MBean and Perl plug-in is compatible with Jboss 7.1.1.Final. It allows you to monitor following

1. HeapMemoryUsage
2. Non heap memory usage
3. CPU usage
4. Take heap dump

Installation:

• Copy the collector.sar (from the mbean/ directory) to your JBoss deploy directory. Port 5566 is then open for the plugin to access it.
• Copy the plugin check_mbean_collector (from the plugin/ directory) to the Nagios plugin directory on the Nagios server.
• Edit your Nagios config to use the check_mbean_collector to monitor any attributes of any MBean.

You might check that the plugin and the MBean is working properly by doing a test run on the Nagios server:

./check_mbean_collector -H jbossserver -p 5566 -m java.lang:type=Memory -a HeapMemoryUsage -w 70 -c 90 Please note that you need the nagios-plugins package installed and of course replace "jbossserver" above with you server name. Plugin usage:

Retrieve some MBean attribute value from a JBoss server through the collector MBean: check_mbean_collector -H host -p port -m mbean_name -a attribute_name -w warning_level -c critical_level

Usage:

check_mbean_collector -H host[,host,..] -p port -m mbean-name -a attribute-name -w warning-level -c critical-level check_mbean_collector [-h | --help] check_mbean_collector [-V | --version]

• [host] The server running JBoss. Giving a comma separated list of hosts switches to a check for a singleton in a cluster.
• [port] The port the deployed collector MBean is listening to
• [mbean_name] The JMX name of the MBean that includes the attribute, e.g. ava.lang:type=Memory Use the ${some.env} notation to refer to a JVM environment variable on the server. In Nagios config files this must be escaped like this: $$\{some.env}
• [attribute_name] The name of the MBean attribute to retrieve, e.g. HeapMemoryUsage Prefix with * to get the difference between two calls (delta). ${...} can be used.
• [warning_level] The level as a number from which on the WARNING status should be set
• [critical_level] The level as a number from which on the CRITICAL status should be set

Commands

$cd /usr/local/nagios/libexec/

1. HeapMemoryUsage : ./check_mbean_collector -H jbossserver -p 5566 -m java.lang:type=Memory -a HeapMemoryUsage -w 70 -c 90
2. Non heap memory usage : ./check_mbean_collector -H jbossserver -p 5566 -m java.lang:type=Memory -a NonHeapMemoryUsage -w 70 -c 90
3. CPU usage : ./check_mbean_collector -H jbossserver -p 5566 -m java.lang:type=OperatingSystem -a CPUUsage -w 70 -c 90
4. Take heap dump : ./heap_util -H jbossserver -p 5566 -m com.sun.management:type=HotSpotDiagnostic

Source

Original project has been modified and made compatible with Jboss AS 7.x. It has also been tested with Jboss 6 EAP. https://github.com/manishdevraj/jboss2nagios/

SourceForge Project

The project is hosted on SoruceForge and has it's own project homepage 
https://sourceforge.net/projects/jboss2nagios/

Reference : http://sourceforge.net/projects/jboss2nagios

My Fellow Coders

Do you remember the last time when you had to clean someone else’s CRAPPY code?  I am sure at least once in our lifetime we had to deal with a mess so grave that it took weeks to do what should have taken hours? Have you seen what should have been a one-line change, made instead in hundreds of different modules? These symptoms are all too common in the life of a coder.

Wonder, why does this happen to code? Why good code turns into bad code? We have lots of explanations for it.

Obvious as it may sound, we complain that the requirements changed so quickly that the original design did not comply. We blame the schedules that were too tight to do things right.

We blather about stupid managers and intolerant customers and useless marketing people.  But the fault, my dear friends, is not in our stars, but in us.

“We are unprofessional!!”

This may be a bitter pill to swallow. How could this mess be our fault? What about the requirements? What about the schedule? What about the stupid managers and the useless marketing peoples? Don’t they share some of the blame?

NO. The managers and marketing people look to us for the information which they can use to make promises and commitments to the clients; and even when they don’t look to us, we should not be shy about telling them what we think. The end users/clients look to us to validate the requirements and the way it will fit into the system. The project managers look to us to help work out the schedule/project plan. We are deeply associated in the planning of the project and in sharing a great deal of the responsibility for any failures; especially if those failures have to do with bad code!

“But wait!” you say. “If I don’t do what my manager says, I’ll be fired.” Probably not.

Most managers want the truth, even when they don’t act like it. Most managers want good code, even when they are obsessing about the schedule.

They may defend the schedule and requirements with passion; but that’s their job. It’s your job to defend the code with equal passion.

To drive this point home, what if you were a doctor and had a patient who demanded that you stop all the silly hand-washing in preparation for surgery because it was taking too much time?  Clearly the patient is the boss; and yet the doctor should absolutely refuse to comply. Why? Because the doctor knows more than the patient about the risks of disease and infection. It would be unprofessional (never mind criminal) for the doctor to comply with the patient.

So too it is unprofessional for programmers to bend to the will of managers who don’t understand the risks of making messes.