Overview
This blog will help you understand the steps required to configure Jboss AS 7.x in clustered mode. Jboss can be configured in the 'standalone' or 'domain' mode. For this we would have to modify from both side Apache as well as JBoss AS7 side, hence let’s see the configuration one at a time.Apache side configuration
Let’s see what all configurations has to be made from Apache side
Copy following .so files to your “<Apache_Home>/modules” folder.
Edit your httpd.conf (i.e. /conf/http.conf) and add following lines at the bottom of the file.
In JBoss AS 7 we have by default two modes which are domain mode and standalone mode. Here, we would be using standalone mode. However in standalone mode also we have different xml files under the configuration folder from which cluster is enabled in standalone-ha.xml and standalone-full-ha.xml, thus make sure you would be using them and not other xml files
We would be seeing two scenarios here one would be creating a cluster on the same box and second when creating a cluster between different boxes.
Scenario 1: Cluster on same box
Once you have unzipped jboss-as-7.1.1.Final.zip , you would have to create two copies ofstandalone folder and rename them as standalone-node1 and standalone-node2 as shown below
/home/user/jboss-as-7.1.1.Final/standalone-node1
/home/user/jboss-as-7.1.1.Final/standalone-node2
Note: Make sure you keep the original copy for standalone folder as it is for future usage.
Give a unique name in the server element, as shown below .
standalone-node1
<server name="standalone-node1" xmlns="urn:jboss:domain:1.2">
standalone-node2
<server name="standalone-node2" xmlns="urn:jboss:domain:1.2">
We have to add the instance-id attribute in web subsystem as shown below in both the standalone nodes.
Last you just have to add the proxy-list in the attribute in mod-cluster-config of modcluster subsystem, which would be having IP Address and Port on which your Apache server is running so that JBoss server can communicate with it, as shown below in both the standalone nodes.
- mod_proxy.so
- mod_proxy_ajp.so
- mod_slotmem.so
- mod_manager.so
- mod_proxy_cluster.so
- mod_advertise.so
############### mod_cluster Setting - STARTED ###############
LoadModule slotmem_module modules/mod_slotmem.so
LoadModule manager_module modules/mod_manager.so
LoadModule proxy_cluster_module modules/mod_proxy_cluster.so
LoadModule advertise_module modules/mod_advertise.so
<VirtualHost 127.0.0.1:80>
<Directory />
Order deny,allow
Allow from all
</Directory>
<Location /mod_cluster_manager>
SetHandler mod_cluster-manager
Order deny,allow
Allow from all
</Location>
KeepAliveTimeout 60
MaxKeepAliveRequests 0
ManagerBalancerName testcluster
AdvertiseFrequency 5
</VirtualHost>
############### mod_cluster Setting - ENDED ###############
You can just add the IP_ADDRESS of the box on which Apache is running instead of “*”
NOTE:
You have to COMMENT the follwoing module (i.e. mod_proxy_balancer.so) in “httpd.conf” file, or else you would get error while starting your Apache, this is been done because we are now using mod_proxy_cluster.so instead of mod_proxy_balancer.so
#LoadModule proxy_balancer_module modules/mod_proxy_balancer.so
You have to do the following things before starting the APACHE
$ setenforce 0
The above command will disable SE Linux for your current running session. However, if you want to disable it permanently then follow the below steps.
$ vi /etc/selinux/config
#SELINUX=enforcing (comment this line and add the below line)
SELINUX=disabled
May be this would need to restart your system, this way you will be sure that the changes have taken place.
JBoss side configuration
Standalone mode
We would be seeing two scenarios here one would be creating a cluster on the same box and second when creating a cluster between different boxes.
Scenario 1: Cluster on same box
Once you have unzipped jboss-as-7.1.1.Final.zip , you would have to create two copies ofstandalone folder and rename them as standalone-node1 and standalone-node2 as shown below
/home/user/jboss-as-7.1.1.Final/standalone-node1
/home/user/jboss-as-7.1.1.Final/standalone-node2
Note: Make sure you keep the original copy for standalone folder as it is for future usage.
Give a unique name in the server element, as shown below .
standalone-node1
<server name="standalone-node1" xmlns="urn:jboss:domain:1.2">
standalone-node2
<server name="standalone-node2" xmlns="urn:jboss:domain:1.2">
We have to add the instance-id attribute in web subsystem as shown below in both the standalone nodes.
<subsystem xmlns="urn:jboss:domain:web:1.1" default-virtual-server="default-host" instance-id="${jboss.node.name}" native="false">
<connector name="http" protocol="HTTP/1.1" scheme="http" socket-binding="http"/>
<connector name="ajp" protocol="AJP/1.3" scheme="http" socket-binding="ajp"/>
.
.
.
</subsystem>
Last you just have to add the proxy-list in the attribute in mod-cluster-config of modcluster subsystem, which would be having IP Address and Port on which your Apache server is running so that JBoss server can communicate with it, as shown below in both the standalone nodes.
<subsystem xmlns="urn:jboss:domain:modcluster:1.0">Now you would have to run the below command to start both the JBoss node in a cluster
<mod-cluster-config advertise-socket="modcluster" proxy-list="1.1.1.1:80">
.
.
.
</mod-cluster-config>
</subsystem>
Node1
./standalone.sh -c standalone-ha.xml -b 127.0.0.1 -u 230.0.0.4 -Djboss.server.base.dir=../standalone-node1 -Djboss.node.name=node1 -Djboss.socket.binding.port-offset=100
Node2
./standalone.sh -c standalone-ha.xml -b 127.0.0.1 -u 230.0.0.4 -Djboss.server.base.dir=../standalone-node2 -Djboss.node.name=node2 -Djboss.socket.binding.port-offset=200
Where:
-c = is for server configuration file to be used
-b = is for binding address
-u = is for multicast address
-Djboss.server.base.dir = is for the path from where node is present
-Djboss.node.name = is for the name of the node
-Djboss.socket.binding.port-offset = is for the port offset on which node would be running
Note: However we need to keep in mind the following things
- Both the nodes should have same multicast address
- Both the nodes should have different node names
- Both the nodes should have different socket binding port-offsets
Once both the node comes up properly you would not see them in cluster, hence to make sure if both of the nodes are in a cluster then you would need to deploy the an application which has the distributable tag in web.xml . You can download one of our sample clustered application by clicking here
After downloading the ClusterWebApp.war you just have to keep it in (/home/user/jboss-as-7.1.1.Final/standalone-nodeX/deployments) both nodes deployments folder, just after that you would see similar below messages in both the nodes prompt, having both node names in there cluster view.
17:31:06,600 INFO [stdout] (pool-14-thread-1)
17:31:06,601 INFO [stdout] (pool-14-thread-1) -------------------------------------------------------------------
17:31:06,601 INFO [stdout] (pool-14-thread-1) GMS: address=standalone-node1/web, cluster=web, physical address=127.0.0.1:55300
17:31:06,602 INFO [stdout] (pool-14-thread-1) -------------------------------------------------------------------
17:31:08,791 INFO [org.infinispan.configuration.cache.EvictionConfigurationBuilder] (MSC service thread 1-2) ISPN000152: Passivation configured without an eviction policy being selected. Only manually evicted entities will be pasivated.
17:31:08,796 INFO [org.infinispan.configuration.cache.EvictionConfigurationBuilder] (MSC service thread 1-5) ISPN000152: Passivation configured without an eviction policy being selected. Only manually evicted entities will be pasivated.
17:31:08,839 INFO [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (pool-15-thread-1) ISPN000078: Starting JGroups Channel
17:31:08,844 INFO [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (pool-15-thread-1) ISPN000094: Received new cluster view: [standalone-node1/web|0] [standalone-node1/web]
17:31:08,845 INFO [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (pool-15-thread-1) ISPN000079: Cache local address is standalone-node1/web, physical addresses are [127.0.0.1:55300]
Considering everything went well you should be able to view clusters by hitting URL http://localhost/mod_cluster_manager . Below image is showing when cluster is been made in on the same box using standalone mode
Scenario 2: Cluster on different boxes
After unzipping JBoss AS 7 in both the boxes [i.e. box-1=10.10.10.10 and box-2=20.20.20.20 ] then you can create just a single copies of standalone folder in respective boxes
Box-1 : 10.10.10.10
/home/user/jboss-as-7.1.1.Final/standalone-node1
Box-2 : 20.20.20.20
/home/user/jboss-as-7.1.1.Final/standalone-node2
Now you would have to run the below command to start both the JBoss node in a cluster
Note: However we need to keep in mind the following things
Both the nodes should have same multicast address
Both the nodes should have different node names
Both the nodes should be running on the IP_ADDRESS or HOST_NAME of the box
Node1 on Box-1 [10.10.10.10]
./standalone.sh -c standalone-ha.xml -b 10.10.10.10 -u 230.0.0.4 -Djboss.server.base.dir=../standalone-node1 -Djboss.node.name=node1
Node2 on Box-2 [20.20.20.20]
./standalone.sh -c standalone-ha.xml -b 20.20.20.20 -u 230.0.0.4 -Djboss.server.base.dir=../standalone-node2 -Djboss.node.name=node2
Here we would not have to worry about the port conflicts as we are running both the nodes on different boxes having different binding address.
Repeat the same step-3 and step-4 of Scenario-1 and you would then see the same cluster view in each running nodes prompts.
If you are looking for running multiple clusters, then you would have to make sure you give a different set of multicast address (i.e. -u option) for each cluster.
Domain mode
Scenario 1: Cluster on same box
Now in “/home/user/jboss-as-7.1.1.Final/domain/configuration/domain.xml” file make the below changes, which is just adding a new server-group (i.e. ha-server-group) which would be using haprofile and ha-sockets socket binding group, where ha is for cluster enabled.
<server-groups>
<server-group name="ha-server-group" profile="ha">
<jvm name="default">
<heap size="64m" max-size="512m"/>
</jvm>
<socket-binding-group ref="ha-sockets"/>
</server-group>
.
.
</server-groups>
Where:
profile: tells which type of profile is been used (i.e. web, messaging, cluster, full)
- socket-binding-group: tells which all type of protocols is been used (i.e. web [http,ajp], messaging, jgroups [udp, tcp], full)
- server-group : tells what profile is been used and what type of sockets is been used
You just have to add the proxy-list in the attribute in mod-cluster-config of modcluster subsystem, which would be having IP Address and Port on which your Apache server is running so that JBoss server can communicate with it, as shown below in profile element that are using in domain.xml.
<subsystem xmlns="urn:jboss:domain:modcluster:1.0">
<mod-cluster-config advertise-socket="modcluster" proxy-list="1.1.1.1:80">
.
.
.
</mod-cluster-config>
</subsystem>
After that you would have to make the below changes in “/home/user/jboss-as-7.1.1.Final/domain/configuration/host.xml” file which is just adding two new JBoss nodes with the name ha-server-1 and ha-server-2 which are using the ha-server-group server group created in the pervious setp and making this servers clusterd enabled.
<servers> <server name="ha-server-1" group="ha-server-group" auto-start="true"> <socket-bindings port-offset="100"/> </server> <server name="ha-server-2" group="ha-server-group" auto-start="true"> <socket-bindings port-offset="200"/> </server>..</servers>
Note: You are giving unique name and port offset for these servers, as both the servers are running on the same box.
Create a Management User using the add-user.sh script as shown below. This is done so that we can access admin console.
bin]$ ./add-user.sh
What type of user do you wish to add?
a) Management User (mgmt-users.properties)
b) Application User (application-users.properties)
(a): a
Enter the details of the new user to add.
Realm (ManagementRealm) :
Username : testuser
Password : testpassword
Re-enter Password : testpassword
About to add user 'testuser' for realm 'ManagementRealm'
Is this correct yes/no? yes
Added user 'testuser' to file '/home/user/jboss-as-7.1.1.Final/standalone/configuration/mgmt-users.properties'
Added user 'testuser' to file '/home/user/jboss-as-7.1.1.Final/domain/configuration/mgmt-users.properties'
Once everything is done start your server by using the below command, however you would not see that the nodes ha-server-1 and ha-server-2 are in a cluster for that you would have to deploy an application which has the distributable tag in web.xml .
bin]$ ./domain.sh
Now you can download one of our sample clustered application by : clicking here and deploy it from admin console from the URL “http://localhost:9990/console”
Just after deploying application and adding it to ha-server-group you would see the below cluster view in the prompt in which the domain is running.
[Server:ha-server-2] 15:12:33,971 INFO [org.jboss.web] (MSC service thread 1-1) JBAS018210: Registering web context: /ClusterWebApp
[Server:ha-server-2] 15:12:34,239 INFO [org.jboss.as.clustering.impl.CoreGroupCommunicationService.lifecycle.web] (Incoming-1,null) JBAS010247: New cluster view for partition web (id: 1, delta: 1, merge: false) : [master:ha-server-2/web, master:ha-server-1/web]
[Server:ha-server-2] 15:12:34,242 INFO [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (Incoming-1,null) ISPN000094: Received new cluster view: [master:ha-server-2/web|1] [master:ha-server-2/web, master:ha-server-1/web]
.
.
[Server:ha-server-1] 15:12:34,377 INFO [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (pool-14-thread-3) ISPN000078: Starting JGroups Channel
[Server:ha-server-1] 15:12:34,378 INFO [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (pool-14-thread-3) ISPN000094: Received new cluster view: [master:ha-server-2/web|1] [master:ha-server-2/web, master:ha-server-1/web]
Troubleshooting for SELinux
Adding policy for mod_cluster under SELinux
Mod_cluster needs to open port and create shared memory and files, therefore some permission have to be added, you need to configure something like:
policy_module(mod_cluster, 1.0)
require {
type unconfined_java_t;
type httpd_log_t;
type httpd_t;
type http_port_t;
class udp_socket node_bind;
class file write;
}
#============= httpd_t ==============
allow httpd_t httpd_log_t:file write;
corenet_tcp_bind_generic_port(httpd_t)
corenet_tcp_bind_soundd_port(httpd_t)
corenet_udp_bind_generic_port(httpd_t)
corenet_udp_bind_http_port(httpd_t)
#============= unconfined_java_t ==============
allow unconfined_java_t http_port_t:udp_socket node_bind;
Put the above in a file for example mod_cluster.te and generate the mod_cluster.pp file.
# make -f /usr/share/selinux/devel/Makefile
Compiling targeted mod_cluster module
/usr/bin/checkmodule: loading policy configuration from tmp/mod_cluster.tmp
/usr/bin/checkmodule: policy configuration loaded
/usr/bin/checkmodule: writing binary representation (version 14) to tmp/mod_cluster.mod
Creating targeted mod_cluster.pp policy package
rm tmp/mod_cluster.mod.fc tmp/mod_cluster.mod
The mod_cluster.pp file should be proceeded by semodule as root:
# semodule -i mod_cluster.pp
You can confirm policy added
# semodule -l
Adding AJP port using semanage
Default port for AJP is 8009. When we configure # of clusters using mod_cluster in JBoss each cluster has its specific AJP port usually configured as per offset value. For example if you are running cluster node 1 with offset 100 then your port will be 8109.
SElinux will not allow apache to connect to AJP port, we need to add this port using following command.
Port contexts
Allow Apache to listen on tcp port 8109
# semanage port -a -t http_port_t -p tcp 8109
See newly added port by following command
# semanage port –l | grep http
You will need to add all the AJP ports based on # of clusters nodes you have configured.
No comments:
Post a Comment