Nginx+Tomcat+Session 高性能群集搭建

精选转载

todayboy 2012-07-25 09:21:25 博主文章分类：Nginx&Squid

文章标签 Tomcat Session Nginx 高性能群集搭建 文章分类 Nginx 服务器

环境:前端nginx做分发(无PHP环境)，后端两台tomcat分别为：
nginx：192.168.3.89 web.test.com
tomcat1：192.168.3.73
tomcat2：192.168.3.78

一、前端Nginx配置
1.前端nginx安装必备的开发包
yum -y install gcc gcc-c++ autoconf libjpeg libjpeg-devel libpng libpng-devel freetype freetype-devel libxml2 libxml2-devel zlib zlib-devel glibc glibc-devel glib2 glib2-devel bzip2 bzip2-devel ncurses ncurses-devel curl curl-devel

2.同步系统时间：
#vim /etc/ntp.conf //添加下面三行内容（19行下面添加）

server 3.cn.pool.ntp.org
server 3.asia.pool.ntp.org
server 0.asia.pool.ntp.org

# service ntpd stop
#ntpdate cn.pool.ntp.org //更新时间
#service ntpd start
#chkconfig ntpd on

3.安装pcre包和zlib以支持重写,正则以及网页压缩
pcre的安装
# unzip pcre-8.21.zip
# ./configure
# make && make install

4.Nginx的安装,给nginx安装jvmroute模块
# groupadd www
# useradd -g www -s /sbin/nologin www
# mkdir /usr/local/webserver/nginx/logs -p
# tar xvf nginx-upstream-jvm-route-0.1.tar.gz
# tar xvf nginx-1.0.11.tar.gz
# cd nginx-1.0.11
# patch -p0 < ../nginx_upstream_jvm_route/jvm_route.patch
# ./configure --prefix=/usr/local/webserver/nginx --sbin-path=/usr/sbin/nginx --conf-path=/etc/nginx/nginx.conf --user=www --group=www --with-http_ssl_module --with-http_stub_status_module --with-http_gzip_static_module --add-module=../nginx_upstream_jvm_route/
# make && make install

# vim /etc/nginx/nginx.conf
user www www;
worker_processes 8;
error_log /usr/local/webserver/nginx/logs/web-error.log crit;
pid        /usr/local/webserver/nginx/nginx.pid;

#Specifies the value for maximum file descriptors that can be opened by this process.
worker_rlimit_nofile 51200;
events
    {
        use epoll;
        worker_connections 51200;
    }

http
    {
        include       mime.types;
        default_type application/octet-stream;
        #charset gb2312
        server_names_hash_bucket_size 128;
        client_header_buffer_size 32k;
        large_client_header_buffers 4 32k;
        client_max_body_size     10m;
        client_body_buffer_size 128k;
        proxy_connect_timeout    600;
        proxy_read_timeout       600;
        proxy_send_timeout       600;
        proxy_buffer_size        8k;
        proxy_buffers            4 32k;
        proxy_busy_buffers_size 64k;
        proxy_temp_file_write_size 64k;

        sendfile on;
        tcp_nopush     on;
        keepalive_timeout 60;
        tcp_nodelay on;

        gzip on;
        gzip_min_length 1k;
        gzip_buffers     4 16k;
        gzip_http_version 1.0;
        gzip_comp_level 2;
        gzip_types       text/plain application/x-javascript text/css application/xml;
        gzip_vary on;

        #limit_zone crawler $binary_remote_addr 10m;

        upstream tomcat {
               server 192.168.3.73:8080 srun_id=tomcat1;
               server 192.168.3.78:8080 srun_id=tomcat2;
               jvm_route $cookie_JSESSIONID|sessionid reverse;
        }

server
    {
    listen       80;
    server_name web.test.com;
        index index.html index.htm index.jsp default.jsp index.do default.do;
#       root /usr/local/tomcat/webapps/test/;

        if ($host = 'test.com' )
        {

                rewrite ^/(.*)$ http://web.test.com/$1 permanent;

        }
        if (-d $request_filename)

        {

                rewrite ^/(.*)([^/])$ http://$host/$1$2/ permanent;

        }

        location ~ \.(jsp|jspx|do)?$
        {
        proxy_redirect off;
        proxy_set_header Host $host;
        proxy_set_header   X-Forwarded-For $remote_addr;
        proxy_pass http://tomcat;
        }

        location ~ .*\.(gif|jpg|jpeg|png|bmp|swf)$

        {

                expires 30d;

        }

        location ~ .*\.(js|css)?$

        {

                expires 1h;

        }
     log_format access '$remote_addr - $remote_user [$time_local] "$request" '
             '$status $body_bytes_sent "$http_referer" '
             '"$http_user_agent" $http_x_forwarded_for';
     access_log /usr/local/webserver/nginx/logs/web-access.log access;

    }
server
{
    listen 80;
    server_name status.test.com;

    location /stub_status {
    stub_status on;
    access_log   off;
    }
}
}

二、后端tomcat的配置
apache-tomcat的安装，而apache-tomcat又需要java环境的支持
1.JDK的安装
# chmod +x jdk-6u29-linux-i586.bin
# ./jdk-6u29-linux-i586.bin
# mv jdk1.6.0_29 /usr/local/
到此JDK已经安装完成

2.到/usr/bin目录下，把原用的java,javac文件删除:
# rm -rf /usr/bin/java
# rm -rf /usr/bin/javac
在/usr/bin 下建立软连接 java
# ln -sv /usr/local/jdk1.6.0_29/bin/java /usr/bin/java
# ln -sv /usr/local/jdk1.6.0_29/bin/javac /usr/bin/javac
# java -version
java version "1.6.0_29"
Java(TM) SE Runtime Environment (build 1.6.0_29-b11)
Java HotSpot(TM) Client VM (build 20.4-b02, mixed mode, sharing)
# javac -version
javac 1.6.0_29

3.为了方便下在的工作建立两个软链接：
# ln -sv /usr/local/jdk1.6.0_29 /usr/local/jdk
# ln -sv /usr/local/jdk1.6.0_29/jre /usr/local/jre

4.Tomcat的安装
# tar xvf apache-tomcat-6.0.33.tar.gz
# mv apache-tomcat-6.0.33 /usr/local/tomcat
设置环境变量:
# vim ~/.bash_profile //仅当前用户有效
#Set Environment by NetSeek
JAVA_HOME=/usr/local/jdk
export JAVA_HOME
JRE_HOME=/usr/local/jre
export JRE_HOME
CLASSPATH=/usr=/usr/local/tomcat/common/lib/:/usr/local/jdk/lib:/usr/local/jre/lib
export CLASSPATH
PATH=$PATH:/usr/local/tomcat/bin/:/usr/local/jdk/bin:/usr/local/jre/bin
export PATH
TOMCAT_HOME=/usr/local/tomcat
export TOMCAT_HOME
# source ~/.bash_profile

5.启动tomcat以检查是否存在错误：
# cd /usr/local/tomcat/bin/
# ./startup.sh
[url]http://IP[/url]地址:8080 可以看到猫头：）成功了。

在另一台机器上也安装tomcat 并配置好。

三、tomcat集群配置，session 同步配置：
1.tomca1 tomcat 2 在不同机器上
Tomcat 1 （IP: 192.168.3.73）
A、修改Engine节点信息： <Engine name="Catalina" defaultHost="localhost" jvmRoute="tomcat1">
B、去掉<Cluster> </Cluster> 的注释符
C、修改Cluster 节点信息
<Cluster className="org.apache.catalina.ha.tcp.SimpleTcpCluster"
         channelSendOptions="8">

          <Manager className="org.apache.catalina.ha.session.DeltaManager"
                   expireSessionsOnShutdown="false"
                   notifyListenersOnReplication="true"/>

          <Channel className="org.apache.catalina.tribes.group.GroupChannel">
            <Membership className="org.apache.catalina.tribes.membership.McastService"
                        mcastBindAddress="192.168.3.73"
                        address="224.0.0.0"
                        port="45564"
                        frequency="500"
                        dropTime="3000"/>

            <Receiver className="org.apache.catalina.tribes.transport.nio.NioReceiver"
                      tcpListenAddress="192.168.3.73"
                      port="4001"
                      autoBind="100"
                      selectorTimeout="100"
                      maxThreads="6"/>

            <Sender className="org.apache.catalina.tribes.transport.ReplicationTransmitter">
              <Transport className="org.apache.catalina.tribes.transport.nio.PooledParallelSender"/>
            </Sender>
            <Interceptor className="org.apache.catalina.tribes.group.interceptors.TcpFailureDetector"/>
            <Interceptor className="org.apache.catalina.tribes.group.interceptors.MessageDispatch15Interceptor"/>
          </Channel>

          <Valve className="org.apache.catalina.ha.tcp.ReplicationValve"
                 filter=""/>
          <Valve className="org.apache.catalina.ha.session.JvmRouteBinderValve"/>

          <Deployer className="org.apache.catalina.ha.deploy.FarmWarDeployer"
                    tempDir="/tmp/war-temp/"
                    deployDir="/tmp/war-deploy/"
                    watchDir="/tmp/war-listen/"
                    watchEnabled="false"/>

          <ClusterListener className="org.apache.catalina.ha.session.JvmRouteSessionIDBinderListener"/>
          <ClusterListener className="org.apache.catalina.ha.session.ClusterSessionListener"/>
        </Cluster>

Tomcat 2 （IP: 192.168.3.78）
A、修改Engine节点信息： <Engine name="Catalina" defaultHost="localhost" jvmRoute="tomcat2">
B、去掉<Cluster> </Cluster> 的注释符
C、修改Cluster 节点信息（同上，只做小动修改即可）

修改web应用里面WEB-INF目录下的web.xml文件，加入标签
<distributable/>
直接加在</web-app>之前就可以了
做tomcat集群必须需要这一步，否则用户的session就无法正常使用。

注意事项：1、mcastAddr="224.0.0.0" 这主广播地址因此需要开启网卡组播功能
route add -net 224.0.0.0 netmask 240.0.0.0 dev eth0
分别在各机器上运行

四、测试集群及session同步
在webapps 下新建test 目录
目录下建WEB-INF目录下的web.xml文件
<web-app xmlns="http://java.sun.com/xml/ns/j2ee"
    xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
    xsi:schemaLocation="http://java.sun.com/xml/ns/j2ee http://java.sun.com/xml/ns/j2ee/web-app_2_4.xsd"
    version="2.5">
       <display-name>TomcatDemo</display-name>
       <distributable/>
</web-app>

新建test.jsp进行测试：
<%@ page contentType="text/html; charset=GBK" %>
<%@ page import="java.util.*" %>
<html><head><title>Cluster App Test</title></head>
<body>
Server Info:
<%
out.println(request.getLocalAddr() + " : " + request.getLocalPort()+"
");%>
<%
out.println("
ID " + session.getId()+"
");

String dataName = request.getParameter("dataName");
session.setAttribute("myname","session?");
if (dataName != null && dataName.length() > 0) {
     String dataValue = request.getParameter("dataValue");
     session.setAttribute(dataName, dataValue);
}
out.print("<b>Session P</b>");
Enumeration e = session.getAttributeNames();
while (e.hasMoreElements()) {
     String name = (String)e.nextElement();
     String value = session.getAttribute(name).toString();
     out.println( name + " = " + value+"
");
         System.out.println( name + " = " + value);
   }
%>
<form action="test.jsp" method="OST">
    name:<input type=text size=20 name="dataName">


    key:<input type=text size=20 name="dataValue">


    <input type=submit>
   </form>
</body>
</html>

完成之后重启所有服务

开启所有服务访问http://web.test.com/test/test.jsp进行测试，如图：

查看tomcat2日志，如图：

由于在测试过程中，发现一次访问tomcat2之后要很长时间才能将请求重定向到tomcat1 上（大概是sticky_session的作用），故在这里我们采用将tomcat2直接停掉的方法来迫使将请求定向到tomcat1上，看能不能实现session的同步。
将tomcat2停掉之后，我们再来访问http://web.test.com/test/test.jsp，看！已经访问的是tomcat1了

查看tomcat1日志

到这里，已基本完成