Questions[mod-jk](server)

I am getting the following Exception from time to time when my users are connecting with their Android devices to out JSON Rest Service:

java.io.IOException: Socket read failed
       at org.apache.coyote.ajp.AjpProcessor.read(AjpProcessor.java:313)
       at org.apache.coyote.ajp.AjpProcessor.readMessage(AjpProcessor.java:364)
       at org.apache.coyote.ajp.AjpProcessor.receive(AjpProcessor.java:331)
       at org.apache.coyote.ajp.AbstractAjpProcessor.refillReadBuffer(AbstractAjpProcessor.java:614)
       at org.apache.coyote.ajp.AbstractAjpProcessor$SocketInputBuffer.doRead(AbstractAjpProcessor.java:1065)
       at org.apache.coyote.Request.doRead(Request.java:422)
       at org.apache.catalina.connector.InputBuffer.realReadBytes(InputBuffer.java:290)
       at org.apache.tomcat.util.buf.ByteChunk.substract(ByteChunk.java:431)
       at org.apache.catalina.connector.InputBuffer.read(InputBuffer.java:315)
       at org.apache.catalina.connector.CoyoteInputStream.read(CoyoteInputStream.java:200)
       at org.codehaus.jackson.impl.ByteSourceBootstrapper.ensureLoaded(ByteSourceBootstrapper.java:340)
       at org.codehaus.jackson.impl.ByteSourceBootstrapper.detectEncoding(ByteSourceBootstrapper.java:137)
       at org.codehaus.jackson.impl.ByteSourceBootstrapper.constructParser(ByteSourceBootstrapper.java:197)
       at org.codehaus.jackson.JsonFactory._createJsonParser(JsonFactory.java:542)
       at org.codehaus.jackson.JsonFactory.createJsonParser(JsonFactory.java:389)
       at org.codehaus.jackson.map.ObjectMapper.readValue(ObjectMapper.java:1454)
       at org.springframework.http.converter.json.MappingJacksonHttpMessageConverter.readInternal(MappingJacksonHttpMessageConverter.java:124)
       at org.springframework.http.converter.AbstractHttpMessageConverter.read(AbstractHttpMessageConverter.java:153)
       at org.springframework.web.bind.annotation.support.HandlerMethodInvoker.readWithMessageConverters(HandlerMethodInvoker.java:641)
       at org.springframework.web.bind.annotation.support.HandlerMethodInvoker.resolveRequestBody(HandlerMethodInvoker.java:605)
       at org.springframework.web.bind.annotation.support.HandlerMethodInvoker.resolveHandlerArguments(HandlerMethodInvoker.java:354)
       at org.springframework.web.bind.annotation.support.HandlerMethodInvoker.invokeHandlerMethod(HandlerMethodInvoker.java:171)
       at org.springframework.web.servlet.mvc.annotation.AnnotationMethodHandlerAdapter.invokeHandlerMethod(AnnotationMethodHandlerAdapter.java:436)
       at org.springframework.web.servlet.mvc.annotation.AnnotationMethodHandlerAdapter.handle(AnnotationMethodHandlerAdapter.java:424)
       at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:923)
       at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:852)
       at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:882)
       at org.springframework.web.servlet.FrameworkServlet.doPost(FrameworkServlet.java:789)

I am running

• Apache/2.2.20
• mod_jk 1.2.36
• Tomcat 7.0.27

Could this have something to do with a timeout? Like the client closes the connection and the server is not able to read from the socket anymore? Or is it more likely a problem of the server configuration?

UPDATE:

Doing some Apache log awk I found the following Apache log lines corresponding to those Tomcat log exceptions:

192.186.30.116 - - [25/Jun/2012:10:47:14 +0200] "POST /myapp/methodX HTTP/1.1" 400 145 "-" "clientApp BlackBerry9360/7.0.0.530 VendorID/168" 10012655 
192.186.30.120 - - [25/Jun/2012:10:53:13 +0200] "POST /myapp/methodY HTTP/1.1" 400 145 "-" "clientApp BlackBerry9800/6.0.0.668 VendorID/124" 10012164 
192.186.30.116 - - [25/Jun/2012:10:53:36 +0200] "POST /myapp/methodZ HTTP/1.1" 400 145 "-" "clientApp BlackBerry9360/7.0.0.530 VendorID/168" 10012677 
192.82.68.16 - - [25/Jun/2012:11:22:31 +0200] "POST /myapp/methodX HTTP/1.1" 400 145 "-" "clientApp BlackBerry9930/7.1.0.402 VendorID/104" 10012667 
192.82.68.16 - - [25/Jun/2012:11:23:21 +0200] "POST /myapp/methodZ HTTP/1.1" 400 145 "-" "clientApp BlackBerry9930/7.1.0.402 VendorID/104" 10012562 

You can see the status code 400 Bad Request sent to the clients for those requests.

The large numbers at the end of the line (e.g. 10012562) show the time the request was processed on the server in microseconds: 10012562 = about 10 seconds

Looks like the connection is somehow terminated after ten seconds? I looked into the AJP configuration but there is no timeout defined - 10 seconds would be the timeout for async requests which I am not using

I am trying Tomcat Clustering with mod_jk for months and so far not so bad but facing a problem during deployment. I am using FarmDeployer to copy and deploy the WAR to other nodes in the cluster but most of the time the WAR is not deployed properly and thus leaving the page in 404 error. Even after removing the exploded war directory and again having tomcat extracted the WAR, the browser couldn't render the actual site until I restart/stop the tomcat service on that particular node(of course, http://node-ip/myapp works if redeployed war but not http://site1.mydomain.net once rendered 404 page). And also I think this problem is browser related(tried all the browsers) as the page rendered on other computers when redeployed after 404 error. I Also tried fail_on_status and so it puts the nodes to error stage which ever render 404 http status and redirect to other node BUT on my testing I found that it completely puts those nodes to error state and no request is sent to those nodes until restart though they are back working.

Workers.properties on load balancer:

workers.tomcat_home=/usr/share/tomcat
workers.java_home=/usr/lib/jvm/java-6-openjdk
ps=/
worker.list=cluster,balancer1,status

worker.balancer1.port=8009        
worker.balancer1.host=localhost
worker.balancer1.type=ajp13
worker.balancer1.lbfactor=2
worker.balancer1.cache_timeout=20
worker.balancer1.socket_timeout=20
#worker.balancer1.fail_on_status=-404,-503

worker.web1.port=8009        
worker.web1.host=192.168.1.8
worker.web1.type=ajp13
worker.web1.lbfactor=4
worker.web1.redirect=web2
worker.web1.cache_timeout=20
worker.web1.socket_timeout=20 
#worker.web1.fail_on_status=-404,-503

worker.web2.port=8009        
worker.web2.host=192.168.1.9
worker.web2.type=ajp13
worker.web2.lbfactor=4
worker.web2.redirect=web1
worker.web2.cache_timeout=20
worker.web2.socket_timeout=20 
#worker.web2.fail_on_status=-404,503

worker.cluster.type=lb
worker.cluster.balance_workers=web1,web2,balancer1
worker.cluster.sticky_session=True
worker.cluster.sticky_session_force=False

# Status worker for managing load balancer
worker.status.type=status

Anybody has any idea to skip 404 error node and instead hit other properly deployed nodes?. Atleast any tips in configuration if anything so that it renders the actual page after facing 404 having stickey session enabled.

Update:1

Apache Virtual Hosting on Load balancer(192.168.1.5 or balancer1):

<VirtualHost *:80>
ServerName site1.mydomain.net
JkAutoAlias /usr/share/tomcat/webapps/myapp
DocumentRoot /usr/share/tomcat/webapps/myapp

JkMount / cluster
JkMount /* cluster
JkMount /*.jsp cluster
  JkUnMount /myapp/*.html cluster
  JkUnMount /myapp/*.jpg  cluster
  JkUnMount /myapp/*.gif  cluster
  JkUnMount /myapp/*.png  cluster
  JkUnMount /myapp/*.css  cluster 

JkUnMount /abc cluster
JkUnMount /abc/* cluster
  JkUnMount /*.html cluster
  JkUnMount /*.jpg  cluster
  JkUnMount /*.gif  cluster
  JkUnMount /*.png  cluster
  JkUnMount /*.css  cluster

ProxyRequests Off
ProxyPreserveHost On
ProxyVia On 
<Proxy balancer://ajpCluster/>
    Order deny,allow
    Allow from all
  BalancerMember ajp://192.168.1.8:8009/ route=web1 ttl=60 timeout=20 retry=10
  BalancerMember ajp://192.168.1.9:8009/ route=web2 ttl=60 timeout=20 retry=10
  BalancerMember ajp://192.168.1.5:8009/ route=balancer1 status=+H ttl=60 

  ProxySet lbmethod=byrequests
  ProxySet stickysession=JSESSIONID|jsessionid
</Proxy>

<Location />
  ProxyPass balancer://ajpCluster/ nofailover=off
  ProxyPassReverse balancer://ajpCluster/
</Location>

</VirtualHost>

Tomcat virtual Hosting common on all the nodes:

<Host name="localhost"  appBase="webapps"
            unpackWARs="true" autoDeploy="true" deployOnStartup="true">
 <Valve className="org.apache.catalina.valves.AccessLogValve" directory="logs"
               prefix="localhost_access_log." suffix=".txt"
               pattern="%h %l %u %t &quot;%r&quot; %s %b" />

       </Host>

<Host name="site1.mydomain.net" debug="0" appBase="webapps" unpackWARs="false" autoDeploy="false" deployOnStartup="false">
<Logger className="org.apache.catalina.logger.FileLogger" directory="logs" prefix="virtual_log1." suffix=".log" timestamp="true"/>
<Context path="" docBase="/usr/share/tomcat/webapps/myapps" debug="0" reloadable="true"/>

NO session replication with tomcat clustering: Disabled for now by commenting <cluster> element as it's consuming lot of memory updating and interacting all time one another in the cluster. For now I have Load balancing and Auto Failover with mod_jk or proxy_ajp BUT with 404 error problem when myapp is unavailable(and available again) as described above. How everybody handling this?

Disclaimier: Not a server admin

I've been scratching my head over this for weeks now (not consistently mind you, as that would be maddening). I've been trying to connect my apache2 server to my tomcat server to the point where if someone encounters *.jsp or any servelet in navigating my web directory, it's handed over to tomcat.

I have both Apache2.0 (port 9099) and Tomcat6 (9089) running on Debian lenny on the same box.

Currently, mod_jk is enabled with mod_jk.conf in $apacheHOME/mods-enabled/ with content:

    # Where to find workers.properties
JkWorkersFile /etc/apache2/workers.properties
    # Where to put jk shared memory
JkShmFile     /var/log/at_jk/mod_jk.shm
    # Where to put jk logs
JkLogFile     /var/log/at_jk/mod_jk.log
    # Set the jk log level [debug/error/info]
JkLogLevel    info
    # Select the timestamp log format
JkLogStampFormat "[%a %b %d %H:%M:%S %Y] "
    # Send servlet for context /examples to worker named worker1
JkMount  /*/servlet/* worker1
    # Send JSPs  for context /examples to worker named worker1
JkMount  /*.jsp worker1

my workers.properties located in $apacheHOME/ with content:

workers.tomcat_home=/var/lib/tomcat6
workers.java_home=/usr/lib/jdk1.6.0_23/db/
worker.list=worker1
ps=/

worker.worker1.port=9081
worker.worker1.host=localhost
worker.worker1.type=ajp13

my web.xml in $tomcatHOME/conf has the following servlets enabled

<servlet>
    <servlet-name>default</servlet-name>
    <servlet-class>org.apache.catalina.servlets.DefaultServlet</servlet-cla$
    <init-param>
        <param-name>debug</param-name>
        <param-value>0</param-value>
    </init-param>
    <init-param>
        <param-name>listings</param-name>
        <param-value>false</param-value>
    </init-param>
    <load-on-startup>1</load-on-startup>
</servlet>


<servlet>
    <servlet-name>jsp</servlet-name>
    <servlet-class>org.apache.jasper.servlet.JspServlet</servlet-class>
    <init-param>
        <param-name>fork</param-name>
        <param-value>false</param-value>
    </init-param>
    <init-param>
        <param-name>xpoweredBy</param-name>
        <param-value>false</param-value>
    </init-param>
    <load-on-startup>3</load-on-startup>
</servlet>

<servlet-mapping>
    <servlet-name>jsp</servlet-name>
    <url-pattern>*.jsp</url-pattern>
</servlet-mapping>

<session-config>
    <session-timeout>30</session-timeout>
</session-config>

server.xml :

--> -->

<!-- Define an AJP 1.3 Connector on port 8009 -->

<Connector port="9081" protocol="AJP/1.3" redirectPort="9088" />

mod_jk.log :

[Mon Feb 21 13:49:35 2011] [2921:3075307264] [info] jk_open_socket::jk_connect.c (566): connect to 127.0.0.1:9081 failed (errno=111)
[Mon Feb 21 13:49:35 2011] [2921:3075307264] [info] ajp_connect_to_endpoint::jk_ajp_common.c (869): Failed opening socket to (127.0.0.1:9081) (errno=111)
[Mon Feb 21 13:49:35 2011] [2921:3075307264] [error] ajp_send_request::jk_ajp_common.c (1359): (worker1) connecting to backend failed. Tomcat is probably not started or is listening on the wrong port (errno=111)
[Mon Feb 21 13:49:35 2011] [2921:3075307264] [info] ajp_service::jk_ajp_common.c (2194): (worker1) sending request to tomcat failed (recoverable), because of error during request sending (attempt=1)
[Mon Feb 21 13:49:35 2011] [2921:3075307264] [info] jk_open_socket::jk_connect.c (566): connect to 127.0.0.1:9081 failed (errno=111)
[Mon Feb 21 13:49:35 2011] [2921:3075307264] [info] ajp_connect_to_endpoint::jk_ajp_common.c (869): Failed opening socket to (127.0.0.1:9081) (errno=111)
[Mon Feb 21 13:49:35 2011] [2921:3075307264] [error] ajp_send_request::jk_ajp_common.c (1359): (worker1) connecting to backend failed. Tomcat is probably not started or is listening on the wrong port (errno=111)
[Mon Feb 21 13:49:35 2011] [2921:3075307264] [info] ajp_service::jk_ajp_common.c (2194): (worker1) sending request to tomcat failed (recoverable), because of error during request sending (attempt=2)
[Mon Feb 21 13:49:35 2011] [2921:3075307264] [error] ajp_service::jk_ajp_common.c (2212): (worker1) Connecting to tomcat failed. Tomcat is probably not started or is listening on the wrong port
[Mon Feb 21 13:49:35 2011] [2921:3075307264] [info] jk_handler::mod_jk.c (2364): Service error=-3 for worker=worker1

There appears to be an issue with port 9081, no process is using it, so I would suppose it's configured incorrectly. Though my tomcat port works fine.

 ickronia:/var/log/at_jk# telnet localhost 9081
 Trying ::1...
 Trying 127.0.0.1...
 telnet: Unable to connect to remote host: Connection refused
 ickronia:/var/log/at_jk#

From what I can tell, there's no funny buisness as both the apache2, tomcat, and mod_jk logs show green; yet whenever I navigate to a jsp, it simply displays the javascript.

I'm unsure what the problem is exactly despite pouring over the logs and documentation for aid. I'm quite a greenhorn in the servelet world.

I really have no idea...

I have a java web application which stores the session id in URL rather than using cookies. I want to run the application on 2 tomcat servers and fronting them with an Apache server for load balancing.

            __ tomcat1
Apache ----/
           \__ tomcat2

By setting the jvmRoute to tomcat1in the Tomcat server.xml file, the url will become http://url;jsessionid=id.tomcat1. So it is possible to achieve session stickyness by looking at the url, and route it to the corresponding server.

I know that mod_proxy and mod_jk and do the load balancing, but their sticky session only work when the session id is stored in cookies. In my case, the session id is encoded in the URL. How can I do the load balancing?

My first question on here on SF so please forgive me if I manage to bork the post. :)

Anyways, I'm using mod_rewrite on one of my machines with a simple rule that redirects to a webapp on another machine. I'm also setting the header 'Access-Control-Allow-Origin' on both machines. The problem is that when I hit the rewrite rule, I loose the 'Access-Control-Allow-Origin' header setting.

Here's an example of the Apache config for the first machine:

NameVirtualHost 10.0.0.2:80
<VirtualHost 10.0.0.2:80>
   DocumentRoot /var/www/host.example.com
   ServerName host.example.com
   JkMount /webapp/* jkworker
   Header set Access-Control-Allow-Origin "*"
   RewriteEngine on
   RewriteRule   ^/otherhost  http://otherhost.example.com/webapp [R,L]
</VirtualHost>

And here's an example of the Apache config for the second:

NameVirtualHost 10.0.1.2:80
<VirtualHost 10.0.1.2:80>
   DocumentRoot /var/www/otherhost.example.com
   ServerName otherhost.example.com
   JkMount /webapp/* jkworker
   Header set Access-Control-Allow-Origin "*"
</VirtualHost>

When I hit host.example.com we see that the header is set:

$ curl -i http://host.example.com/
HTTP/1.1 302 Moved Temporarily
Server: Apache/2.2.11 (FreeBSD) mod_ssl/2.2.11 OpenSSL/0.9.7e-p1 DAV/2 mod_jk/1.2.26
Content-Length: 0
Access-Control-Allow-Origin: *
Content-Type: text/html;charset=ISO-8859-1

And when I hit otherhost.example.com we see that it too is setting the header:

$ curl -i http://otherhost.example.com
HTTP/1.1 200 OK
Server: Apache/2.0.46 (Red Hat)
Location: http://otherhost.example.com/index.htm
Content-Length: 0
Access-Control-Allow-Origin: *
Content-Type: text/html;charset=UTF-8

But when I try to hit the rewrite rule at host.example.com/otherhost we get no love:

$ curl -i http://host.example.com/otherhost/
HTTP/1.1 302 Found
Server: Apache/2.2.11 (FreeBSD) mod_ssl/2.2.11 OpenSSL/0.9.7e-p1 DAV/2 mod_jk/1.2.26
Location: http://otherhost.example.com/
Content-Length: 0
Content-Type: text/html; charset=iso-8859-1

Can anybody point out what I'm doing wrong here? Could mod_jk be part of the problem?