Java Mailing List Archive

http://www.junlu.com/

Home » users-digest.tomcat »

users Digest 14 Mar 2013 20:41:17 -0000 Issue 11296

users-digest-help

2013-03-14


Author LoginPost Reply

users Digest 14 Mar 2013 20:41:17 -0000 Issue 11296

Topics (messages 240377 through 240393)

Re: RE:tomcat 6.0.35 in production maintaince
 240377 by: fachhoch

Re: AW:AJP suddenly Stopps acting: ajp on 7009 and 9009 : connections keept open
 240378 by: André Warnier

Re: tomcat 6.0.35 in production maintaince
 240379 by: Daniel Mikusa

Re: AJP suddenly Stopps acting: ajp on 7009 and 9009 : connections keept open
 240380 by: Mark Thomas

Procrun and Tomcat service/OS shutdown on Windows
 240381 by: Thomas, Steve
 240383 by: Harris, Jeffrey E.
 240384 by: Caldarale, Charles R
 240386 by: Thomas, Steve
 240387 by: Thomas, Steve
 240390 by: Konstantin Kolinko
 240391 by: Harris, Jeffrey E.

configuring tomcat7 with apache 2.2.22
 240382 by: solar.essnmag.com
 240385 by: Harris, Jeffrey E.
 240392 by: André Warnier

Re: Deadlock when using jetty 8 JDBCSessionManager and Tomcat 7 JDBC Connector
 240388 by: Colin Ingarfield

NullPointerException in MapperListener; Tomcat#start() does not create a Container?
 240389 by: Nick Williams

tomcat dead, service won't start
 240393 by: solar.essnmag.com

Administrivia:

---------------------------------------------------------------------
To post to the list, e-mail: users@(protected)
To unsubscribe, e-mail: users-digest-unsubscribe@(protected)
For additional commands, e-mail: users-digest-help@(protected)

----------------------------------------------------------------------


Attachment: users_240377.eml (zipped)
mine app is running behind  amazon load balancer may be its ipaddress from
that aws , I dont know , but all I care is not to create session , to aws
load balancer I gave the url to ping my app which is a servlet  and this
forwards to jsp in that jsp I have
<%@(protected)" %>

this should not create a session , may be  some thing else is creating
a new session.


is there any way I can figure out what was the request resource being
called every few minutes and which request resource is creating new session
?





--
Sent from the Tomcat - User mailing list archive at Nabble.com.


Attachment: users_240378.eml (zipped)
David Kumar wrote:
> Hey,
>
> thanks for note..
> Attached you can find a new list.
> So, java is keeping these connections in close_wait.
>

Yes. Whether this is the java that runs tomcat is not immediately evident.
But it also seems that they concern port numbers like 7009, 9009 etc. which must be your
AJP Connectors, so let's assume the CLOSE-WAITs all involve Tomcat.

The first observation hat I would make is that this looks like a pretty busy system.

Then, there are 2100+ lines in that list. Let's assume first that they are all internal
connections (which probable they are not, but for simplicity), so that there are actually
2 lines for the same connection, like these :

tcp     0    0 127.0.0.1:39872      127.0.0.1:7009       FIN_WAIT2
20766/apache2
...
tcp6     1    0 127.0.0.1:7009       127.0.0.1:39872      CLOSE_WAIT 20564/java

..

So let's say that we have in total about 1000 connections active in some state.
Compared to that, there are 54 connections in CLOSE_WAIT, which is like 5%.
That does not really sound "pathological" to me, considering that CLOSE_WAIT is a normal
state through which any TCP connection goes at some point.

If you look again at the above connection, and if I am not confusing my clients and
servers again, it looks here like :
- Apache http (PID 20766), as a client, has established a connection to port 7009 of the
server Tomcat (PID 20564). That must be an AJP connection, from the mod_jk module of
Apache, to the AJP Connector in Tomcat.
- A while later, Apache closes the connection. It sends a "FIN" packet to Tomcat, and
waits for Tomcat to acknowledge this FIN packet.
- when it receives the FIN packet from Apache, Tomcat responds with an ACK, and then goes
to the CLOSE_WAIT state.
- when Apache receives the ACK from Tomcat, it goes to state FIN_WAIT2.
 (and that is the current state of the above connection)
- now Tomcat is supposed to send a FIN to Apache, and receive an ACK in return.
- if Apache receives the FIN from Tomcat, it sends back an ACK, and goes into state
TIME_WAIT (where it doesn't expect anything anymore), and then after a short moment, it
discards that connection.

If there are connections that remain in the CLOSE_WAIT state for very long, it means that
something in the last 2 steps above is not working.

Now in your listing, we see about 1900 lines in the TIME_WAIT state, all of them *to* an
AJP port of Tomcat. This seems to suggest that the Apache side is working as it should.
But if the Tomcat side stays in CLOSE_WAIT for a long time, then it would suggest that on
the Tomcat side, that connection is never properly close()'d.

But this is the request/response connection, so I doubt that there would be a bug there,
otherwise we'd have problems reports filling this list every day.

Might there be that there is somewhere a discrepancy between the keep-alive settings,
between Apache and Tomcat ?

or maybe your server is just not strong enough for the load you are putting on it ?




>
>
> How about a "netstat -t -pan" ? it's a bit easier to find the relevant connections.
> If needed, tell us which processes are Tomcat instances/threads and which are Apache httpd
> (if it is not evident from the list).
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@(protected)
> For additional commands, e-mail: users-help@(protected)
>
>
> Mit freundlichen Grüßen
> David Kumar
> Softwareentwickler, B. Sc.
> Abteilung Infotech - Interaktiv
> TELESTAR-DIGITAL GmbH
> Am Weiher 14
> D-56766 Ulmen
>
> http://www.telestar.de/
>
>
>
> ------------------------------------------------------------------------
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@(protected)
> For additional commands, e-mail: users-help@(protected)



Attachment: users_240379.eml (zipped)
On Mar 14, 2013, at 10:56 AM, fachhoch wrote:

> mine app is running behind  amazon load balancer may be its ipaddress from
> that aws , I dont know , but all I care is not to create session , to aws
> load balancer I gave the url to ping my app which is a servlet  and this
> forwards to jsp in that jsp I have
> <%@(protected)" %>
>
> this should not create a session , may be  some thing else is creating
> a new session.

That's certainly possible.

> is there any way I can figure out what was the request resource being
> called every few minutes

Configure an AccessLogValve if you don't have one already. Then look at the log to see what URL the IP address is requesting.

https://tomcat.apache.org/tomcat-6.0-doc/config/valve.html#Access_Log_Valve

> and which request resource is creating new session

Look at your code. Setting "session=false" in your JSP will simply stop the JSP from creating a session by default. If your code attempts to accesses the session at any point, that will also cause a session to be created.

Dan


>
>
>
>
>
> --
> View this message in context: http://tomcat.10.n6.nabble.com/tomcat-6-0-35-in-production-maintaince-tp4995740p4995908.html
> Sent from the Tomcat - User mailing list archive at Nabble.com.
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@(protected)
> For additional commands, e-mail: users-help@(protected)


Attachment: users_240380.eml (zipped)
On 14/03/2013 13:10, David Kumar wrote:

> I tried to just restart the Apache, didn't work. As suggested I
> tried to use the option -DisableReuse. That did make our situation a
> lot more horrible instead of days / week we it took just minutes to
> shut all AJP ports down.

Interesting.

If the problem was too many sockets in CLOSE_WAIT, consider looking at
the connectionLinger setting on your AJP connector's in Tomcat.

Mark


Attachment: users_240381.eml (zipped)
Hi -

Running Tomcat 7.0.23 or 7.0.37 (32 or 64-bit) installed as a service (either via service.bat or the exe installer) on a Windows 7 64-bit OS, we are seeing an issue where the Windows shutdown kills Tomcat before our webapp shutdown sequence has time to execute fully. (Specifically, we just want to make sure our instance of HSQLDB shuts down correctly, otherwise corruption can ensue.)

Details:

Initially we were running with 32-bit Tomcat 7.0.23 and saw that our shutdown sequence was not being logged at all when one of our customers shut down his laptop. It looked like the process was just being killed. I found a commons-daemon/procrun bug and corresponding fix that seemed like it should address the issue, namely

http://stackoverflow.com/questions/13578196/how-to-gracefully-shutdown-procrun/14150785#14150785

https://issues.apache.org/jira/browse/DAEMON-274

I subsequently upgraded Tomcat to 7.0.36 (32-bit, zip) to get the updated commons-daemon (http://tomcat.apache.org/tomcat-7.0-doc/changelog.html), but to no avail. I thought perhaps the 32-bit Tomcat/64-bit OS might be a disconnect, so I installed the 64-bit version, but got the same result. In short, it looks like we're either doing something wrong with our code, or there's a new wrinkle in the OS-service handshaking, or the bug wasn't fixed correctly...maybe in that order.

Details below on how our code manifested the problem as well as other steps to reproduce.

Our database shutdown code is located in the destroy() function of a class implementing org.springframework.beans.factory.DisposableBean. I added a Thread.sleep(5 min) call to reproduce it on my machine. As long as I shut down the service through the Services panel on Windows, the shutdown sequence fully executes (and takes 5 min, as expected). But if I just shut down Windows, the sequence is interrupted.

(As an aside, I don't expect the shutdown to take anywhere near that long in practice, but wanted to make sure the problem manifested itself so that I could address the bug. We are seeing this with a decent customer test machine, but I can't reproduce it on my machine w/o changes.)

Thinking it might be Spring, I moved the shutdown delay to a ServletContextListener. contextDestroyed() method. Same effect.

I moved the delay again, and reproduced the same problem in a standalone servlet that overrides HttpServlet.destroy(). I've posted the code at the link below:

http://pastebin.com/yYgrQ2sE

This is the output recorded in the stdout log file for an OS shutdown, and then a shutdown of the service by hand. We, of course, favor the second. :-)

2013-03-14 10:05:40 Commons Daemon procrun stdout initialized
StatusServlet.init()
Entering StatusServlet.destroy()
Simulating long shutdown sequence.

2013-03-14 10:12:29 Commons Daemon procrun stdout initialized
StatusServlet.init()
Entering StatusServlet.destroy()
Simulating long shutdown sequence.
Simulation complete--sequence finished.
Exiting StatusServlet.destroy()

Can we guarantee that Windows won't just kill our Tomcat process and potentially corrupt our database? That's the question.

I'd be grateful for some help on this. Thanks for your time and attention.

Steve T
This message is intended only for the named recipient. If you are not the intended recipient, you are notified that disclosing, copying, distributing or taking any action based on the contents of this information is strictly prohibited.



Attachment: users_240383.eml (zipped)
Edit the registry so Tomcat depends on the HSQLDB shutdown. This only works if HSQLDB is also started as a service.

Jeffrey Harris

> -----Original Message-----
> From: Thomas, Steve [mailto:sthomas@(protected)]
> Sent: Thursday, March 14, 2013 12:00 PM
> To: users@(protected)
> Subject: Procrun and Tomcat service/OS shutdown on Windows
>
> Hi -
>
> Running Tomcat 7.0.23 or 7.0.37 (32 or 64-bit) installed as a service
> (either via service.bat or the exe installer) on a Windows 7 64-bit OS,
> we are seeing an issue where the Windows shutdown kills Tomcat before
> our webapp shutdown sequence has time to execute fully. (Specifically,
> we just want to make sure our instance of HSQLDB shuts down correctly,
> otherwise corruption can ensue.)
>
> Details:
>
> Initially we were running with 32-bit Tomcat 7.0.23 and saw that our
> shutdown sequence was not being logged at all when one of our customers
> shut down his laptop. It looked like the process was just being
> killed. I found a commons-daemon/procrun bug and corresponding fix
> that seemed like it should address the issue, namely
>
> http://stackoverflow.com/questions/13578196/how-to-gracefully-shutdown-
> procrun/14150785#14150785
>
> https://issues.apache.org/jira/browse/DAEMON-274
>
> I subsequently upgraded Tomcat to 7.0.36 (32-bit, zip) to get the
> updated commons-daemon (http://tomcat.apache.org/tomcat-7.0-
> doc/changelog.html), but to no avail. I thought perhaps the 32-bit
> Tomcat/64-bit OS might be a disconnect, so I installed the 64-bit
> version, but got the same result. In short, it looks like we're either
> doing something wrong with our code, or there's a new wrinkle in the
> OS-service handshaking, or the bug wasn't fixed correctly...maybe in
> that order.
>
> Details below on how our code manifested the problem as well as other
> steps to reproduce.
>
> Our database shutdown code is located in the destroy() function of a
> class implementing org.springframework.beans.factory.DisposableBean. I
> added a Thread.sleep(5 min) call to reproduce it on my machine. As
> long as I shut down the service through the Services panel on Windows,
> the shutdown sequence fully executes (and takes 5 min, as expected).
> But if I just shut down Windows, the sequence is interrupted.
>
> (As an aside, I don't expect the shutdown to take anywhere near that
> long in practice, but wanted to make sure the problem manifested itself
> so that I could address the bug. We are seeing this with a decent
> customer test machine, but I can't reproduce it on my machine w/o
> changes.)
>
> Thinking it might be Spring, I moved the shutdown delay to a
> ServletContextListener. contextDestroyed() method. Same effect.
>
> I moved the delay again, and reproduced the same problem in a
> standalone servlet that overrides HttpServlet.destroy(). I've posted
> the code at the link below:
>
> http://pastebin.com/yYgrQ2sE
>
> This is the output recorded in the stdout log file for an OS shutdown,
> and then a shutdown of the service by hand. We, of course, favor the
> second. :-)
>
> 2013-03-14 10:05:40 Commons Daemon procrun stdout initialized
> StatusServlet.init()
> Entering StatusServlet.destroy()
> Simulating long shutdown sequence.
>
> 2013-03-14 10:12:29 Commons Daemon procrun stdout initialized
> StatusServlet.init()
> Entering StatusServlet.destroy()
> Simulating long shutdown sequence.
> Simulation complete--sequence finished.
> Exiting StatusServlet.destroy()
>
> Can we guarantee that Windows won't just kill our Tomcat process and
> potentially corrupt our database? That's the question.
>
> I'd be grateful for some help on this. Thanks for your time and
> attention.
>
> Steve T
> This message is intended only for the named recipient. If you are not
> the intended recipient, you are notified that disclosing, copying,
> distributing or taking any action based on the contents of this
> information is strictly prohibited.
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@(protected)
> For additional commands, e-mail: users-help@(protected)

Edit the service entry in the registry (under HKEY_Local_Machine\system\currentcontrolset\services\<Tomcat Service Name>) so Tomcat depends on HSQLDB. This only works if HSQLDB is also started as a service. If HSQLDB is started some other way (i.e., by the Tomcat web app), you can try and install it as a service using the srvany utility (or possibly the sc utility).

If you can configure Tomcat to be dependent on HSQLDB, this will also force HSQLDB to start before Tomcat.

There is a method discussed at http://blogs.technet.com/b/askperf/archive/2008/02/04/ws2008-service-shutdown-and-crash-handling.aspx that you might try.

Finally you might also want to try delaying the shutdown timer on the system to give Tomcat and/or HSQLDB more time to shutdown. It might be possible that it is taking longer than the 12 seconds Windows allows by default for a service to shutdown. That timer can be changed at HKEY_LOCAL_MACHINE\System\CurrentControlSet\Control (WaitToKillServiceTimeout value; the data is in milliseconds). However, Windows 7 and Windows Server 2008 R2 need a hotfix to change this setting (see http://support.microsoft.com/kb/2549760).

Jeffrey Harris

This e-mail and any attachments are intended only for the use of the addressee(s) named herein and may contain proprietary information. If you are not the intended recipient of this e-mail or believe that you received this email in error, please take immediate action to notify the sender of the apparent error by reply e-mail; permanently delete the e-mail and any attachments from your computer; and do not disseminate, distribute, use, or copy this message and any attachments.


Attachment: users_240384.eml (zipped)
> From: Thomas, Steve [mailto:sthomas@(protected)]
> Subject: Procrun and Tomcat service/OS shutdown on Windows

> Can we guarantee that Windows won't just kill our Tomcat process and potentially
> corrupt our database?

If the integrity of your database depends on an orderly shutdown sequence, you need a better database. What happens if there's a power outage? What about some other abrupt hardware or software failure?

- Chuck


THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY MATERIAL and is thus for use only by the intended recipient. If you received this in error, please contact the sender and delete the e-mail and its attachments from all computers.



Attachment: users_240386.eml (zipped)
Unfortunately, that appears to be the case with HSQLDB. We maintain a set of nightly backups to address cases like those you've cited; however, if we can avoid issues arising from just shutting down the OS, that would help.

Regards,

Steve

-----Original Message-----
From: Caldarale, Charles R [mailto:Chuck.Caldarale@(protected)]
Sent: Thursday, March 14, 2013 1:00 PM
To: Tomcat Users List
Subject: RE: Procrun and Tomcat service/OS shutdown on Windows

> From: Thomas, Steve [mailto:sthomas@(protected)]
> Subject: Procrun and Tomcat service/OS shutdown on Windows

> Can we guarantee that Windows won't just kill our Tomcat process and
> potentially corrupt our database?

If the integrity of your database depends on an orderly shutdown sequence, you need a better database. What happens if there's a power outage? What about some other abrupt hardware or software failure?

- Chuck


THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY MATERIAL and is thus for use only by the intended recipient. If you received this in error, please contact the sender and delete the e-mail and its attachments from all computers.


---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@(protected)
For additional commands, e-mail: users-help@(protected)

This message is intended only for the named recipient. If you are not the intended recipient, you are notified that disclosing, copying, distributing or taking any action based on the contents of this information is strictly prohibited.



Attachment: users_240387.eml (zipped)
Thanks, Jeffrey. That may be a possibility for the long-term. --Steve

-----Original Message-----
From: Harris, Jeffrey E. [mailto:Jeffrey.Harris@(protected)]
Sent: Thursday, March 14, 2013 12:52 PM
To: Tomcat Users List
Subject: RE: Procrun and Tomcat service/OS shutdown on Windows

Edit the registry so Tomcat depends on the HSQLDB shutdown. This only works if HSQLDB is also started as a service.

Jeffrey Harris

> -----Original Message-----
> From: Thomas, Steve [mailto:sthomas@(protected)]
> Sent: Thursday, March 14, 2013 12:00 PM
> To: users@(protected)
> Subject: Procrun and Tomcat service/OS shutdown on Windows
>
> Hi -
>
> Running Tomcat 7.0.23 or 7.0.37 (32 or 64-bit) installed as a service
> (either via service.bat or the exe installer) on a Windows 7 64-bit
> OS, we are seeing an issue where the Windows shutdown kills Tomcat
> before our webapp shutdown sequence has time to execute fully.
> (Specifically, we just want to make sure our instance of HSQLDB shuts
> down correctly, otherwise corruption can ensue.)
>
> Details:
>
> Initially we were running with 32-bit Tomcat 7.0.23 and saw that our
> shutdown sequence was not being logged at all when one of our
> customers shut down his laptop. It looked like the process was just
> being killed. I found a commons-daemon/procrun bug and corresponding
> fix that seemed like it should address the issue, namely
>
> http://stackoverflow.com/questions/13578196/how-to-gracefully-shutdown
> -
> procrun/14150785#14150785
>
> https://issues.apache.org/jira/browse/DAEMON-274
>
> I subsequently upgraded Tomcat to 7.0.36 (32-bit, zip) to get the
> updated commons-daemon (http://tomcat.apache.org/tomcat-7.0-
> doc/changelog.html), but to no avail. I thought perhaps the 32-bit
> Tomcat/64-bit OS might be a disconnect, so I installed the 64-bit
> version, but got the same result. In short, it looks like we're
> either doing something wrong with our code, or there's a new wrinkle
> in the OS-service handshaking, or the bug wasn't fixed
> correctly...maybe in that order.
>
> Details below on how our code manifested the problem as well as other
> steps to reproduce.
>
> Our database shutdown code is located in the destroy() function of a
> class implementing org.springframework.beans.factory.DisposableBean.
> I added a Thread.sleep(5 min) call to reproduce it on my machine. As
> long as I shut down the service through the Services panel on Windows,
> the shutdown sequence fully executes (and takes 5 min, as expected).
> But if I just shut down Windows, the sequence is interrupted.
>
> (As an aside, I don't expect the shutdown to take anywhere near that
> long in practice, but wanted to make sure the problem manifested
> itself so that I could address the bug. We are seeing this with a
> decent customer test machine, but I can't reproduce it on my machine
> w/o
> changes.)
>
> Thinking it might be Spring, I moved the shutdown delay to a
> ServletContextListener. contextDestroyed() method. Same effect.
>
> I moved the delay again, and reproduced the same problem in a
> standalone servlet that overrides HttpServlet.destroy(). I've posted
> the code at the link below:
>
> http://pastebin.com/yYgrQ2sE
>
> This is the output recorded in the stdout log file for an OS shutdown,
> and then a shutdown of the service by hand. We, of course, favor the
> second. :-)
>
> 2013-03-14 10:05:40 Commons Daemon procrun stdout initialized
> StatusServlet.init()
> Entering StatusServlet.destroy()
> Simulating long shutdown sequence.
>
> 2013-03-14 10:12:29 Commons Daemon procrun stdout initialized
> StatusServlet.init()
> Entering StatusServlet.destroy()
> Simulating long shutdown sequence.
> Simulation complete--sequence finished.
> Exiting StatusServlet.destroy()
>
> Can we guarantee that Windows won't just kill our Tomcat process and
> potentially corrupt our database? That's the question.
>
> I'd be grateful for some help on this. Thanks for your time and
> attention.
>
> Steve T
> This message is intended only for the named recipient. If you are not
> the intended recipient, you are notified that disclosing, copying,
> distributing or taking any action based on the contents of this
> information is strictly prohibited.
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@(protected)
> For additional commands, e-mail: users-help@(protected)

Edit the service entry in the registry (under HKEY_Local_Machine\system\currentcontrolset\services\<Tomcat Service Name>) so Tomcat depends on HSQLDB. This only works if HSQLDB is also started as a service. If HSQLDB is started some other way (i.e., by the Tomcat web app), you can try and install it as a service using the srvany utility (or possibly the sc utility).

If you can configure Tomcat to be dependent on HSQLDB, this will also force HSQLDB to start before Tomcat.

There is a method discussed at http://blogs.technet.com/b/askperf/archive/2008/02/04/ws2008-service-shutdown-and-crash-handling.aspx that you might try.

Finally you might also want to try delaying the shutdown timer on the system to give Tomcat and/or HSQLDB more time to shutdown. It might be possible that it is taking longer than the 12 seconds Windows allows by default for a service to shutdown. That timer can be changed at HKEY_LOCAL_MACHINE\System\CurrentControlSet\Control (WaitToKillServiceTimeout value; the data is in milliseconds). However, Windows 7 and Windows Server 2008 R2 need a hotfix to change this setting (see http://support.microsoft.com/kb/2549760).

Jeffrey Harris

This e-mail and any attachments are intended only for the use of the addressee(s) named herein and may contain proprietary information. If you are not the intended recipient of this e-mail or believe that you received this email in error, please take immediate action to notify the sender of the apparent error by reply e-mail; permanently delete the e-mail and any attachments from your computer; and do not disseminate, distribute, use, or copy this message and any attachments.

---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@(protected)
For additional commands, e-mail: users-help@(protected)

This message is intended only for the named recipient. If you are not the intended recipient, you are notified that disclosing, copying, distributing or taking any action based on the contents of this information is strictly prohibited.



Attachment: users_240390.eml (zipped)
2013/3/14 Thomas, Steve <sthomas@(protected)>:
> Hi -
>
> Running Tomcat 7.0.23 or 7.0.37 (32 or 64-bit) installed as a service (either via service.bat or the exe installer) on a Windows 7 64-bit OS, we are seeing an issue where the Windows shutdown kills Tomcat before our webapp shutdown sequence has time to execute fully. (Specifically, we just want to make sure our instance of HSQLDB shuts down correctly, otherwise corruption can ensue.)
>
> Details:
>
> Initially we were running with 32-bit Tomcat 7.0.23 and saw that our shutdown sequence was not being logged at all when one of our customers shut down his laptop. It looked like the process was just being killed. I found a commons-daemon/procrun bug and corresponding fix that seemed like it should address the issue, namely
>
> http://stackoverflow.com/questions/13578196/how-to-gracefully-shutdown-procrun/14150785#14150785
>
> https://issues.apache.org/jira/browse/DAEMON-274
>
> I subsequently upgraded Tomcat to 7.0.36 (32-bit, zip) to get the updated commons-daemon (http://tomcat.apache.org/tomcat-7.0-doc/changelog.html), but to no avail. I thought perhaps the 32-bit Tomcat/64-bit OS might be a disconnect, so I installed the 64-bit version, but got the same result. In short, it looks like we're either doing something wrong with our code, or there's a new wrinkle in the OS-service handshaking, or the bug wasn't fixed correctly...maybe in that order.
>
> Details below on how our code manifested the problem as well as other steps to reproduce.
>
> Our database shutdown code is located in the destroy() function of a class implementing org.springframework.beans.factory.DisposableBean. I added a Thread.sleep(5 min) call to reproduce it on my machine. As long as I shut down the service through the Services panel on Windows, the shutdown sequence fully executes (and takes 5 min, as expected). But if I just shut down Windows, the sequence is interrupted.



1. From the above I would say that is a Windows feature, that it does
not wait for service to shutdown properly.

I'd look into Windows support and MSDN, whether it is a configuration
issue or something that be workaround by proper coding.

http://support.microsoft.com/kb/146092/en-us

http://msdn.microsoft.com/en-us/library/windows/desktop/dd371756%28v=vs.85%29.aspx

2. On the "Shutdown" tab of the procrun service configuration dialog
(Tomcat7w.exe) there is "Timeout" field.

I have "0" (sec) there.

3. Does your database need 5min to shutdown?

Are you doing something unnecessary (such as compacting it instead of
a simple shutdown)?

Are you using a lot of "memory" tables that HSQL writes out as SQL,
instead of "cached" (on-disk) binary ones?

4. I have several HSQL databases running in-process in Tomcat for my
personal needs. They are configured as <Resource closeMethod="close"
url="...;shutdown=true" /> in server.xml. They shutdown together with
Tomcat (without any additional listeners) and I did not notice any
problems.

5. Is HSQL database corruptible by sudden kill? AFAIK it has own
journal and backup, and there is nothing wrong with aborting it.

The only thing that when it is properly shut down, it looks more
pretty with lesser number of files.


> (As an aside, I don't expect the shutdown to take anywhere near that long in practice, but wanted to make sure the problem manifested itself so that I could address the bug. We are seeing this with a decent customer test machine, but I can't reproduce it on my machine w/o changes.)
>
> Thinking it might be Spring, I moved the shutdown delay to a ServletContextListener. contextDestroyed() method. Same effect.
>
> I moved the delay again, and reproduced the same problem in a standalone servlet that overrides HttpServlet.destroy(). I've posted the code at the link below:
>
> http://pastebin.com/yYgrQ2sE
>
> This is the output recorded in the stdout log file for an OS shutdown, and then a shutdown of the service by hand. We, of course, favor the second. :-)
>
> 2013-03-14 10:05:40 Commons Daemon procrun stdout initialized
> StatusServlet.init()
> Entering StatusServlet.destroy()
> Simulating long shutdown sequence.
>
> 2013-03-14 10:12:29 Commons Daemon procrun stdout initialized
> StatusServlet.init()
> Entering StatusServlet.destroy()
> Simulating long shutdown sequence.
> Simulation complete--sequence finished.
> Exiting StatusServlet.destroy()
>
> Can we guarantee that Windows won't just kill our Tomcat process and potentially corrupt our database? That's the question.
>
> I'd be grateful for some help on this. Thanks for your time and attention.
>

Best regards,
Konstantin Kolinko


Attachment: users_240391.eml (zipped)


> -----Original Message-----
> From: Konstantin Kolinko [mailto:knst.kolinko@(protected)]
> Sent: Thursday, March 14, 2013 4:01 PM
> To: Tomcat Users List
> Subject: Re: Procrun and Tomcat service/OS shutdown on Windows
>
> 2013/3/14 Thomas, Steve <sthomas@(protected)>:
> > Hi -
> >
> > Running Tomcat 7.0.23 or 7.0.37 (32 or 64-bit) installed as a service
> > (either via service.bat or the exe installer) on a Windows 7 64-bit
> > OS, we are seeing an issue where the Windows shutdown kills Tomcat
> > before our webapp shutdown sequence has time to execute fully.
> > (Specifically, we just want to make sure our instance of HSQLDB shuts
> > down correctly, otherwise corruption can ensue.)
> >
> > Details:
> >
> > Initially we were running with 32-bit Tomcat 7.0.23 and saw that our
> > shutdown sequence was not being logged at all when one of our
> > customers shut down his laptop. It looked like the process was just
> > being killed. I found a commons-daemon/procrun bug and corresponding
> > fix that seemed like it should address the issue, namely
> >
> > http://stackoverflow.com/questions/13578196/how-to-gracefully-
> shutdown
> > -procrun/14150785#14150785
> >
> > https://issues.apache.org/jira/browse/DAEMON-274
> >
> > I subsequently upgraded Tomcat to 7.0.36 (32-bit, zip) to get the
> updated commons-daemon (http://tomcat.apache.org/tomcat-7.0-
> doc/changelog.html), but to no avail. I thought perhaps the 32-bit
> Tomcat/64-bit OS might be a disconnect, so I installed the 64-bit
> version, but got the same result. In short, it looks like we're either
> doing something wrong with our code, or there's a new wrinkle in the
> OS-service handshaking, or the bug wasn't fixed correctly...maybe in
> that order.
> >
> > Details below on how our code manifested the problem as well as other
> steps to reproduce.
> >
> > Our database shutdown code is located in the destroy() function of a
> class implementing org.springframework.beans.factory.DisposableBean. I
> added a Thread.sleep(5 min) call to reproduce it on my machine. As
> long as I shut down the service through the Services panel on Windows,
> the shutdown sequence fully executes (and takes 5 min, as expected).
> But if I just shut down Windows, the sequence is interrupted.
>
>
>
> 1. From the above I would say that is a Windows feature, that it does
> not wait for service to shutdown properly.
>
> I'd look into Windows support and MSDN, whether it is a configuration
> issue or something that be workaround by proper coding.
>

I proposed three options to Steve to change the Windows configuration to either allow more time, or
to make Tomcat shutdown dependent on the database shutdown, or both.

> 2. On the "Shutdown" tab of the procrun service configuration dialog
> (Tomcat7w.exe) there is "Timeout" field.
>
> I have "0" (sec) there.

A good option, but it will not necessarily override the Windows default of 12 seconds to shut down a service.

Jeffrey Harris

This e-mail and any attachments are intended only for the use of the addressee(s) named herein and may contain proprietary information. If you are not the intended recipient of this e-mail or believe that you received this email in error, please take immediate action to notify the sender of the apparent error by reply e-mail; permanently delete the e-mail and any attachments from your computer; and do not disseminate, distribute, use, or copy this message and any attachments.


Attachment: users_240382.eml (zipped)
Although not a newbie to building websites in html and php, and physical computing in C/C++, I'm having a dickens of a
time understanding the documentation of Integrating Tomcat 7 with my existing Apache 2.2.22 on WinXP (company
supplied development pc, nothing I can do about that). Can I get a bit of handholding please?

I'm trying to evaluate a java servlet that requires tomcat.


Steve Spence, KK4HFJ
http://arduinotronics.blogspot.com
http://www.essnmag.com


Attachment: users_240385.eml (zipped)
> -----Original Message-----
> From: solar@(protected)]
> Sent: Thursday, March 14, 2013 12:51 PM
> To: users@(protected)
> Subject: configuring tomcat7 with apache 2.2.22
>
> Although not a newbie to building websites in html and php, and
> physical computing in C/C++, I'm having a dickens of a time
> understanding the documentation of Integrating Tomcat 7 with my
> existing Apache 2.2.22 on WinXP (company supplied development pc,
> nothing I can do about that). Can I get a bit of handholding please?
>
> I'm trying to evaluate a java servlet that requires tomcat.
>
>
> Steve Spence, KK4HFJ
> http://arduinotronics.blogspot.com
> http://www.essnmag.com
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@(protected)
> For additional commands, e-mail: users-help@(protected)

You can install the mod_jk connector on Apache to provide ajp_13 protocol support and pass connection requests to Tomcat via Apache (for example, apache.mycompany.com/myTomcatApp would forward the request to Tomcat for processing as if the user had connected directly to Tomcat).

You need to set up a workers.properties file on Apache with the connection information and JKMap settings to tell Apache which URI to forward to Tomcat, and which servers to forward to. You need to enable the AJP_13 connector (usually on port 8009) in Tomcat to receive the forwarded requests from Apache. The default parameters are in the server.xml, and usually commented out.

Jeffrey Harris

This e-mail and any attachments are intended only for the use of the addressee(s) named herein and may contain proprietary information. If you are not the intended recipient of this e-mail or believe that you received this email in error, please take immediate action to notify the sender of the apparent error by reply e-mail; permanently delete the e-mail and any attachments from your computer; and do not disseminate, distribute, use, or copy this message and any attachments.

Attachment: users_240392.eml (zipped)
solar@(protected):
> Although not a newbie to building websites in html and php, and physical computing in C/C++, I'm having a dickens of a
> time understanding the documentation of Integrating Tomcat 7 with my existing Apache 2.2.22 on WinXP (company
> supplied development pc, nothing I can do about that). Can I get a bit of handholding please?
>
> I'm trying to evaluate a java servlet that requires tomcat.
>

Actually, if the purpose is purely to evaluate a java servlet, you do not really need
Apache httpd in front. Tomcat will act as a stand-alone webserver just fine, and will
provide access to your servlet.

The simplest under Windows XP :
- from the tomcat website tomcat.apache.org, download tomcat (the "Windows service
installer" version is easiest in this case).
- install it
- check if the Tomcat service is running. If it is, stop it (for the next step).
- under the top directory of that installation, find the "webapps" sub-directory. Copy
your servlet there (if it comes as a .war file) (*).
- start tomcat (the service)
- point your browser at : http://localhost:8080/(name of your servlet)

and there you go.

Then once that works, you can think about configuring Apache as a front-end to tomcat and
try that.


(*) if your servlet (actually, I suppose "webapp" would be a better name) does not come
packaged as a .war file, then :
- unzip or copy it in some new directory, to see what it looks like
- it should be a bunch of files under some top directory (say "myApp")
- copy that whole directory and all its files (including the top directory "myApp") under
the "webapps" directory of Tomcat, so that you have :
C:\tomcatx.y\webapps\myApp
- start tomcat and proceed like above








Attachment: users_240388.eml (zipped)
(Sorry I cannot reply correctly b/c I was on the digest list)

The deadlocked threads:
Deadlock Detection:

Found one Java-level deadlock:
=============================

"qtp1840392480-3740":
waiting to lock Monitor@(protected)
(Object@(protected)),
which is held by "PoolCleaner[2009981184:1363034108768]"
"PoolCleaner[2009981184:1363034108768]":
waiting to lock Monitor@(protected)
(Object@(protected)),
which is held by "qtp1840392480-3740"

Found a total of 1 deadlock.

Here are the stack traces:
Thread 12820: (state = BLOCKED)
- com.mysql.jdbc.ConnectionImpl.getCharacterSetMetadata() @bci=0,
line=2851 (Compiled frame)
- com.mysql.jdbc.Field.getStringFromBytes(int, int) @bci=37, line=717
(Compiled frame)
- com.mysql.jdbc.Field.getName() @bci=17, line=631 (Interpreted frame)
- com.mysql.jdbc.ResultSetImpl.buildIndexMapping() @bci=78, line=752
(Compiled frame)
- com.mysql.jdbc.ResultSetImpl.findColumn(java.lang.String) @bci=12,
line=1110 (Interpreted frame)
- com.mysql.jdbc.ResultSetImpl.getString(java.lang.String) @bci=3,
line=5609 (Interpreted frame)
- org.eclipse.jetty.server.session.JDBCSessionManager$1.run()
@bci=111, line=844 (Interpreted frame)
- org.eclipse.jetty.server.handler.ContextHandler.handle(java.lang.Runnable)
@bci=53, line=1119 (Interpreted frame)
- org.eclipse.jetty.server.session.JDBCSessionManager.loadSession(java.lang.String,
java.lang.String, java.lang.String) @bci=61, line=884 (Interpreted
frame)
- org.eclipse.jetty.server.session.JDBCSessionManager.getSession(java.lang.String)
@bci=345, line=518 (Interpreted frame)
- org.eclipse.jetty.server.session.JDBCSessionManager.getSession(java.lang.String)
@bci=2, line=69 (Interpreted frame)
- org.eclipse.jetty.server.session.AbstractSessionManager.getHttpSession(java.lang.String)
@bci=13, line=272 (Interpreted frame)
- org.eclipse.jetty.server.session.SessionHandler.checkRequestedSessionId(org.eclipse.jetty.server.Request,
javax.servlet.http.HttpServletRequest) @bci=192, line=277 (Interpreted
frame)
- org.eclipse.jetty.server.session.SessionHandler.doScope(java.lang.String,
org.eclipse.jetty.server.Request,
javax.servlet.http.HttpServletRequest,
javax.servlet.http.HttpServletResponse) @bci=47, line=158 (Interpreted
frame)
- org.eclipse.jetty.server.handler.ContextHandler.doScope(java.lang.String,
org.eclipse.jetty.server.Request,
javax.servlet.http.HttpServletRequest,
javax.servlet.http.HttpServletResponse) @bci=416, line=999
(Interpreted frame)
- org.eclipse.jetty.server.handler.ScopedHandler.handle(java.lang.String,
org.eclipse.jetty.server.Request,
javax.servlet.http.HttpServletRequest,
javax.servlet.http.HttpServletResponse) @bci=13, line=117 (Interpreted
frame)
- org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(java.lang.String,
org.eclipse.jetty.server.Request,
javax.servlet.http.HttpServletRequest,
javax.servlet.http.HttpServletResponse) @bci=399, line=250
(Interpreted frame)
- org.eclipse.jetty.server.handler.HandlerCollection.handle(java.lang.String,
org.eclipse.jetty.server.Request,
javax.servlet.http.HttpServletRequest,
javax.servlet.http.HttpServletResponse) @bci=42, line=149 (Compiled
frame)
- org.eclipse.jetty.server.handler.HandlerWrapper.handle(java.lang.String,
org.eclipse.jetty.server.Request,
javax.servlet.http.HttpServletRequest,
javax.servlet.http.HttpServletResponse) @bci=23, line=111 (Compiled
frame)
- org.eclipse.jetty.server.Server.handle(org.eclipse.jetty.server.AbstractHttpConnection)
@bci=134, line=350 (Compiled frame)
- org.eclipse.jetty.server.AbstractHttpConnection.handleRequest()
@bci=228, line=454 (Compiled frame)
- org.eclipse.jetty.server.AbstractHttpConnection.headerComplete()
@bci=448, line=890 (Interpreted frame)
- java.util.HashMap.get(java.lang.Object) @bci=74, line=320 (Compiled frame)
- org.eclipse.jetty.io.BufferCache.get(org.eclipse.jetty.io.Buffer)
@bci=5, line=59 (Compiled frame)
- org.eclipse.jetty.http.HttpParser.parseNext() @bci=625, line=371
(Compiled frame)
- org.eclipse.jetty.http.HttpParser.parseAvailable() @bci=1, line=230
(Compiled frame)
- org.eclipse.jetty.server.AsyncHttpConnection.handle() @bci=80,
line=77 (Compiled frame)
- org.eclipse.jetty.io.nio.SslConnection.handle() @bci=36, line=191
(Compiled frame)
- org.eclipse.jetty.io.nio.SelectChannelEndPoint.handle() @bci=10,
line=606 (Compiled frame)
- org.eclipse.jetty.io.nio.SelectChannelEndPoint$1.run() @bci=4,
line=46 (Interpreted frame)
- org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(java.lang.Runnable)
@bci=1, line=603 (Compiled frame)
- org.eclipse.jetty.util.thread.QueuedThreadPool$3.run() @bci=47,
line=538 (Compiled frame)
- java.lang.Thread.run() @bci=11, line=679 (Interpreted frame)

Locked ownable synchronizers:
  - None

Thread 890: (state = BLOCKED)
- com.mysql.jdbc.ResultSetImpl.realClose(boolean) @bci=0, line=7195
(Interpreted frame)
- com.mysql.jdbc.ResultSetImpl.close() @bci=2, line=909 (Interpreted frame)
- com.mysql.jdbc.StatementImpl.realClose(boolean, boolean) @bci=126,
line=2478 (Interpreted frame)
- com.mysql.jdbc.PreparedStatement.realClose(boolean, boolean)
@bci=71, line=3098 (Interpreted frame)
- com.mysql.jdbc.ConnectionImpl.closeAllOpenStatements() @bci=90,
line=1628 (Interpreted frame)
- com.mysql.jdbc.ConnectionImpl.realClose(boolean, boolean, boolean,
java.lang.Throwable) @bci=176, line=4388 (Interpreted frame)
- com.mysql.jdbc.ConnectionImpl.close() @bci=32, line=1601 (Interpreted frame)
- org.apache.tomcat.jdbc.pool.PooledConnection.disconnect(boolean)
@bci=47, line=330 (Interpreted frame)
- org.apache.tomcat.jdbc.pool.PooledConnection.release() @bci=2,
line=489 (Interpreted frame)
- org.apache.tomcat.jdbc.pool.ConnectionPool.release(org.apache.tomcat.jdbc.pool.PooledConnection)
@bci=10, line=573 (Interpreted frame)
- org.apache.tomcat.jdbc.pool.ConnectionPool.abandon(org.apache.tomcat.jdbc.pool.PooledConnection)
@bci=82, line=532 (Interpreted frame)
- org.apache.tomcat.jdbc.pool.ConnectionPool.checkAbandoned()
@bci=130, line=950 (Interpreted frame)
- org.apache.tomcat.jdbc.pool.ConnectionPool$PoolCleaner.run()
@bci=70, line=1340 (Interpreted frame)
- java.util.TimerThread.mainLoop() @bci=221, line=534 (Compiled frame)
- java.util.TimerThread.run() @bci=1, line=484 (Interpreted frame)

Locked ownable synchronizers:
  - <0x00000006c01b3860>, (a
java/util/concurrent/locks/ReentrantReadWriteLock$NonfairSync)


Once I dug up these stack traces I started to wonder if the mysql
driver was the problem (or contributing to the problem.) I was using
Connector/J version 5.1.19 when the deadlock occurred. I found this
bug: http://bugs.mysql.com/bug.php?id=61247
which sounds a lot like what appears to have happened. I'm interested
in your thoughts on this.

In the meantime I have upgraded to latest Connector/J which includes a
fix for this bug. I was running the old driver for months before this
deadlock, though, so it will be difficult to know if it fixes the
issue or not.

Thanks,
Colin


Attachment: users_240389.eml (zipped)
Using a variety of tutorials I found online and the documentation for o.a.c.startup.Tomcat, I created the following main method to start up an embedded Tomcat. I'm using 7.0.37 Tomcat JARs.

  public static void main(String... arguments) throws Exception
  {
    Tomcat tomcat = new Tomcat();
    tomcat.setBaseDir(".basedir");
    tomcat.setPort(8973);
    tomcat.enableNaming();
    tomcat.init();
    tomcat.start();

    System.out.println("X: " + tomcat.getConnector().getService().getContainer());

    tomcat.getServer().await();
  }

The System.out.println is for debugging purposes, because I'm getting a NullPointerException. Obviously I'm doing something wrong, because about an hour of Googling turned up precisely zero results of anyone who's getting a NullPointerException in MapperListener#findDefaultHost. For some reason, it looks like a Container is never created. What gives? Here's the full output of running the JAR file:

Mar 14, 2013 2:39:04 PM org.apache.coyote.AbstractProtocol init
INFO: Initializing ProtocolHandler ["http-bio-8973"]
Mar 14, 2013 2:39:04 PM org.apache.catalina.core.StandardService startInternal
INFO: Starting service Tomcat
Mar 14, 2013 2:39:04 PM org.apache.coyote.AbstractProtocol start
INFO: Starting ProtocolHandler ["http-bio-8973"]
Mar 14, 2013 2:39:04 PM org.apache.catalina.core.StandardService startInternal
SEVERE: Failed to start connector [Connector[HTTP/1.1-8973]]
org.apache.catalina.LifecycleException: Failed to start component [Connector[HTTP/1.1-8973]]
    at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:154)
    at org.apache.catalina.core.StandardService.startInternal (StandardService.java:459)
    at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:150)
    at org.apache.catalina.core.StandardServer.startInternal (StandardServer.java:732)
    at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:150)
    at org.apache.catalina.startup.Tomcat.start (Tomcat.java:335)
    at com.ul.Bootstrap.main(Bootstrap.java:15)
Caused by: org.apache.catalina.LifecycleException: Failed to start component [org.apache.catalina.connector.MapperListener@(protected)]
    at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:154)
    at org.apache.catalina.connector.Connector.startInternal (Connector.java:1022)
    at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:150)
    ... 6 more
Caused by: java.lang.NullPointerException
    at org.apache.catalina.connector.MapperListener.findDefaultHost (MapperListener.java:252)
    at org.apache.catalina.connector.MapperListener.startInternal (MapperListener.java:104)
    at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:150)
    ... 8 more

X: null

Attachment: users_240393.eml (zipped)
I had tomcat working earlier today, now it won't start. Not sure what I did, as I was in the process of trying to
integrate it with apache.

attaching log, but that probably won't go through.

Mar 14, 2013 4:21:57 PM org.apache.catalina.startup.Catalina load
WARNING: Catalina.start using conf/server.xml: Error at (105, 117) : org.apache.jk.config.ApacheConfig
Mar 14, 2013 4:21:57 PM org.apache.catalina.startup.Catalina start
SEVERE: Cannot start server. Server instance is not configured.


Steve Spence, KK4HFJ
http://arduinotronics.blogspot.com
http://www.essnmag.com
©2008 junlu.com - Jax Systems, LLC, U.S.A.