Coldfusion is automatically restarting, a lot. It comes in waves, everything seems fine for a while then the server struggles for a few minutes, restarts a lot then settles down again.
I have Fusion Reactor installed, but when CF goes down FR stops logging so it's not really helping. Looking through the archived logs just shows gaps in the logs.
These are all the occourances of the phrase "Coldfusion started" today.
[root@server2 logs]# grep -i "Coldfusion started" server.log | grep "11/27/12"
"Information","main","11/27/12","01:49:35",,"ColdFusion started"
"Information","main","11/27/12","01:50:46",,"ColdFusion started"
"Information","main","11/27/12","01:52:39",,"ColdFusion started"
"Information","main","11/27/12","01:54:08",,"ColdFusion started"
"Information","main","11/27/12","01:55:12",,"ColdFusion started"
"Information","main","11/27/12","01:56:29",,"ColdFusion started"
"Information","main","11/27/12","01:57:36",,"ColdFusion started"
"Information","main","11/27/12","01:58:57",,"ColdFusion started"
"Information","main","11/27/12","01:59:56",,"ColdFusion started"
"Information","main","11/27/12","02:01:38",,"ColdFusion started"
"Information","main","11/27/12","02:03:11",,"ColdFusion started"
"Information","main","11/27/12","02:04:41",,"ColdFusion started"
"Information","main","11/27/12","02:07:53",,"ColdFusion started"
"Information","main","11/27/12","02:10:45",,"ColdFusion started"
"Information","main","11/27/12","02:11:49",,"ColdFusion started"
"Information","main","11/27/12","02:13:09",,"ColdFusion started"
"Information","main","11/27/12","02:14:18",,"ColdFusion started"
"Information","main","11/27/12","02:15:44",,"ColdFusion started"
"Information","main","11/27/12","02:17:06",,"ColdFusion started"
"Information","main","11/27/12","02:34:19",,"ColdFusion started"
"Information","main","11/27/12","03:01:20",,"ColdFusion started"
"Information","main","11/27/12","05:25:59",,"ColdFusion started"
"Information","main","11/27/12","06:30:48",,"ColdFusion started"
"Information","main","11/27/12","06:36:20",,"ColdFusion started"
"Information","main","11/27/12","09:34:07",,"ColdFusion started"
"Information","main","11/27/12","09:35:39",,"ColdFusion started"
"Information","main","11/27/12","09:36:41",,"ColdFusion started"
"Information","main","11/27/12","09:39:15",,"ColdFusion started"
"Information","main","11/27/12","09:40:42",,"ColdFusion started"
"Information","main","11/27/12","09:42:55",,"ColdFusion started"
"Information","main","11/27/12","09:44:23",,"ColdFusion started"
"Information","main","11/27/12","09:46:18",,"ColdFusion started"
"Information","main","11/27/12","09:47:35",,"ColdFusion started"
"Information","main","11/27/12","09:48:53",,"ColdFusion started"
"Information","main","11/27/12","09:50:04",,"ColdFusion started"
"Information","main","11/27/12","09:51:51",,"ColdFusion started"
"Information","main","11/27/12","09:53:05",,"ColdFusion started"
"Information","main","11/27/12","09:54:24",,"ColdFusion started"
"Information","main","11/27/12","09:55:28",,"ColdFusion started"
"Information","main","11/27/12","09:56:38",,"ColdFusion started"
"Information","main","11/27/12","09:58:03",,"ColdFusion started"
"Information","main","11/27/12","09:59:03",,"ColdFusion started"
"Information","main","11/27/12","10:04:37",,"ColdFusion started"
"Information","main","11/27/12","12:04:02",,"ColdFusion started"
I've been looking at the live server metrics in FR on a second screen all day, the CPU, Memory and requests all seemed fine about 12 midday, then the server rebooted.
Looking at the logs for the hour between 9am and 10am (more than 15 restarts in the hour), the CPU never went over 44% usage and the Memory never exceeded 53% usage - in the recorded stats at least.
There is no JDBC tracking at the moment, so I'll add that to tracking and see if it's MySQL causing a problem, but can anyone help me narrow down the problem, what would cause Cold Fusion to auto restart, and I'm assuming the auto restart is only happening because Fusion Reactor is installed.
It's a Red Hat 5 LAMP stack running Coldfusion 9 and Fusion Reactor 4.5.2
The ColdFusion Administrator has a section for log files. I would look at all of them. Application.log might be particularly useful.
Also consider logging all of the incoming requests to a DB. The logging should also include the query strings.