I have an old SUSE 8.1 server which just recently started acting crazy.
If I start a program once:
/usr/local/jdk1.6.0_21/bin/java -cp example-ser-0.8.jar:/usr/local/lib/java/slf4j-api-1.6.1.jar:/usr/local/lib/java/logback-core-0.9.24.jar:/usr/local/lib/java/logback-classic-0.9.24.jar:/usr/local/lib/java/jcl-over-slf4j-1.6.1.jar:/usr/local/lib/java/commons-cli-1.2.jar:/usr/local/lib/java/commons-lang-2.5.jar:/usr/local/lib/java/mysql-connector-java-5.1.13-bin.jar:/usr/local/lib/java/xBaseJ-20091203.jar:/usr/local/lib/java/Translator-0.7.0.jar:/usr/local/lib/java/antlr-3.2.jar:/usr/local/lib/java/joda-time-1.6.jar -Dlogback.configurationFile=logback-test.xml example.ser.Scanner abc12345
There will be 10 instances running (same user, same arguments) as shown by ps ax -l | grep Scanner
:
F S UID PID PPID C PRI NI ADDR SZ WCHAN TTY TIME CMD 000 S 502 20228 20227 0 75 0 - 98222 rt_sig pts/11 0:00 /usr/local/jdk1.6.0_21/bin/java -cp /usr/local/lib/java/example-ser-0.7.jar:/usr/local/lib/java/slf4j-api-1.6.1.jar:/usr/local/lib/java/logback-core-0.9.24.jar:/usr/local/lib/java/logback-classic-0.9.24.jar:/usr/local/lib/java/jcl-over-slf4j-1.6.1.jar:/usr/local/lib/java/commons-cli-1.2.jar:/usr/local/lib/java/commons-lang-2.5.jar:/usr/local/lib/java/mysql-connector-java-5.1.13-bin.jar:/usr/local/lib/java/xBaseJ-20091203.jar:/usr/local/lib/java/CampaignRulesTranslator-0.7.0.jar:/usr/local/lib/java/antlr-3.2.jar:/usr/local/lib/java/joda-time-1.6.jar -Dlogback.configurationFile=/usr/local/lib/java/logback.xml example.ser.Scanner abc12345 002 S 502 20229 20228 0 75 0 - 98222 schedu pts/11 0:00 /usr/local/jdk1.6.0_21/bin/java -cp /usr/local/lib/java/example-ser-0.7.jar:/usr/local/lib/java/slf4j-api-1.6.1.jar:/usr/local/lib/java/logback-core-0.9.24.jar:/usr/local/lib/java/logback-classic-0.9.24.jar:/usr/local/lib/java/jcl-over-slf4j-1.6.1.jar:/usr/local/lib/java/commons-cli-1.2.jar:/usr/local/lib/java/commons-lang-2.5.jar:/usr/local/lib/java/mysql-connector-java-5.1.13-bin.jar:/usr/local/lib/java/xBaseJ-20091203.jar:/usr/local/lib/java/CampaignRulesTranslator-0.7.0.jar:/usr/local/lib/java/antlr-3.2.jar:/usr/local/lib/java/joda-time-1.6.jar -Dlogback.configurationFile=/usr/local/lib/java/logback.xml example.ser.Scanner abc12345 002 S 502 20230 20229 0 75 0 - 98222 schedu pts/11 0:14 /usr/local/jdk1.6.0_21/bin/java -cp /usr/local/lib/java/example-ser-0.7.jar:/usr/local/lib/java/slf4j-api-1.6.1.jar:/usr/local/lib/java/logback-core-0.9.24.jar:/usr/local/lib/java/logback-classic-0.9.24.jar:/usr/local/lib/java/jcl-over-slf4j-1.6.1.jar:/usr/local/lib/java/commons-cli-1.2.jar:/usr/local/lib/java/commons-lang-2.5.jar:/usr/local/lib/java/mysql-connector-java-5.1.13-bin.jar:/usr/local/lib/java/xBaseJ-20091203.jar:/usr/local/lib/java/CampaignRulesTranslator-0.7.0.jar:/usr/local/lib/java/antlr-3.2.jar:/usr/local/lib/java/joda-time-1.6.jar -Dlogback.configurationFile=/usr/local/lib/java/logback.xml example.ser.Scanner abc12345 002 S 502 20231 20229 0 75 0 - 98222 schedu pts/11 0:01 /usr/local/jdk1.6.0_21/bin/java -cp /usr/local/lib/java/example-ser-0.7.jar:/usr/local/lib/java/slf4j-api-1.6.1.jar:/usr/local/lib/java/logback-core-0.9.24.jar:/usr/local/lib/java/logback-classic-0.9.24.jar:/usr/local/lib/java/jcl-over-slf4j-1.6.1.jar:/usr/local/lib/java/commons-cli-1.2.jar:/usr/local/lib/java/commons-lang-2.5.jar:/usr/local/lib/java/mysql-connector-java-5.1.13-bin.jar:/usr/local/lib/java/xBaseJ-20091203.jar:/usr/local/lib/java/CampaignRulesTranslator-0.7.0.jar:/usr/local/lib/java/antlr-3.2.jar:/usr/local/lib/java/joda-time-1.6.jar -Dlogback.configurationFile=/usr/local/lib/java/logback.xml example.ser.Scanner abc12345 002 S 502 20232 20229 0 76 0 - 98222 rt_sig pts/11 0:00 /usr/local/jdk1.6.0_21/bin/java -cp /usr/local/lib/java/example-ser-0.7.jar:/usr/local/lib/java/slf4j-api-1.6.1.jar:/usr/local/lib/java/logback-core-0.9.24.jar:/usr/local/lib/java/logback-classic-0.9.24.jar:/usr/local/lib/java/jcl-over-slf4j-1.6.1.jar:/usr/local/lib/java/commons-cli-1.2.jar:/usr/local/lib/java/commons-lang-2.5.jar:/usr/local/lib/java/mysql-connector-java-5.1.13-bin.jar:/usr/local/lib/java/xBaseJ-20091203.jar:/usr/local/lib/java/CampaignRulesTranslator-0.7.0.jar:/usr/local/lib/java/antlr-3.2.jar:/usr/local/lib/java/joda-time-1.6.jar -Dlogback.configurationFile=/usr/local/lib/java/logback.xml example.ser.Scanner abc12345 002 S 502 20233 20229 0 76 0 - 98222 rt_sig pts/11 0:00 /usr/local/jdk1.6.0_21/bin/java -cp /usr/local/lib/java/example-ser-0.7.jar:/usr/local/lib/java/slf4j-api-1.6.1.jar:/usr/local/lib/java/logback-core-0.9.24.jar:/usr/local/lib/java/logback-classic-0.9.24.jar:/usr/local/lib/java/jcl-over-slf4j-1.6.1.jar:/usr/local/lib/java/commons-cli-1.2.jar:/usr/local/lib/java/commons-lang-2.5.jar:/usr/local/lib/java/mysql-connector-java-5.1.13-bin.jar:/usr/local/lib/java/xBaseJ-20091203.jar:/usr/local/lib/java/CampaignRulesTranslator-0.7.0.jar:/usr/local/lib/java/antlr-3.2.jar:/usr/local/lib/java/joda-time-1.6.jar -Dlogback.configurationFile=/usr/local/lib/java/logback.xml example.ser.Scanner abc12345 002 S 502 20234 20229 0 80 0 - 98222 rt_sig pts/11 0:00 /usr/local/jdk1.6.0_21/bin/java -cp /usr/local/lib/java/example-ser-0.7.jar:/usr/local/lib/java/slf4j-api-1.6.1.jar:/usr/local/lib/java/logback-core-0.9.24.jar:/usr/local/lib/java/logback-classic-0.9.24.jar:/usr/local/lib/java/jcl-over-slf4j-1.6.1.jar:/usr/local/lib/java/commons-cli-1.2.jar:/usr/local/lib/java/commons-lang-2.5.jar:/usr/local/lib/java/mysql-connector-java-5.1.13-bin.jar:/usr/local/lib/java/xBaseJ-20091203.jar:/usr/local/lib/java/CampaignRulesTranslator-0.7.0.jar:/usr/local/lib/java/antlr-3.2.jar:/usr/local/lib/java/joda-time-1.6.jar -Dlogback.configurationFile=/usr/local/lib/java/logback.xml example.ser.Scanner abc12345 002 S 502 20235 20229 0 75 0 - 98222 rt_sig pts/11 0:00 /usr/local/jdk1.6.0_21/bin/java -cp /usr/local/lib/java/example-ser-0.7.jar:/usr/local/lib/java/slf4j-api-1.6.1.jar:/usr/local/lib/java/logback-core-0.9.24.jar:/usr/local/lib/java/logback-classic-0.9.24.jar:/usr/local/lib/java/jcl-over-slf4j-1.6.1.jar:/usr/local/lib/java/commons-cli-1.2.jar:/usr/local/lib/java/commons-lang-2.5.jar:/usr/local/lib/java/mysql-connector-java-5.1.13-bin.jar:/usr/local/lib/java/xBaseJ-20091203.jar:/usr/local/lib/java/CampaignRulesTranslator-0.7.0.jar:/usr/local/lib/java/antlr-3.2.jar:/usr/local/lib/java/joda-time-1.6.jar -Dlogback.configurationFile=/usr/local/lib/java/logback.xml example.ser.Scanner abc12345 002 S 502 20236 20229 0 80 0 - 98222 rt_sig pts/11 0:00 /usr/local/jdk1.6.0_21/bin/java -cp /usr/local/lib/java/example-ser-0.7.jar:/usr/local/lib/java/slf4j-api-1.6.1.jar:/usr/local/lib/java/logback-core-0.9.24.jar:/usr/local/lib/java/logback-classic-0.9.24.jar:/usr/local/lib/java/jcl-over-slf4j-1.6.1.jar:/usr/local/lib/java/commons-cli-1.2.jar:/usr/local/lib/java/commons-lang-2.5.jar:/usr/local/lib/java/mysql-connector-java-5.1.13-bin.jar:/usr/local/lib/java/xBaseJ-20091203.jar:/usr/local/lib/java/CampaignRulesTranslator-0.7.0.jar:/usr/local/lib/java/antlr-3.2.jar:/usr/local/lib/java/joda-time-1.6.jar -Dlogback.configurationFile=/usr/local/lib/java/logback.xml example.ser.Scanner abc12345 002 S 502 20237 20229 0 75 0 - 98222 schedu pts/11 0:00 /usr/local/jdk1.6.0_21/bin/java -cp /usr/local/lib/java/example-ser-0.7.jar:/usr/local/lib/java/slf4j-api-1.6.1.jar:/usr/local/lib/java/logback-core-0.9.24.jar:/usr/local/lib/java/logback-classic-0.9.24.jar:/usr/local/lib/java/jcl-over-slf4j-1.6.1.jar:/usr/local/lib/java/commons-cli-1.2.jar:/usr/local/lib/java/commons-lang-2.5.jar:/usr/local/lib/java/mysql-connector-java-5.1.13-bin.jar:/usr/local/lib/java/xBaseJ-20091203.jar:/usr/local/lib/java/CampaignRulesTranslator-0.7.0.jar:/usr/local/lib/java/antlr-3.2.jar:/usr/local/lib/java/joda-time-1.6.jar -Dlogback.configurationFile=/usr/local/lib/java/logback.xml example.ser.Scanner abc12345 000 S 0 25155 19094 0 80 0 - 406 pipe_w pts/2 0:00 grep Scanner
If I ^C
the terminal where I started the program, all 10 disappear. This program doesn't do any forking or create any extra threads. I also tried several older versions of this program going back years before this problem and they're all experiencing this now as well. What could be going on here?
The program itself might not fork but the libs/jars provided on the classpath might be forking , I would first try to check those libs code for forks. If that fails, try to execute it with a different JVM version and/or environment and see what happens.