I have (well, had up until this morning) an Openfire intranet XMPP server that had been running merrily along for 370 days without being touched, and a total of about 3 years of running. Users authenticate to AD on an adjacent server, and are divided by OU into groups. All that functionality work[ed|s] great.
My problem is that the service crashed this morning, my assistant got a 'no Java heap space' error (and no screenshot, sorry) when attempting to use the admin console to determine what the problem was. We went to the physical console and restarted the service, and it restarted fine, or at least without visible error.
But now, trying to go to the admin console brings up the initial setup page. I groaned, but dug out my notes and re-entered all the LDAP info and DB connection info. With some satisfaction I hit the 'Log into Admin Console' button, and waited...only to be returned to the setup screen I had just left, asking me what my preferred system language was. So:
The MySQL server is running, and the connection info is correct in the /opt/openfire/conf/openfire.xml file. I tested the connection by using the credentials in that file to view the ofProperty table. Checking the contents of the ofProperty table reveals that the settings were saved there, including all the LDAP info, the name and other fields, the group definitions, everything. But Openfire refuses to load those from the database.
Things I have checked:
- Examining /var/log/messages shows nothing out of the ordinary.
- MySQL is running fine and able to take connections from localhost as well as the MySQL workbench on my machine.
- My disk is 3% used.
- Memory is not a problem, 0k swap is used.
- This machine only runs openfire, nothing else.
- Netstat shows only the openfire process listening on 9090 and 9091, the admin ports.
- Running ps only shows the admin process.
- No updates had been installed.
- No configs had been changed.
- This machine is not accessible from the internet, so hacking seems unlikely. Also there are no other signs of it.
EDIT: Log clippings showing a connection refused, but which apparently only happened earlier in time:
2012.05.16 09:27:47 org.jivesoftware.database.DbConnectionManager - Unable to get a connection from the database pool (attempt 10 out of 10).
com.mysql.jdbc.CommunicationsException: Communications link failure due to underlying exception:
** BEGIN NESTED EXCEPTION **
java.net.ConnectException
MESSAGE: Connection refused
STACKTRACE:
java.net.ConnectException: Connection refused
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.PlainSocketImpl.doConnect(Unknown Source)
at java.net.PlainSocketImpl.connectToAddress(Unknown Source)
at java.net.PlainSocketImpl.connect(Unknown Source)
at java.net.SocksSocketImpl.connect(Unknown Source)
at java.net.Socket.connect(Unknown Source)
at java.net.Socket.connect(Unknown Source)
at java.net.Socket.<init>(Unknown Source)
at java.net.Socket.<init>(Unknown Source)
at com.mysql.jdbc.StandardSocketFactory.connect(StandardSocketFactory.java:256)
at com.mysql.jdbc.MysqlIO.<init>(MysqlIO.java:271)
at com.mysql.jdbc.Connection.createNewIO(Connection.java:2771)
at com.mysql.jdbc.Connection.<init>(Connection.java:1555)
at com.mysql.jdbc.NonRegisteringDriver.connect(NonRegisteringDriver.java:285)
at java.sql.DriverManager.getConnection(Unknown Source)
at java.sql.DriverManager.getConnection(Unknown Source)
at org.logicalcobwebs.proxool.DefaultConnectionBuilder.buildConnection(DefaultConnectionBuilder.java:39)
at org.logicalcobwebs.proxool.Prototyper.buildConnection(Prototyper.java:159)
at org.logicalcobwebs.proxool.ConnectionPool.getConnection(ConnectionPool.java:211)
at org.logicalcobwebs.proxool.ProxoolDriver.connect(ProxoolDriver.java:89)
at java.sql.DriverManager.getConnection(Unknown Source)
at java.sql.DriverManager.getConnection(Unknown Source)
at org.jivesoftware.database.DefaultConnectionProvider.getConnection(DefaultConnectionProvider.java:86)
at org.jivesoftware.database.DbConnectionManager.getConnection(DbConnectionManager.java:124)
at org.jivesoftware.openfire.XMPPServer.verifyDataSource(XMPPServer.java:754)
at org.jivesoftware.openfire.XMPPServer.start(XMPPServer.java:480)
at org.jivesoftware.openfire.XMPPServer.<init>(XMPPServer.java:212)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(Unknown Source)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(Unknown Source)
at java.lang.reflect.Constructor.newInstance(Unknown Source)
at java.lang.Class.newInstance0(Unknown Source)
at java.lang.Class.newInstance(Unknown Source)
at org.jivesoftware.openfire.starter.ServerStarter.start(ServerStarter.java:113)
at org.jivesoftware.openfire.starter.ServerStarter.main(ServerStarter.java:58)
** END NESTED EXCEPTION **
Last packet sent to the server was 1 ms ago.
at com.mysql.jdbc.Connection.createNewIO(Connection.java:2847)
at com.mysql.jdbc.Connection.<init>(Connection.java:1555)
at com.mysql.jdbc.NonRegisteringDriver.connect(NonRegisteringDriver.java:285)
at java.sql.DriverManager.getConnection(Unknown Source)
at java.sql.DriverManager.getConnection(Unknown Source)
at org.logicalcobwebs.proxool.DefaultConnectionBuilder.buildConnection(DefaultConnectionBuilder.java:39)
at org.logicalcobwebs.proxool.Prototyper.buildConnection(Prototyper.java:159)
at org.logicalcobwebs.proxool.ConnectionPool.getConnection(ConnectionPool.java:211)
at org.logicalcobwebs.proxool.ProxoolDriver.connect(ProxoolDriver.java:89)
at java.sql.DriverManager.getConnection(Unknown Source)
at java.sql.DriverManager.getConnection(Unknown Source)
at org.jivesoftware.database.DefaultConnectionProvider.getConnection(DefaultConnectionProvider.java:86)
at org.jivesoftware.database.DbConnectionManager.getConnection(DbConnectionManager.java:124)
at org.jivesoftware.openfire.XMPPServer.verifyDataSource(XMPPServer.java:754)
at org.jivesoftware.openfire.XMPPServer.start(XMPPServer.java:480)
at org.jivesoftware.openfire.XMPPServer.<init>(XMPPServer.java:212)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(Unknown Source)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(Unknown Source)
at java.lang.reflect.Constructor.newInstance(Unknown Source)
at java.lang.Class.newInstance0(Unknown Source)
at java.lang.Class.newInstance(Unknown Source)
at org.jivesoftware.openfire.starter.ServerStarter.start(ServerStarter.java:113)
at org.jivesoftware.openfire.starter.ServerStarter.main(ServerStarter.java:58)
2012.05.16 09:27:48 org.jivesoftware.openfire.XMPPServer - Server halted
2012.05.16 09:36:34 org.jivesoftware.openfire.pubsub.PubSubModule - Publish-Subscribe domain: pubsub.cc.server.local
2012.05.16 09:36:36 org.jivesoftware.openfire.muc.spi.MultiUserChatServiceImpl - Multi User Chat domain: conference.cc.server.local
2012.05.16 09:37:48 org.jivesoftware.openfire.XMPPServer - Openfire 3.7.1 [May 16, 2012 9:37:48 AM]
2012.05.16 09:38:01 org.jivesoftware.openfire.container.AdminConsolePlugin - Admin console listening at:
http://127.0.0.1:9090
https://127.0.0.1:9091
2012.05.16 09:54:21 org.jivesoftware.openfire.XMPPServer - Server halted
2012.05.16 09:54:33 org.jivesoftware.openfire.XMPPServer - Openfire 3.7.1 [May 16, 2012 9:54:33 AM]
2012.05.16 09:54:46 org.jivesoftware.openfire.container.AdminConsolePlugin - Admin console listening at:
http://127.0.0.1:9090
https://127.0.0.1:9091
2012.05.16 10:07:24 org.jivesoftware.openfire.XMPPServer - Server halted
2012.05.16 10:07:37 org.jivesoftware.openfire.XMPPServer - Openfire 3.7.1 [May 16, 2012 10:07:37 AM]
2012.05.16 10:07:49 org.jivesoftware.openfire.container.AdminConsolePlugin - Admin console listening at:
http://127.0.0.1:9090
https://127.0.0.1:9091
2012.05.16 10:26:39 org.jivesoftware.openfire.XMPPServer - Server halted
2012.05.16 10:28:54 org.jivesoftware.openfire.XMPPServer - Openfire 3.7.1 [May 16, 2012 10:28:54 AM]
2012.05.16 10:29:10 org.jivesoftware.openfire.container.AdminConsolePlugin - Admin console listening at:
http://127.0.0.1:9090
https://127.0.0.1:9091
2012.05.16 10:33:25 org.jivesoftware.openfire.XMPPServer - Server halted
2012.05.16 10:33:38 org.jivesoftware.openfire.XMPPServer - Openfire 3.7.1 [May 16, 2012 10:33:38 AM]
2012.05.16 10:33:51 org.jivesoftware.openfire.container.AdminConsolePlugin - Admin console listening at:
http://127.0.0.1:9090
https://127.0.0.1:9091
2012.05.16 10:51:38 org.jivesoftware.openfire.XMPPServer - Server halted
2012.05.16 10:51:51 org.jivesoftware.openfire.XMPPServer - Openfire 3.7.1 [May 16, 2012 10:51:51 AM]
2012.05.16 10:52:04 org.jivesoftware.openfire.container.AdminConsolePlugin - Admin console listening at:
http://127.0.0.1:9090
https://127.0.0.1:9091
2012.05.16 10:53:01 org.jivesoftware.openfire.XMPPServer - Server halted
2012.05.16 10:53:35 org.jivesoftware.openfire.XMPPServer - Openfire 3.7.1 [May 16, 2012 10:53:35 AM]
2012.05.16 10:53:49 org.jivesoftware.openfire.container.AdminConsolePlugin - Admin console listening at:
http://127.0.0.1:9090
https://127.0.0.1:9091
After that one failure, I had attempted variously to reconfigure and restart the service, as you can see the admin console comes up.
Long story short: Got a look at another (working) installation elsewhere. Turns out there is a value called
<setup>
that resides in the fileopenfire.xml
of a working installation as follows:The setup line was missing from my
openfire.xml
for some reason.Interestingly, the file is owned and only writable by root, so some privileged process deleted the line (somehow???) when the setup process ran, while apparently leaving the connection parameters alone. But, since the file was only writable by root, the setup script run by the JRE wasn't able to write to the file. Also, I'm a moron, and missed this line among the Java exceptions:
java.io.IOException: XML properties file must be writable: openfire.xml
Apparently it's writable to some parts of the setup process but not others? I don't know. But if anyone else has this problem, hopefully they'll find the answer here. I was able to correct the problem and get Openfire to load immediately by adding the
<setup>
value to openfire.xml.