If I try to connect to MySQL with
telnet myhost.com 3306
I get a network not reachable
error, even when I flush all iptables
rules. I can reach other ports like ssh
(with the same iptables
rules applying as for ssh
).
Below is what I see in netstat
. It gives me the impression that there is more to accessing ports than iptables
: ports like 10025 are closed in my iptables
.
And I see MySQL open for tcp but not for tcp6 (ssh is open for both).
This issue started after upgrading from Ubuntu server 12.04 to 14.04. Anyone any suggestion as to the connection time out?
$ netstat -tlp 3306
(No info could be read for "-p": geteuid()=1000 but you should be root.)
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name
tcp 0 0 *:pop3s *:* LISTEN -
tcp 0 0 localhost:10023 *:* LISTEN -
tcp 0 0 localhost:10024 *:* LISTEN -
tcp 0 0 localhost:10025 *:* LISTEN -
tcp 0 0 *:mysql *:* LISTEN -
tcp 0 0 *:pop3 *:* LISTEN -
tcp 0 0 localhost:spamd *:* LISTEN -
tcp 0 0 *:imap2 *:* LISTEN -
tcp 0 0 *:urd *:* LISTEN -
tcp 0 0 *:smtp *:* LISTEN -
tcp 0 0 *:sieve *:* LISTEN -
tcp 0 0 *:ssh *:* LISTEN -
tcp 0 0 *:imaps *:* LISTEN -
tcp6 0 0 [::]:pop3s [::]:* LISTEN -
tcp6 0 0 localhost:10023 [::]:* LISTEN -
tcp6 0 0 [::]:pop3 [::]:* LISTEN -
tcp6 0 0 localhost:spamd [::]:* LISTEN -
tcp6 0 0 [::]:imap2 [::]:* LISTEN -
tcp6 0 0 [::]:http [::]:* LISTEN -
tcp6 0 0 [::]:urd [::]:* LISTEN -
tcp6 0 0 [::]:smtp [::]:* LISTEN -
tcp6 0 0 [::]:https [::]:* LISTEN -
tcp6 0 0 [::]:sieve [::]:* LISTEN -
tcp6 0 0 [::]:ssh [::]:* LISTEN -
tcp6 0 0 [::]:imaps [::]:* LISTEN -
150407 12:31:07 [Note] /usr/sbin/mysqld: Normal shutdown
150407 12:31:07 [Note] Event Scheduler: Purging the queue. 0 events
150407 12:31:07 InnoDB: Starting shutdown...
150407 12:31:10 InnoDB: Shutdown completed; log sequence number 574674933
150407 12:31:10 [Note] /usr/sbin/mysqld: Shutdown complete
150407 12:31:11 mysqld_safe Starting mysqld daemon with databases from /var/lib/mysql
150407 12:31:11 [Warning] Using unique option prefix key_buffer instead of key_buffer_size is deprecated and will be removed in a future release. Please use the full name instead.
150407 12:31:12 [Warning] Using unique option prefix myisam-recover instead of myisam-recover-options is deprecated and will be removed in a future release. Please use the full name instead.
150407 12:31:12 [Note] Plugin 'FEDERATED' is disabled.
150407 12:31:12 InnoDB: The InnoDB memory heap is disabled
150407 12:31:12 InnoDB: Mutexes and rw_locks use GCC atomic builtins
150407 12:31:12 InnoDB: Compressed tables use zlib 1.2.8
150407 12:31:12 InnoDB: Using Linux native AIO
150407 12:31:12 InnoDB: Initializing buffer pool, size = 128.0M
150407 12:31:12 InnoDB: Completed initialization of buffer pool
150407 12:31:12 InnoDB: highest supported file format is Barracuda.
150407 12:31:12 [Warning] Using unique option prefix myisam-recover instead of myisam-recover-options is deprecated and will be removed in a future release. Please use the full name instead.
150407 12:31:12 [Note] Plugin 'FEDERATED' is disabled.
150407 12:31:12 InnoDB: The InnoDB memory heap is disabled
150407 12:31:12 InnoDB: Mutexes and rw_locks use GCC atomic builtins
150407 12:31:12 InnoDB: Compressed tables use zlib 1.2.8
150407 12:31:12 InnoDB: Using Linux native AIO
150407 12:31:12 InnoDB: Initializing buffer pool, size = 128.0M
150407 12:31:12 InnoDB: Completed initialization of buffer pool
InnoDB: Unable to lock ./ibdata1, error: 11
InnoDB: Check that you do not already have another mysqld process
InnoDB: using the same InnoDB data or log files.
150407 12:31:12 InnoDB: Retrying to lock the first data file
InnoDB: Unable to lock ./ibdata1, error: 11
InnoDB: Check that you do not already have another mysqld process
InnoDB: using the same InnoDB data or log files.
150407 12:31:13 InnoDB: Waiting for the background threads to start
InnoDB: Unable to lock ./ibdata1, error: 11
InnoDB: Check that you do not already have another mysqld process
InnoDB: using the same InnoDB data or log files.
150407 12:31:14 InnoDB: 5.5.41 started; log sequence number 574674933
150407 12:31:14 [Note] Server hostname (bind-address): '0.0.0.0'; port: 3306
150407 12:31:14 [Note] - '0.0.0.0' resolves to '0.0.0.0';
150407 12:31:14 [Note] Server socket created on IP: '0.0.0.0'.
150407 12:31:15 [Note] Event Scheduler: Loaded 0 events
150407 12:31:15 [Note] /usr/sbin/mysqld: ready for connections.
Version: '5.5.41-0ubuntu0.14.04.1' socket: '/var/run/mysqld/mysqld.sock' port: 3306 (Ubuntu)
InnoDB: Unable to lock ./ibdata1, error: 11
InnoDB: Check that you do not already have another mysqld process
InnoDB: using the same InnoDB data or log files.
...
InnoDB: Unable to lock ./ibdata1, error: 11
InnoDB: Check that you do not already have another mysqld process
InnoDB: using the same InnoDB data or log files.
150407 12:32:52 InnoDB: Unable to open the first data file
InnoDB: Error in opening ./ibdata1
150407 12:32:52 InnoDB: Operating system error number 11 in a file operation.
InnoDB: Error number 11 means 'Resource temporarily unavailable'.
InnoDB: Some operating system error numbers are described at
InnoDB: http://dev.mysql.com/doc/refman/5.5/en/operating-system-error-codes.html
150407 12:32:52 InnoDB: Could not open or create data files.
150407 12:32:52 InnoDB: If you tried to add new data files, and it failed here,
150407 12:32:52 InnoDB: you should now edit innodb_data_file_path in my.cnf back
150407 12:32:52 InnoDB: to what it was, and remove the new ibdata files InnoDB created
150407 12:32:52 InnoDB: in this failed attempt. InnoDB only wrote those files full of
150407 12:32:52 InnoDB: zeros, but did not yet use them in any way. But be careful: do not
150407 12:32:52 InnoDB: remove old data files which contain your precious data!
150407 12:32:52 [ERROR] Plugin 'InnoDB' init function returned error.
150407 12:32:52 [ERROR] Plugin 'InnoDB' registration as a STORAGE ENGINE failed.
150407 12:32:52 [ERROR] Unknown/unsupported storage engine: InnoDB
150407 12:32:52 [ERROR] Aborting
150407 12:32:52 [Note] /usr/sbin/mysqld: Shutdown complete
150407 12:32:52 [Warning] Using unique option prefix myisam-recover instead of myisam-recover-options is deprecated and will be removed in a future release. Please use the full name instead.
150407 12:32:52 [Note] Plugin 'FEDERATED' is disabled.
......
output of netstat after changing bind-address to 0.0.0.0:
tcp 0 0 0.0.0.0:3306 0.0.0.0:* LISTEN 18890/mysqld
Output dig:
; <<>> DiG 9.9.5-3ubuntu0.2-Ubuntu <<>> myhost.com ip r get 123.45.67.890 telnet 123.45.67.890 3306
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 55636
;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1
;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 4000
;; QUESTION SECTION:
;myhost.com. IN A
;; ANSWER SECTION:
myhost.com. 3600 IN A 123.45.67.890
;; Query time: 856 msec
;; SERVER: 127.0.1.1#53(127.0.1.1)
;; WHEN: Tue Apr 07 22:55:03 CEST 2015
;; MSG SIZE rcvd: 60
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NXDOMAIN, id: 35733
;; flags: qr rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 1, ADDITIONAL: 1
;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 4000
;; QUESTION SECTION:
;ip. IN A
;; AUTHORITY SECTION:
. 528 IN SOA a.root-servers.net. nstld.verisign-grs.com. 2015040701 1800 900 604800 86400
;; Query time: 159 msec
;; SERVER: 127.0.1.1#53(127.0.1.1)
;; WHEN: Tue Apr 07 22:55:03 CEST 2015
;; MSG SIZE rcvd: 106
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NXDOMAIN, id: 17760
;; flags: qr rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 1, ADDITIONAL: 1
;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 4000
;; QUESTION SECTION:
;r. IN A
;; AUTHORITY SECTION:
. 528 IN SOA a.root-servers.net. nstld.verisign-grs.com. 2015040701 1800 900 604800 86400
;; Query time: 55 msec
;; SERVER: 127.0.1.1#53(127.0.1.1)
;; WHEN: Tue Apr 07 22:55:03 CEST 2015
;; MSG SIZE rcvd: 105
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NXDOMAIN, id: 20236
;; flags: qr rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 1, ADDITIONAL: 1
;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 4000
;; QUESTION SECTION:
;get. IN A
;; AUTHORITY SECTION:
. 527 IN SOA a.root-servers.net. nstld.verisign-grs.com. 2015040701 1800 900 604800 86400
;; Query time: 62 msec
;; SERVER: 127.0.1.1#53(127.0.1.1)
;; WHEN: Tue Apr 07 22:55:04 CEST 2015
;; MSG SIZE rcvd: 107
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 29568
;; flags: qr aa rd ra ad; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 0
;; QUESTION SECTION:
;123.45.67.890. IN A
;; ANSWER SECTION:
123.45.67.890. 0 IN A 123.45.67.890
;; Query time: 0 msec
;; SERVER: 127.0.1.1#53(127.0.1.1)
/etc/mysql/my.cnf:
#
# The MySQL database server configuration file.
#
# You can copy this to one of:
# - "/etc/mysql/my.cnf" to set global options,
# - "~/.my.cnf" to set user-specific options.
#
# One can use all long options that the program supports.
# Run program with --help to get a list of available options and with
# --print-defaults to see which it would actually understand and use.
#
# For explanations see
# http://dev.mysql.com/doc/mysql/en/server-system-variables.html
# This will be passed to all mysql clients
# It has been reported that passwords should be enclosed with ticks/quotes
# escpecially if they contain "#" chars...
# Remember to edit /etc/mysql/debian.cnf when changing the socket location.
[client]
port = 3306
socket = /var/run/mysqld/mysqld.sock
# Here is entries for some specific programs
# The following values assume you have at least 32M ram
# This was formally known as [safe_mysqld]. Both versions are currently parsed.
[mysqld_safe]
socket = /var/run/mysqld/mysqld.sock
nice = 0
[mysqld]
#
# * Basic Settings
#
user = mysql
pid-file = /var/run/mysqld/mysqld.pid
socket = /var/run/mysqld/mysqld.sock
port = 3306
basedir = /usr
datadir = /var/lib/mysql
tmpdir = /tmp
lc-messages-dir = /usr/share/mysql
#skip-external-locking
#
# Instead of skip-networking the default is now to listen only on
# localhost which is more compatible and is not less secure.
#bind-address = 127.0.0.1
bind-address = 0.0.0.0
#
# * Fine Tuning
#
key_buffer = 16M
max_allowed_packet = 16M
thread_stack = 192K
thread_cache_size = 8
# This replaces the startup script and checks MyISAM tables if needed
# the first time they are touched
myisam-recover = BACKUP
#max_connections = 100
#table_cache = 64
#thread_concurrency = 10
#
# * Query Cache Configuration
#
query_cache_limit = 1M
query_cache_size = 16M
#
# * Logging and Replication
#
# Both location gets rotated by the cronjob.
# Be aware that this log type is a performance killer.
# As of 5.1 you can enable the log at runtime!
#general_log_file = /var/log/mysql/mysql.log
#general_log = 1
#
# Error log - should be very few entries.
#
log_error = /var/log/mysql/error.log
#
# Here you can see queries with especially long duration
#log_slow_queries = /var/log/mysql/mysql-slow.log
#long_query_time = 2
#log-queries-not-using-indexes
#
# The following can be used as easy to replay backup logs or for replication.
# note: if you are setting up a replication slave, see README.Debian about
# other settings you may need to change.
#server-id = 1
#log_bin = /var/log/mysql/mysql-bin.log
expire_logs_days = 10
max_binlog_size = 100M
#binlog_do_db = include_database_name
#binlog_ignore_db = include_database_name
#
# * InnoDB
#
# InnoDB is enabled by default with a 10MB datafile in /var/lib/mysql/.
# Read the manual for more InnoDB related options. There are many!
#
# * Security Features
#
# Read the manual, too, if you want chroot!
# chroot = /var/lib/mysql/
#
# For generating SSL certificates I recommend the OpenSSL GUI "tinyca".
#
# ssl-ca=/etc/mysql/cacert.pem
# ssl-cert=/etc/mysql/server-cert.pem
# ssl-key=/etc/mysql/server-key.pem
[mysqldump]
quick
quote-names
max_allowed_packet = 16M
[mysql]
#no-auto-rehash # faster start of mysql but no tab completition
[isamchk]
key_buffer = 16M
#
# * IMPORTANT: Additional settings that can override those from this file!
# The files must end with '.cnf', otherwise they'll be ignored.
#
!includedir /etc/mysql/conf.d/
traceroute:
1 192.168.1.1 (192.168.1.1) 4.728 ms 4.720 ms 4.707 ms
2 1.16.15.37.dynamic.jazztel.es (37.15.16.1) 26.522 ms 26.529 ms 28.352 ms
3 10.255.160.254 (10.255.160.254) 30.024 ms 30.017 ms 29.987 ms
4 41.217.106.212.static.jazztel.es (212.106.217.41) 44.086 ms 45.217.106.212.static.jazztel.es (212.106.217.45) 52.257 ms 41.217.106.212.static.jazztel.es (212.106.217.41) 42.428 ms
5 * 42.217.106.212.static.jazztel.es (212.106.217.42) 47.672 ms 52.229 ms
6 129.216.106.212.static.jazztel.es (212.106.216.129) 57.838 ms 61.308 ms *
7 142.216.106.212.static.jazztel.es (212.106.216.142) 89.549 ms 106.063 ms *
8 142.216.106.212.static.jazztel.es (212.106.216.142) 76.570 ms 195.66.225.53 (195.66.225.53) 87.575 ms 142.216.106.212.static.jazztel.es (212.106.216.142) 84.337 ms
9 195.66.225.53 (195.66.225.53) 106.011 ms 76.555 ms 105.993 ms
10 openpeering.pcextreme.nl (82.150.154.35) 84.274 ms telecity2.openpeering.nl (82.150.154.26) 87.533 ms nikhef.openpeering.nl (82.150.154.25) 105.973 ms
11 openpeering.pcextreme.nl (82.150.154.35) 87.506 ms 87.474 ms 185.27.173.130 (185.27.173.130) 79.570 ms
12 185.27.173.150 (185.27.173.150) 95.558 ms 95.510 ms 185.27.173.130 (185.27.173.130) 81.846 ms
13 185.27.173.150 (185.27.173.150) 68.465 ms * 84.567 ms
14 * * *
15 * * *
16 * * *
17 * * *
18 * * *
19 * * *
20 * * *
21 * * *
22 * * *
23 * * *
24 * * *
25 * * *
26 * * *
27 * * *
28 * * *
29 * * *
30 * * *
How would I go about network unreachable using plain telnet to some port?
On the client
Name resolution (what are you actually connecting to?)
What is the result? What IP? IPv4 or IPv6? (It would help not obfuscating the output.)
Oh and BTW what is the OS you are trying from? If Linux:
On the server
Is the service listening (If it's not listening, no way to connect to it)
On the server try a local connection
What is the result?
During telnet from client start
Do you see packets coming from the client? (make sure tshark is running on the correct interface, if many)
If no packets: something on the net is blocking them (NOT a potential iptables firewall on the server, we will get to that next.)
If packets are indeed seen: no network problem, check iptables
What does it say? What about
iptables -t raw -vnL
,iptables -t mangle -vnL
,iptables -t nat -vnL
?Is Selinux active? Or some other host protection?
I see locking errors during shutdown. Are you maybe having multiple mysqlds running by mistake? What does
ps -ef | grep mysql
say? Is startup as messy as shutdown?Please reply with detailed results.
This has been an issue with me before whereby I cannot connect to a remote MySQL server via the internet. I did a few quick troubleshooting tests, such as:
By the information you have provided, you seem to have done the above as I can see it has bound to tcp with the line:
Then I recalled an article I read on Stackoverflow from apesa, which consisted of:
-> answered by Apesa: Remote connections MySQL Ubuntu
Generally this fixed my issue as I did not configure my.cnf
If bind-address is set to 127.0.0.1 in /etc/mysql/my.cnf try changing it to the following, which will allow MySQL to bind to all available interfaces:
bind-address = 0.0.0.0
Also, try granting yourself network access in MySQL itself:
mysql> GRANT ALL ON . to your-user-name@'your-hostname' IDENTIFIED BY 'your-password';
what about tcpwrapers: /etc/hosts.{allow,deny} files? Looking at your netstat output it looks like MySQL is listening on all IPv4 addresses and your iptables are open too. So it boils down to either IPv6 issue or tcpwrappers as they sit in the middle of things and are not easy to debug.