Home / server / Questions / 327294
Accepted
Sarah Ryan
Asked: 2011-11-03 21:46:12 +0800 CST

OpenVPN Segmentation Fault

  • 772

I have a server running CentOS 5.7 and I'm trying to get OpenVPN 2.2.0 to work. To start off, I had this working for quite a while now, but decided to try and make it highly available by using heartbeat for failover to another server. Now I can't even get it to work at all for some reason.

Whenever I try and start openvpn by running /etc/init.d/openvpn start it keeps giving me the error

/etc/init.d/openvpn: line 148: 5820 Segmentation fault $openvpn --daemon --writepid $piddir/$bn.pid --config $c --cd $work

That section of the script is:

 ...

 # Start every .conf in $work and run .sh if exists
        errors=0
        successes=0
        for c in `/bin/ls *.conf 2>/dev/null`; do
            bn=${c%%.conf}
            if [ -f "$bn.sh" ]; then
                . $bn.sh
            fi
            rm -f $piddir/$bn.pid
            $openvpn --daemon --writepid $piddir/$bn.pid --config $c --cd $work
            if [ $? = 0 ]; then
                successes=1
            else
                errors=1
            fi
        done
...

Here are my init scripts:

/etc/openvpn/bridge-start

#!/bin/bash

. /etc/sysconfig/network-scripts/ifcfg-eth1

#################################
# Set up Ethernet bridge on Linux
# Requires: bridge-utils
#################################

# Define Bridge Interface
br="br0"

# Define list of TAP interfaces to be bridged,
# for example tap="tap0 tap1 tap2".
tap="tap0"

# Define physical ethernet interface to be bridged
# with TAP interface(s) above.
eth=$DEVICE
eth_ip=$IPADDR
eth_netmask=$NETMASK
eth_broadcast="10.224.45.159"

for t in $tap; do
    openvpn --mktun --dev $t
done

brctl addbr $br
brctl addif $br $eth

for t in $tap; do
    brctl addif $br $t
done

for t in $tap; do
    ifconfig $t 0.0.0.0 promisc up
done

ifconfig $eth 0.0.0.0 promisc up

ifconfig $br $eth_ip netmask $eth_netmask broadcast $eth_broadcast

/etc/openvpn/openvpn.conf

port        1194
proto       tcp

# Bridge
dev tap0
server-bridge   10.224.45.130 255.255.255.0 10.224.45.140 10.224.45.160

ca      /etc/openvpn/keys/ca.crt
cert        /etc/openvpn/keys/server.crt
key     /etc/openvpn/keys/server.key
dh      /etc/openvpn/keys/dh1024.pem
keepalive   10 120
status      /var/log/openvpn/status.log
log     /var/log/openvpn/openvpn.log
verb        0
client-to-client
comp-lzo
user nobody
group users
persist-key
persist-tun
client-cert-not-required
ifconfig-pool-persist ipp.txt
plugin /usr/lib64/openvpn/plugin/lib/openvpn-auth-ldap.so "/etc/openvpn/auth/ldap.conf"

/etc/openvpn/auth/ldap.conf

<LDAP>
        # LDAP server URL
        URL             ldap://ldap.****.com

        # Bind DN (If your LDAP server doesn't support anonymous binds)
        BindDN          uid=readonly,ou=Users,dc=****,dc=com

        # Bind Password
        Password        <hidden>

        # Network timeout (in seconds)
        Timeout         15

        # Follow LDAP Referrals (anonymously)
        FollowReferrals yes
</LDAP>

<Authorization>
        # Base DN
        BaseDN          "ou=Users,dc=****,dc=com"

        # User Search Filter
        SearchFilter    "(&(uid=%u))"

        # Require Group Membership
        RequireGroup    false
</Authorization>

I have no clue what the reason for the segfault could be. I've searched Google for hours now with no luck finding anything related to my issue.

UPDATE:

Output of uname -a:

Linux master-srv1 2.6.18-274.7.1.el5xen #1 SMP Thu Oct 20 17:06:34 EDT 2011 x86_64 x86_64 x86_64 GNU/Linux

Output of yum info openvpn:

Installed Packages
Name       : openvpn
Arch       : x86_64
Version    : 2.2.0
Release    : 3.el5.rf
Size       : 1.0 M
Repo       : installed
Summary    : Robust and highly flexible VPN daemon
URL        : http://openvpn.net/
License    : GPL
Description: OpenVPN is a robust and highly flexible tunneling application.
           : 
           : OpenVPN supports SSL/TLS security, ethernet bridging, TCP or UDP tunnel
           : transport through proxies or NAT, support for dynamic IP addresses and
           : DHCP, scalability to hundreds or thousands of users, and portability to
           : most major OS platforms.

Output of strace /etc/init.d/openvpn start: (Couldn't fit entire output, so I've pasted the end of it)

...

stat("/etc/sysconfig/network", {st_mode=S_IFREG|0644, st_size=79, ...}) = 0
access("/etc/sysconfig/network", X_OK)  = -1 EACCES (Permission denied)
access("/etc/sysconfig/network", R_OK)  = 0
open("/etc/sysconfig/network", O_RDONLY) = 3
fstat(3, {st_mode=S_IFREG|0644, st_size=79, ...}) = 0
read(3, "NETWORKING=yes\nNETWORKING_IPV6=y"..., 79) = 79
close(3)                                = 0
rt_sigprocmask(SIG_BLOCK, NULL, [], 8)  = 0
rt_sigprocmask(SIG_BLOCK, NULL, [], 8)  = 0
rt_sigprocmask(SIG_BLOCK, NULL, [], 8)  = 0
rt_sigprocmask(SIG_BLOCK, NULL, [], 8)  = 0
rt_sigprocmask(SIG_BLOCK, NULL, [], 8)  = 0
rt_sigprocmask(SIG_BLOCK, NULL, [], 8)  = 0
rt_sigprocmask(SIG_BLOCK, NULL, [], 8)  = 0
rt_sigprocmask(SIG_BLOCK, NULL, [], 8)  = 0
rt_sigprocmask(SIG_BLOCK, NULL, [], 8)  = 0
rt_sigprocmask(SIG_SETMASK, [], NULL, 8) = 0
rt_sigprocmask(SIG_BLOCK, NULL, [], 8)  = 0
rt_sigprocmask(SIG_BLOCK, NULL, [], 8)  = 0
rt_sigprocmask(SIG_BLOCK, NULL, [], 8)  = 0
rt_sigprocmask(SIG_BLOCK, NULL, [], 8)  = 0
stat("/usr/sbin/openvpn", {st_mode=S_IFREG|0755, st_size=614892, ...}) = 0
rt_sigprocmask(SIG_SETMASK, [], NULL, 8) = 0
rt_sigprocmask(SIG_BLOCK, NULL, [], 8)  = 0
rt_sigprocmask(SIG_BLOCK, NULL, [], 8)  = 0
rt_sigprocmask(SIG_BLOCK, NULL, [], 8)  = 0
fstat(1, {st_mode=S_IFCHR|0620, st_rdev=makedev(136, 0), ...}) = 0
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x2b10685b9000
write(1, "Starting openvpn: ", 18Starting openvpn: )      = 18
rt_sigprocmask(SIG_BLOCK, [INT CHLD], [], 8) = 0
rt_sigprocmask(SIG_BLOCK, [CHLD], [INT CHLD], 8) = 0
rt_sigprocmask(SIG_SETMASK, [INT CHLD], NULL, 8) = 0
lseek(255, -7, SEEK_CUR)                = 5545
clone(child_stack=0, flags=CLONE_CHILD_CLEARTID|CLONE_CHILD_SETTID|SIGCHLD, child_tidptr=0x2b1064fe4fe0) = 24440
rt_sigprocmask(SIG_SETMASK, [], NULL, 8) = 0
rt_sigprocmask(SIG_BLOCK, [CHLD], [], 8) = 0
rt_sigprocmask(SIG_SETMASK, [], NULL, 8) = 0
rt_sigprocmask(SIG_BLOCK, [CHLD], [], 8) = 0
rt_sigaction(SIGINT, {0x436f40, [], SA_RESTORER, 0x34e10302d0}, {SIG_DFL, [], SA_RESTORER, 0x34e10302d0}, 8) = 0
wait4(-1, [{WIFEXITED(s) && WEXITSTATUS(s) == 0}], 0, NULL) = 24440
rt_sigprocmask(SIG_SETMASK, [], NULL, 8) = 0
--- SIGCHLD (Child exited) @ 0 (0) ---
wait4(-1, 0x7fff46543fe4, WNOHANG, NULL) = -1 ECHILD (No child processes)
rt_sigreturn(0xffffffffffffffff)        = 0
rt_sigaction(SIGINT, {SIG_DFL, [], SA_RESTORER, 0x34e10302d0}, {0x436f40, [], SA_RESTORER, 0x34e10302d0}, 8) = 0
rt_sigprocmask(SIG_BLOCK, NULL, [], 8)  = 0
stat("/etc/openvpn/openvpn-startup", {st_mode=S_IFREG|0740, st_size=143, ...}) = 0
rt_sigprocmask(SIG_SETMASK, [], NULL, 8) = 0
rt_sigprocmask(SIG_BLOCK, [INT CHLD], [], 8) = 0
clone(child_stack=0, flags=CLONE_CHILD_CLEARTID|CLONE_CHILD_SETTID|SIGCHLD, child_tidptr=0x2b1064fe4fe0) = 24441
rt_sigprocmask(SIG_SETMASK, [], NULL, 8) = 0
rt_sigprocmask(SIG_BLOCK, [CHLD], [], 8) = 0
rt_sigprocmask(SIG_SETMASK, [], NULL, 8) = 0
rt_sigprocmask(SIG_BLOCK, [CHLD], [], 8) = 0
rt_sigaction(SIGINT, {0x436f40, [], SA_RESTORER, 0x34e10302d0}, {SIG_DFL, [], SA_RESTORER, 0x34e10302d0}, 8) = 0
wait4(-1, 
Running bridge-start...Done
[{WIFEXITED(s) && WEXITSTATUS(s) == 0}], 0, NULL) = 24441
rt_sigprocmask(SIG_SETMASK, [], NULL, 8) = 0
--- SIGCHLD (Child exited) @ 0 (0) ---
wait4(-1, 0x7fff46544144, WNOHANG, NULL) = -1 ECHILD (No child processes)
rt_sigreturn(0xffffffffffffffff)        = 0
rt_sigaction(SIGINT, {SIG_DFL, [], SA_RESTORER, 0x34e10302d0}, {0x436f40, [], SA_RESTORER, 0x34e10302d0}, 8) = 0
rt_sigprocmask(SIG_BLOCK, NULL, [], 8)  = 0
stat("/var/run/openvpn", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0
rt_sigprocmask(SIG_SETMASK, [], NULL, 8) = 0
rt_sigprocmask(SIG_BLOCK, NULL, [], 8)  = 0
stat("/var/lock/subsys/openvpn", 0x7fff46544910) = -1 ENOENT (No such file or directory)
rt_sigprocmask(SIG_SETMASK, [], NULL, 8) = 0
open("/var/run/openvpn/", O_RDONLY|O_NONBLOCK|O_DIRECTORY) = 3
fcntl(3, F_SETFD, FD_CLOEXEC)           = 0
brk(0x1170c000)                         = 0x1170c000
getdents(3, /* 2 entries */, 32768)     = 48
getdents(3, /* 0 entries */, 32768)     = 0
brk(0x11704000)                         = 0x11704000
close(3)                                = 0
stat(".", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0
stat("/sbin/rm", 0x7fff46544b50)        = -1 ENOENT (No such file or directory)
stat("/usr/sbin/rm", 0x7fff46544b50)    = -1 ENOENT (No such file or directory)
stat("/bin/rm", {st_mode=S_IFREG|0755, st_size=47088, ...}) = 0
access("/bin/rm", X_OK)                 = 0
access("/bin/rm", R_OK)                 = 0
stat("/bin/rm", {st_mode=S_IFREG|0755, st_size=47088, ...}) = 0
access("/bin/rm", X_OK)                 = 0
access("/bin/rm", R_OK)                 = 0
rt_sigprocmask(SIG_BLOCK, [INT CHLD], [], 8) = 0
clone(child_stack=0, flags=CLONE_CHILD_CLEARTID|CLONE_CHILD_SETTID|SIGCHLD, child_tidptr=0x2b1064fe4fe0) = 24460
rt_sigprocmask(SIG_SETMASK, [], NULL, 8) = 0
rt_sigprocmask(SIG_BLOCK, [CHLD], [], 8) = 0
rt_sigprocmask(SIG_SETMASK, [], NULL, 8) = 0
rt_sigprocmask(SIG_BLOCK, [CHLD], [], 8) = 0
rt_sigaction(SIGINT, {0x436f40, [], SA_RESTORER, 0x34e10302d0}, {SIG_DFL, [], SA_RESTORER, 0x34e10302d0}, 8) = 0
wait4(-1, [{WIFEXITED(s) && WEXITSTATUS(s) == 0}], 0, NULL) = 24460
rt_sigprocmask(SIG_SETMASK, [], NULL, 8) = 0
--- SIGCHLD (Child exited) @ 0 (0) ---
wait4(-1, 0x7fff46544824, WNOHANG, NULL) = -1 ECHILD (No child processes)
rt_sigreturn(0xffffffffffffffff)        = 0
rt_sigaction(SIGINT, {SIG_DFL, [], SA_RESTORER, 0x34e10302d0}, {0x436f40, [], SA_RESTORER, 0x34e10302d0}, 8) = 0
stat("/etc", {st_mode=S_IFDIR|0755, st_size=12288, ...}) = 0
stat("/etc/openvpn", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0
chdir("/etc/openvpn")                   = 0
pipe([3, 4])                            = 0
rt_sigprocmask(SIG_BLOCK, [CHLD], [], 8) = 0
rt_sigprocmask(SIG_SETMASK, [], NULL, 8) = 0
rt_sigprocmask(SIG_BLOCK, [INT CHLD], [], 8) = 0
clone(child_stack=0, flags=CLONE_CHILD_CLEARTID|CLONE_CHILD_SETTID|SIGCHLD, child_tidptr=0x2b1064fe4fe0) = 24461
rt_sigprocmask(SIG_SETMASK, [], NULL, 8) = 0
rt_sigaction(SIGCHLD, {0x436360, [], SA_RESTORER, 0x34e10302d0}, {0x436360, [], SA_RESTORER, 0x34e10302d0}, 8) = 0
close(4)                                = 0
read(3, "openvpn.conf\n", 128)          = 13
read(3, "", 128)                        = 0
--- SIGCHLD (Child exited) @ 0 (0) ---
wait4(-1, [{WIFEXITED(s) && WEXITSTATUS(s) == 0}], WNOHANG, NULL) = 24461
wait4(-1, 0x7fff46544e94, WNOHANG, NULL) = -1 ECHILD (No child processes)
rt_sigreturn(0)                         = 0
close(3)                                = 0
rt_sigprocmask(SIG_BLOCK, [CHLD], [], 8) = 0
rt_sigaction(SIGINT, {0x436f40, [], SA_RESTORER, 0x34e10302d0}, {SIG_DFL, [], SA_RESTORER, 0x34e10302d0}, 8) = 0
rt_sigprocmask(SIG_SETMASK, [], NULL, 8) = 0
rt_sigaction(SIGINT, {SIG_DFL, [], SA_RESTORER, 0x34e10302d0}, {0x436f40, [], SA_RESTORER, 0x34e10302d0}, 8) = 0
rt_sigprocmask(SIG_BLOCK, NULL, [], 8)  = 0
stat("openvpn.sh", 0x7fff46544840)      = -1 ENOENT (No such file or directory)
rt_sigprocmask(SIG_SETMASK, [], NULL, 8) = 0
stat("/bin/rm", {st_mode=S_IFREG|0755, st_size=47088, ...}) = 0
access("/bin/rm", X_OK)                 = 0
access("/bin/rm", R_OK)                 = 0
rt_sigprocmask(SIG_BLOCK, [INT CHLD], [], 8) = 0
rt_sigprocmask(SIG_BLOCK, [CHLD], [INT CHLD], 8) = 0
rt_sigprocmask(SIG_SETMASK, [INT CHLD], NULL, 8) = 0
clone(child_stack=0, flags=CLONE_CHILD_CLEARTID|CLONE_CHILD_SETTID|SIGCHLD, child_tidptr=0x2b1064fe4fe0) = 24463
rt_sigprocmask(SIG_SETMASK, [], NULL, 8) = 0
--- SIGCHLD (Child exited) @ 0 (0) ---
wait4(-1, [{WIFEXITED(s) && WEXITSTATUS(s) == 0}], WNOHANG, NULL) = 24463
wait4(-1, 0x7fff46544684, WNOHANG, NULL) = -1 ECHILD (No child processes)
rt_sigreturn(0)                         = 0
rt_sigprocmask(SIG_BLOCK, [CHLD], [], 8) = 0
rt_sigprocmask(SIG_SETMASK, [], NULL, 8) = 0
rt_sigprocmask(SIG_BLOCK, [CHLD], [], 8) = 0
rt_sigaction(SIGINT, {0x436f40, [], SA_RESTORER, 0x34e10302d0}, {SIG_DFL, [], SA_RESTORER, 0x34e10302d0}, 8) = 0
rt_sigprocmask(SIG_SETMASK, [], NULL, 8) = 0
rt_sigaction(SIGINT, {SIG_DFL, [], SA_RESTORER, 0x34e10302d0}, {0x436f40, [], SA_RESTORER, 0x34e10302d0}, 8) = 0
write(1, "/usr/sbin/openvpn --daemon --wri"..., 107/usr/sbin/openvpn --daemon --writepid /var/run/openvpn/openvpn.pid --config openvpn.conf --cd /etc/openvpn
) = 107
rt_sigprocmask(SIG_BLOCK, [INT CHLD], [], 8) = 0
clone(child_stack=0, flags=CLONE_CHILD_CLEARTID|CLONE_CHILD_SETTID|SIGCHLD, child_tidptr=0x2b1064fe4fe0) = 24464
rt_sigprocmask(SIG_SETMASK, [], NULL, 8) = 0
rt_sigprocmask(SIG_BLOCK, [CHLD], [], 8) = 0
rt_sigprocmask(SIG_SETMASK, [], NULL, 8) = 0
rt_sigprocmask(SIG_BLOCK, [CHLD], [], 8) = 0
rt_sigaction(SIGINT, {0x436f40, [], SA_RESTORER, 0x34e10302d0}, {SIG_DFL, [], SA_RESTORER, 0x34e10302d0}, 8) = 0
wait4(-1, [{WIFSIGNALED(s) && WTERMSIG(s) == SIGSEGV}], 0, NULL) = 24464
fstat(2, {st_mode=S_IFCHR|0620, st_rdev=makedev(136, 0), ...}) = 0
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x2b10685ba000
open("/usr/share/locale/en_US.UTF-8/LC_MESSAGES/libc.mo", O_RDONLY) = -1 ENOENT (No such file or directory)
open("/usr/share/locale/en_US.utf8/LC_MESSAGES/libc.mo", O_RDONLY) = -1 ENOENT (No such file or directory)
open("/usr/share/locale/en_US/LC_MESSAGES/libc.mo", O_RDONLY) = -1 ENOENT (No such file or directory)
open("/usr/share/locale/en.UTF-8/LC_MESSAGES/libc.mo", O_RDONLY) = -1 ENOENT (No such file or directory)
open("/usr/share/locale/en.utf8/LC_MESSAGES/libc.mo", O_RDONLY) = -1 ENOENT (No such file or directory)
open("/usr/share/locale/en/LC_MESSAGES/libc.mo", O_RDONLY) = -1 ENOENT (No such file or directory)
write(2, "/etc/init.d/openvpn: line 148: 2"..., 129/etc/init.d/openvpn: line 148: 24464 Segmentation fault      $openvpn --daemon --writepid $piddir/$bn.pid --config $c --cd $work
) = 129
rt_sigprocmask(SIG_SETMASK, [], NULL, 8) = 0
--- SIGCHLD (Child exited) @ 0 (0) ---
wait4(-1, 0x7fff46544b74, WNOHANG, NULL) = -1 ECHILD (No child processes)
rt_sigreturn(0xffffffffffffffff)        = 0
rt_sigaction(SIGINT, {SIG_DFL, [], SA_RESTORER, 0x34e10302d0}, {0x436f40, [], SA_RESTORER, 0x34e10302d0}, 8) = 0
rt_sigprocmask(SIG_BLOCK, NULL, [], 8)  = 0
rt_sigprocmask(SIG_SETMASK, [], NULL, 8) = 0
rt_sigprocmask(SIG_BLOCK, [CHLD], [], 8) = 0
rt_sigprocmask(SIG_SETMASK, [], NULL, 8) = 0
rt_sigprocmask(SIG_BLOCK, NULL, [], 8)  = 0
rt_sigprocmask(SIG_SETMASK, [], NULL, 8) = 0
rt_sigprocmask(SIG_BLOCK, NULL, [], 8)  = 0
rt_sigprocmask(SIG_BLOCK, NULL, [], 8)  = 0
rt_sigprocmask(SIG_SETMASK, [], NULL, 8) = 0
rt_sigprocmask(SIG_BLOCK, NULL, [], 8)  = 0
rt_sigprocmask(SIG_BLOCK, NULL, [], 8)  = 0
rt_sigprocmask(SIG_SETMASK, [], NULL, 8) = 0
write(1, "\33[60G", 5                                      )                  = 5
write(1, "[", 1[)                        = 1
rt_sigprocmask(SIG_BLOCK, NULL, [], 8)  = 0
rt_sigprocmask(SIG_SETMASK, [], NULL, 8) = 0
write(1, "\33[0;31m", 7)                = 7
write(1, "FAILED", 6FAILED)                   = 6
rt_sigprocmask(SIG_BLOCK, NULL, [], 8)  = 0
rt_sigprocmask(SIG_SETMASK, [], NULL, 8) = 0
write(1, "\33[0;39m", 7)                = 7
write(1, "]", 1])                        = 1
)                       = 1
rt_sigprocmask(SIG_SETMASK, [], NULL, 8) = 0
rt_sigprocmask(SIG_BLOCK, NULL, [], 8)  = 0
stat("/usr/bin/rhgb-client", 0x7fff465449a0) = -1 ENOENT (No such file or directory)
rt_sigprocmask(SIG_SETMASK, [], NULL, 8) = 0
rt_sigprocmask(SIG_SETMASK, [], NULL, 8) = 0
write(1, "\n", 1
)                       = 1
rt_sigprocmask(SIG_BLOCK, NULL, [], 8)  = 0
rt_sigprocmask(SIG_SETMASK, [], NULL, 8) = 0
rt_sigprocmask(SIG_BLOCK, NULL, [], 8)  = 0
read(255, "exit 0\n", 5552)             = 7
rt_sigprocmask(SIG_SETMASK, [], NULL, 8) = 0
exit_group(0)                           = ?
centos openvpn segmentation-fault

2 Answers

  1. Best Answer

    Boban P. was absolutely correct in his comment about the openvpn-auth-ldap plugin being the cause of the segfault. I ruled it out initially since I commented the plugin line out and tried restarting openvpn and it gave the same error. However, openvpn apparently was still trying to use the plugin, so it was still throwing the segfault. I don't know exactly what the issue with the plugin was, but my best guess is that I installed the incorrect version for my system. I made sure to install the one for CentOS from http://pkgs.org/centos-5-rhel-5/flexbox-x86_64/openvpn-auth-ldap-2.0.3-3.centos5.x86_64.rpm.html and it works flawlessly again.

    The credit should go to Boban P. for his suggestion regarding openvpn-auth-ldap being the issue.

    • 1

  2. I'd uninstall heartbeat, or at least look at the packages associated with heartbeat. They do something with network interfaces, if I recall, and may be interfering with tun0.

    I'm not sure if this is the best way to implement HA on OpenVPN. My impression is that the recommended way is to use multiple "server" lines in the client configuration. This won't be seemless, but OpenVPN isn't going to fail over seemlessly anyway, even with Heartbeat, because the session will need to be reinitialized.

    • 0