Naftuli Kay's questions

I have been using a suite of filesystem testing tools to benchmark and abuse a GlusterFS volume. The volume is a replica 3 volume spread out over 6 hosts.

Fio, iozone, and Bonnie indicate to me that Gluster is working just fine and the bandwidth is roughly equal to that of the client and server network adapters, so performance can't really be improved. Most of my test cases operated on 32gb files, apart from iozone and Bonnie.

I have gotten reports of split brain occurring for certain files which are being concurrently written to by multiple clients. All of the documentation I have read seems to indicate that split brain largely occurs when network partitions happen, and this is clearly not the case, judging from the logs.

Unfortunately, this split brain seems to occur only when using a certain hosted service, and I have zero introspection into how that service operates, what version of Gluster client it has, etc. The servers are running the latest 4.0 release.

Judging from the failure case I have been presented with ("split brain happens when two containers are writing to the same file at the same time"), I need a test that will reproduce a similar situation.

I could definitely write my own test case in C or Rust, but is there something out there which will test this exact case without having to write anything?

I do have access (but not introspection) into this hosted service, so I will probably test that too. I'm also scratching my head at the actual problem: what is the desired outcome when two programs write different data to the same file at the same time?

EDIT: The servers are running the latest CentOS 7 release. My testing client server is also running the same. The underlying filesystem is XFS.

Is there a specific test case that I can use to try to recreate the problem?

I am on CentOS 7.4 attempting to configure dnsmasq for DNS caching and per-zone DNS servers for running Consul. Essentially, the dnsmasq configuration boils down to this:

# filter everything with consul in the name and send to local consul resolver
server=/consul/127.0.0.1#8600

The problem I'm having is getting the local resolver into /etc/resolv.conf without replacing any other resolvers in the file.

I used to be able to do this using dhclient configuration like so:

prepend domain-name-servers 127.0.0.1;

Since everything is NetworkManager now, I have discovered that the way to use dnsmasq is to add the following line to the [main] section of /etc/NetworkManager/NetworkManager.conf:

dns=dnsmasq

Before restarting NetworkManager, I see that my /etc/resolv.conf contains the network DNS server received by DCHP:

# Generated by NetworkManager
search nowhere
nameserver 10.0.2.3
options single-request-reopen

Upon restarting NetworkManager with the above configuration change, it supersedes my network DNS server with 127.0.0.1:

# Generated by NetworkManager
search nowhere
nameserver 127.0.0.1
options single-request-reopen

Since dnsmasq uses the /etc/resolv.conf to detect local resolvers, I'd like to leave the resolver(s) in /etc/resolv.conf so that dnsmasq can discover them.

Is there a way to achieve the previous functionality of having NetworkManager just prepend the local resolver instead?

I am on Ubuntu 16.04 with the latest NGINX installed from their official repository:

$ sudo nginx -v
nginx version: nginx/1.11.8

I have the following defined in /etc/nginx/sites-available/greendot.conf:

# local dnsmasq
resolver 127.0.0.1;

map $http_upgrade $connection_upgrade {
  default upgrade;
  '' close;
}

upstream websocket {
  zone elixr 64k;
  server greendot-elixr-1:4000 resolve;
  server greendot-elixr-2:4000 resolve;
}

server {
  listen 4000;
  location / {
    proxy_pass http://websocket;
    proxy_http_version 1.1;
    proxy_set_header Upgrade $http_upgrade;
    proxy_set_header Connection $connection_upgrade;
  }
}

I have followed the documentation on setting up DNS resolved upstream servers, and yet this configuration fails:

$ sudo nginx -T
nginx: [emerg] invalid parameter "resolve" in /etc/nginx/sites-enabled/greendot-nginx.conf:11
nginx: configuration file /etc/nginx/nginx.conf test failed

My goal is to simply create a server which uses DNS to periodically resolve its members. What am I doing wrong?

I'm using NGINX 1.8.1 on CentOS 6 and 7. During our deploys, it is required that NGINX is stopped, the deploy executes, and NGINX starts again. Unfortunately, NGINX during a stop seems to just 500 all connections. As I'm using Ansible to update 2/10 hosts at a time, it's likely therefore that many of our clients will get 500'd throughout our deploy process.

Is there a way via NGINX settings to have the NGINX process drain all live connections (ie wait for them to complete) while rejecting all new connections during a stop?

I know that Amazon's Elastic Load Balancers do have this functionality, is there an equivalent for NGINX?

Typically, load balancers like Amazon's Elastic Load Balancers use a DNS record set with multiple A records to provide multiple load balancer instances which can handle traffic to requesting endpoints:

$ dig +short my-fancy-elb.us-east-1.elb.amazonaws.com
10.0.1.1
10.0.1.2

If I attempt to curl this URL in verbose mode, I notice that curl seems to round-robin attempts to the two IP addresses:

$ curl -ivs http://my-fancy-elb.us-east-1.elb.amazonaws.com | grep -i 'connected'
* Connected to my-fancy-elb.us-east-1.elb.amazonaws.com (10.0.1.1)
$ curl -ivs http://my-fancy-elb.us-east-1.elb.amazonaws.com | grep -i 'connected'
* Connected to my-fancy-elb.us-east-1.elb.amazonaws.com (10.0.1.2)

Is the fact that curl does round-robin on the A records described in the record set done by the curl binary itself or is it something that the Linux kernel does for it?

TCP exists at layer 4 and DNS exists at layer 7, so I'd imagine that individual binaries and libraries would have to implement their own load-balancing and failover: fetching the DNS record set for the given domain name and choosing a TCP address to connect to from that set.

Can I reasonably expect that programming languages, browsers, and libraries like curl will do load-balancing and failover on A records for me?

At my organization we have a number of simple-to-use base AMIs for different services such as ECS and Docker. Since many of our projects involve CloudFormation, we're using cfn-bootstrap, which consists of a couple of scripts and a service which run on boot to install certain packages and do certain configuration management tasks for us.

On startup of a system, an equivalent of the following script must be executed:

#!/bin/bash

# capture stderr only
output="$(cfn-init -s $STACK_NAME -r $RESOURCE_NAME --region $REGION >/dev/null)"

# if it failed, signal to CloudFormation that it failed and include a reason
returncode=$?
if [[ $returncode == 0]]; then
    cfn-signal -e $returncode -r "$output"
    exit $returncode
fi

# otherwise, signal success
cfn-signal -s

I was thinking of running this as a systemd oneshot service which runs After=network.target and WantedBy=multi-user.target.

The only problem is that I'd like my AMI to be flexible and only execute this if a certain file exists. Rather than embedding the above script into the EC2 user data, I can have the user data just define an environment file which defines the variables I need and only run my one-shot service if that environment file exists:

#cloud-init
write_files:
    - path: /etc/sysconfig/cloudformation
      # ...
      content: |
          CFN_STACK_NAME="stack-name"
          CFN_RESOURCE="resource-name"
          CFN_REGION="region"

Is there a way to make systemd only run a service if a given condition is met?

I'm trying to have OpenVPN authenticate users via PAM over LDAP to an Active Directory server.

Here are the relevant parts of my configuration files:

/etc/openvpn/server.conf:

# ...
plugin /usr/lib64/openvpn/plugins/openvpn-plugin-auth-pam.so ldap-login

/etc/nslcd.conf:

uri ldap://prod-adc1.mydomain.local
base DC=MyDomain,DC=Local
uid nslcd
gid ldap

/etc/pam.d/ldap-login:

auth      sufficient  pam_ldap.so minimum_uid=1000 use_first_pass
auth      required    pam_deny.so
password  sufficient  pam_ldap.so minimum_uid=1000 use_first_pass
password  required    pam_deny.so

When I try authenticating, I see the following in the logs:

/var/log/secure:

Dec  4 22:22:42 localhost openvpn[25505]: pam_ldap(ldap-login:auth): failed to get password: Authentication failure

/var/log/messages:

Dec  4 22:38:28 localhost openvpn[25504]: 1.2.3.4:37503 TLS: Initial packet from [AF_INET]1.2.3.4:37503, sid=c2d806cc 5c7c7ace
Dec  4 22:38:28 localhost openvpn[25504]: 1.2.3.4:37503 PLUGIN_CALL: POST /usr/lib64/openvpn/plugins/openvpn-plugin-auth-pam.so/PLUGIN_AUTH_USER_PASS_VERIFY status=1
Dec  4 22:38:28 localhost openvpn[25504]: 1.2.3.4:37503 PLUGIN_CALL: plugin function PLUGIN_AUTH_USER_PASS_VERIFY failed with status 1: /usr/lib64/openvpn/plugins/openvpn-plugin-auth-pam.so
Dec  4 22:38:28 localhost openvpn[25504]: 1.2.3.4:37503 TLS Auth Error: Auth Username/Password verification failed for peer
Dec  4 22:38:28 localhost openvpn[25504]: 1.2.3.4:37503 Control Channel: TLSv1.2, cipher TLSv1/SSLv3 DHE-RSA-AES256-GCM-SHA384
Dec  4 22:38:28 localhost openvpn[25504]: 1.2.3.4:37503 Peer Connection Initiated with [AF_INET]1.2.3.4:37503
Dec  4 22:38:31 localhost openvpn[25504]: 1.2.3.4:37503 PUSH: Received control message: 'PUSH_REQUEST'
Dec  4 22:38:31 localhost openvpn[25504]: 1.2.3.4:37503 Delayed exit in 5 seconds
Dec  4 22:38:31 localhost openvpn[25504]: 1.2.3.4:37503 SENT CONTROL [UNDEF]: 'AUTH_FAILED' (status=1)
Dec  4 22:38:36 localhost openvpn[25504]: 1.2.3.4:37503 SIGTERM[soft,delayed-exit] received, client-instance exiting

Have I configured PAM wrong? Is there something I'm missing in my login configuration?

EDIT: I'm able to login if I do the following:

ldapsearch -h prod-adc1.mydomain.local -b "DC=MyDomain,DC=Local" -x -D "[email protected]" -W

How do I map this over to the PAM LDAP module to get OpenVPN logins working as expected?

Here's my OpenVPN server configuration:

port 1194
proto udp

dev tun

ca ca.crt
cert server.crt
key server.key
dh dh4096.pem

topology subnet

server 10.224.127.0 255.255.255.0 # AKA 10.224.127.0/24

ifconfig-pool-persist ipp.txt

push "route 10.0.0.0 255.0.0.0"
push "dhcp-option DNS 10.0.9.102"
push "dhcp-option DNS 10.0.8.186"
push "dhcp-option DOMAIN dc1.lam.myname.net aws.myname.net myname.local"

keepalive 10 120

comp-lzo

max-clients 10

user nobody
group nobody

persist-key
persist-tun

status openvpn-status.log

verb 3

cipher AES-256-CBC
auth SHA256
tls-version-min 1.2
tls-cipher TLS-DHE-RSA-WITH-AES-256-GCM-SHA384 TLS-DHE-RSA-WITH-AES-256-CBC-SHA256 TLS-DHE-RSA-WITH-AES-128-GCM-SHA256 TLS-DHE-RSA-WITH-AES-128-CBC-SHA256 TLS-DHE-RSA-WITH-AES-256-CBC-SHA TLS-DHE-RSA-WITH_AES-128-CBC-SHA

client-cert-not-required
username-as-common-name
plugin /usr/lib64/openvpn/plugins/openvpn-plugin-auth-pam.so login

I basically want to do the following:

• Use our internal DNS servers for name resolution.
• Allocate client IP addresses within the 10.224.127.0/24 range.
• Only route traffic going to 10.0.0.0/8 over the VPN link.
• Use a few search domains for DNS lookups.
• Generic hardening and good crypto.
• Use PAM for logging in.

So it "basically" works, with a few caveats. DNS resolution only works for a few seconds after the VPN server comes online and then stops working. I can route TCP packets just fine regardless of whether DNS is working or not.

Why does DNS only work for the first few seconds of my connection? What am I doing wrong?

The clients I plan on using are Linux and OSX. How can I push DNS resolution down to them?

I'm often working with tunneled ports where I'll SSH forward a remote port 80 to a local port 8080.

One of the interesting problems I run into is that when I have a rule like this in NGINX:

rewrite ^/(.+)/$ /$1 permanent;

The rewrite will take a request that looks like this:

curl -is -X GET http://localhost:8080/one/two/three/

And redirect it to a URL like this:

http://localhost/one/two/three

It strips the port from the host header and I get redirected to port 80, which is unbound and breaks things.

Can I configure NGINX to respect the host port (as present in the Host header) when doing redirects?

As can be seen in the request headers, the Host header includes the port and I'd like NGINX to use this value for all redirects to maintain the original port the client used to access the server.

My NGINX site config looks like this:

server {
  listen *:80;
  server_name           _;

  index  index.html index.htm;

  access_log            /var/log/nginx/default.access.log combined;
  error_log             /var/log/nginx/default.error.log;

  location / {
    root      /vagrant/_site;
    index     index.html index.htm index.php;
    try_files $uri $uri.html $uri/ =404;
  }

  port_in_redirect on;
  server_name_in_redirect off;
}

The exact steps to reproduce look like this:

$ curl -is http://localhost:8080/2015/08/from-hell-flying-united-airlines
HTTP/1.1 301 Moved Permanently
Server: nginx/1.8.0
Date: Wed, 02 Sep 2015 19:43:10 GMT
Content-Type: text/html
Content-Length: 184
Location: http://localhost/2015/08/from-hell-flying-united-airlines/
Connection: keep-alive

<html>
<head><title>301 Moved Permanently</title></head>
<body bgcolor="white">
<center><h1>301 Moved Permanently</h1></center>
<hr><center>nginx/1.8.0</center>
</body>
</html>

I'm not sure why it's redirecting to append a slash in the first place, but the redirect is breaking everything.

I have two directories which I need to serve static assets out of:

1. /srv/web: Static assets which include images, JavaScript, HTML, etc.
2. /srv/php: Dynamic PHP scripts alongside some static assets.

I'm using NGINX and have configured it like so:

server {
    listen 80;
    server_name _;

    # root /;
    index index.php index.html index.htm;
    try_files /srv/web/$uri /srv/php/$uri =404;

    location ~ \.php$ {
        root /srv/php;
        try_files $uri =404;
        fastcgi_pass unix:/var/run/php5-fpm.sock;
        fastcgi_index index.php;
#       fastcgi_param SCRIPT_FILENAME /srv/php$fastcgi_script_name;
        include fastcgi_params;
    }
}

I'm on Ubuntu 14.04, PHP FPM package version is 5.5.9, NGINX package version is 1.4.6.

The simple goal here is to serve static files out of /srv/web first, failing that /srv/php, failing that, return 404. All files ending in \.php$ will be requested from PHP FPM over the Unix socket, and this is working.

The problem I'm currently having is that the index directive on the server isn't working as planned. I have an index.html file in /srv/web, and when I do

curl -is http://localhost/

I get a 404.

Is this the most ideal way of setting up an NGINX site with multiple filesystem folders to serve from alongside PHP? Any ideas as to why my static index isn't working?

Update

As per AD7six's answer below, I've updated my configuration to look like this:

server {
    listen 80;
    server_name _; # listen at all host names

    # serve static files first from /srv/web, then from /srv/php, and any dynamic PHP files from
    # FastCGI/FPM at the Unix socket.
    location / {
        root /srv/web;
        index index.html index.htm;
        try_files $uri $uri/ @php;
    }

    location @php {
        root /srv/php;
        index index.php;
        try_files $uri $uri/ =404;
    }

    location ~ \.php$ {
        root /srv/php;
        fastcgi_pass unix:/var/run/php5-fpm.sock;
        fastcgi_index index.php;
        fastcgi_param SCRIPT_FILENAME /srv/php/$fastcgi_script_name;
        include fastcgi_params;
    }
}

My directory listing looks like this:

/srv/
|-- php
|   |-- demo.php
|   |-- index.php
|   `-- phpstatic.txt
`-- web
    |-- static.html
    `-- subdir
        `-- index.html

3 directories, 5 files

Getting static files and PHP files works as planned and getting /subdir/ with its index works fine, but if I GET /, I get a 403 forbidden, and nginx complains of directory listing:

2015/08/24 21:50:59 [error] 9159#0: *7 directory index of "/srv/web/" is forbidden, client: 127.0.0.1, server: _, request: "GET / HTTP/1.1", host: "localhost"

Not sure why this is failing, but it smells like progress at least.

I've just generated a ECDSA key with ssh-keygen:

ssh-keygen -t ecdsa -b 521 

I then proceeded to copy this key to my server:

cat .ssh/id_ecdsa.pub | ssh myserver "tee -a .ssh/authorized_keys"

I've verified that my key is in the file.

However, when I attempt to connect, my connection is rejected:

ssh -v -i .ssh/id_ecdsa myserver

Logs:

OpenSSH_5.9p1 Debian-5ubuntu1.1, OpenSSL 1.0.1 14 Mar 2012
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: /etc/ssh/ssh_config line 19: Applying options for *
debug1: Connecting to myserver [192.168.1.1] port 22.
debug1: Connection established.
debug1: identity file .ssh/id_ecdsa type 3
debug1: Checking blacklist file /usr/share/ssh/blacklist.ECDSA-521
debug1: Checking blacklist file /etc/ssh/blacklist.ECDSA-521
debug1: identity file .ssh/id_ecdsa-cert type -1
debug1: Remote protocol version 2.0, remote software version OpenSSH_6.1
debug1: match: OpenSSH_6.1 pat OpenSSH*
debug1: Enabling compatibility mode for protocol 2.0
debug1: Local version string SSH-2.0-OpenSSH_5.9p1 Debian-5ubuntu1.1
debug1: SSH2_MSG_KEXINIT sent
debug1: SSH2_MSG_KEXINIT received
debug1: kex: server->client aes128-ctr hmac-md5 none
debug1: kex: client->server aes128-ctr hmac-md5 none
debug1: SSH2_MSG_KEX_DH_GEX_REQUEST(1024<1024<8192) sent
debug1: expecting SSH2_MSG_KEX_DH_GEX_GROUP
debug1: SSH2_MSG_KEX_DH_GEX_INIT sent
debug1: expecting SSH2_MSG_KEX_DH_GEX_REPLY
debug1: Server host key: RSA 10:27:b8:78:2c:e1:e3:42:8e:e3:66:c4:cc:4e:f1:c0
debug1: Host 'myserver' is known and matches the RSA host key.
debug1: Found key in /home/naftuli/.ssh/known_hosts:73
debug1: ssh_rsa_verify: signature correct
debug1: SSH2_MSG_NEWKEYS sent
debug1: expecting SSH2_MSG_NEWKEYS
debug1: SSH2_MSG_NEWKEYS received
debug1: Roaming not allowed by server
debug1: SSH2_MSG_SERVICE_REQUEST sent
debug1: SSH2_MSG_SERVICE_ACCEPT received
debug1: Authentications that can continue: publickey
debug1: Next authentication method: publickey
debug1: Offering ECDSA public key: .ssh/id_ecdsa
debug1: Authentications that can continue: publickey
debug1: No more authentication methods to try.
Permission denied (publickey).

Found this in the server logs:

auth.info sshd[13874]: userauth_pubkey: unsupported public key algorithm: ecdsa-sha2-nistp521 [preauth]

Both my client and the server are using OpenSSH. The server's OpenSSH version is OpenSSH 6.1, my client's OpenSSH version is OpenSSH 5.9.

How can I tell which key algorithms are supported by my server?

Interestingly enough, on my Ubuntu derivative with nginx installed with apt, the www-data user has a shell:

$ cat /etc/passwd
www-data:x:33:33:www-data:/var/www:/bin/sh

Shouldn't this be set to something like /bin/false? Even though the user can't log in, isn't it dangerous to provide a shell for a system user like this by default?

I have two domains that point to the same server, one we'll call home and one we'll call web.

I'm running nginx on port 80 for HTTP and 443 for HTTPS. In my server definitions, I've defined two servers:

server {
    listen 80;
    server_name web;
    # ...
}

server {
    listen 443;
    server_name web;
    # ...
}

In practice, it works just fine. However, when I try accessing home, which points to the same IP address as web, I get served web rather than getting a 404 or the like.

How can I configure nginx to 404 requests that don't match a server name? Do I need to define a default server which just bounces things down to 404s?

I have a subdomain NS record which looks like this:

NS home ns1.nameserver.com
NS home ns2.nameserver.com

I'd like another subdomain super to point to the result of this NS name, but when I do:

CNAME super home

The name doesn't resolve.

This is a machine configured with dynamic DNS, hence home points to a 3rd-party dynamic DNS provider. To save myself the trouble of updating multiple dynamic DNS names, I'd like to simply have super resolve to the same IP address as home.

Is there a way to do this, or do I have to also configure a dynamic DNS NS name for super which is updated separately?

I have a server which serves LuCI for me on localhost:8080. I'm trying to proxy to it via nginx, but I'd like to rewrite the URLs so that instead of https://myserver/cgi-bin/luci, I have https://myserver/.

Here's what I've got so far, which doesn't seem to be working:

upstream luci {
    server localhost:8080;
}

server {
    # ...
    location / {
        proxy_pass http://luci;
        proxy_redirect / /cgi-bin/luci;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $remote_addr;
    }
}

However, with this configuration, I'm still seeing my urls prefixed with /cgi-bin/luci, and they still work somehow.

How can I rewrite my incoming requests from https://myserver/^(.*)$ to http://luci/cgi-bin/luci$1?

I'm building a service where I'll need to store a ton of files on a filesystem. I've heard in the past that it's better to separate up files by folder when dealing with large amounts of files. Is this really accurate for modern Linux filesystems such as ext3, ext4, and btrfs? If so, how much should I divide up files? Should I keep 1000 files to a folder?

The current setup I have divides them up into hundreds, thousands, tens of thousands, and hundreds of thousands like so:

stream id: 123456
1/2/3/4/123456.file

stream id: 654321
6/5/4/3/654321.file

stream id: 5
0/0/0/0/5.file

Does this make sense for my application?

According to the Wiki here, you can only allow certain users to log in over FTP using the following configuration in your /etc/vsftp.conf file:

userlist_enable=YES
userlist_file=/etc/vsftp.user_list
userlist_deny=NO

I've configured my system to use this configuration, and I only have one user which I'd like to expose over FTP named streams, so my /etc/vsftp.user_list looks like this:

streams

Interestingly enough, I cannot log in once I enable to user list. If I change userlist_enable to NO, then things work properly, but if I enable it, I can't log in all, it just keeps trying to reconnect. I don't get a login failed message, it just keeps trying to reconnect when using lftp.

My /etc/vsftp.conf file is available on Pastebin here and my /etc/vsftp.user_list is available here.

What am I doing wrong here? I'd just like to only make the streams user able to log in.

I'm working for a client here who has set up an EC2 instance for me to configure. I've installed the software asked for, now all I need to do is open up some ports. I'm currently SSH'd into the machine, it's running Ubuntu 10.04.

I've installed the ec2-ami-tools and the ec2-api-tools packages, and have tried running the following, as has been recommended elsewhere:

ec2-authorize -p 1024 default

Unfortunately, this just fails and leaves me with this nasty error message:

Required option '-K, --private-key KEY' missing (-h for usage).'

Evidently I need a key to do this, so I copied my PEM key which I received with the instance to the server and ran it again with the key:

ec2-authorize -p 1024 -K mykey.pem default

Now it's asking for a certificate:

Reqired option `-C, --cert CERT' missing (-h for usage).`

Ugh. I don't really have a certificate, so what exactly is it asking me for? I don't have access to the EC2 web interface, so how can I open a port from the Ubuntu server itself?

Is there any bandwidth overhead on using SSL on a TCP connection? I understand, of course, the processing/memory usage overhead in encrypting and decrypting packets, but as far as bandwidth is concerned, what is the difference, if any?

For example, given a XML file which is 64KB, will there be any tangible difference in the transfer size of the file over HTTP vs. HTTPS? (Ignoring mod_deflate and mod_gzip, of course)

I can't seem to get a symlink working in my Apache VirtualHost, no matter what I try and I see the following error in the error log:

Symbolic link not allowed or link target not accessible: /var/www/carddesigner

I can browse the actual symlink from Linux with no problems whatsoever:

$ ls -l /var/www | grep "carddesigner"
lrwxrwxrwx 1 rfkrocktk rfkrocktk   64 2011-02-28 16:52 carddesigner -> /home/rfkrocktk/Documents/Projects/Work/carddesigner/build/main/

Additionally, I've made sure that the my VirtualHost allows the FollowSymLinks option:

/etc/apache2/sites-enabled/000-localhost:

<VirtualHost 127.0.0.1:80>
    ServerAdmin ##########

    DocumentRoot /var/www
    <Directory />
        Options FollowSymLinks
        AllowOverride None
    </Directory>

    <Directory /var/www/>
        Options Indexes FollowSymLinks MultiViews
        AllowOverride None
        Order allow,deny
        allow from all
    </Directory>

    ScriptAlias /cgi-bin/ /usr/lib/cgi-bin/
    <Directory "/usr/lib/cgi-bin">
        AllowOverride None
        Options +ExecCGI -MultiViews +SymLinksIfOwnerMatch
        Order allow,deny
        Deny from all
    </Directory>

    ErrorLog /var/log/apache2/error.log

    # Possible values include: debug, info, notice, warn, error, crit,
    # alert, emerg.
    LogLevel debug

    CustomLog /var/log/apache2/access.log combined

    Alias /doc/ "/usr/share/doc/"
    <Directory "/usr/share/doc/">
        Options Indexes MultiViews FollowSymLinks
        AllowOverride None
        Order deny,allow
        Deny from all
        Allow from 127.0.0.0/255.0.0.0 ::1/128
    </Directory>

    RewriteEngine On
    RewriteLog "/var/log/apache2/mod_rewrite.log"
    RewriteLogLevel 9
</VirtualHost>

I can't seem to find any other configuration files that seem to override this and/or prevent symlinks from being loaded. Any ideas?

Here are my permissions on the actual referenced files:

$ ls -l ~/Documents/Projects/Work/carddesigner/build/main
total 12
drwxrwxrwx 5 rfkrocktk rfkrocktk 4096 2011-02-28 16:11 advanced
drwxrwxrwx 2 rfkrocktk rfkrocktk 4096 2011-02-28 16:10 core
drwxrwxrwx 2 rfkrocktk rfkrocktk 4096 2011-02-28 16:10 simple

Seems like the permissions are good to go, right?