Postfix + Dovecot taking too long to send emails

I'm getting 504 errors when trying to send emails on my Rails 4.2 site using dovecot and postfix. I also have an asynchronous email server set up using Redis and Sidekiq, when I send asynchronous emails they take 5 to 30 minutes to send.

My mail server is running on the same box as my site, but the interesting thing is that when I run my development server on my home computer, it connects to my site's mail server and sends mail quickly and without issue.

I have tried looking through my mail.log as well as every other log in my /var/log directory but I don't see anything during the period while I'm waiting for mail to be sent but a few spam attacks (I'm running SpamAssassin to shut them down).

My rails server log is also not showing anything useful, t just logs that the email was rendered (not sent) and then restarts the server after a minute.

On both my development and prod servers I'm using the following to connect:

config.action_mailer.delivery_method = :smtp
config.action_mailer.smtp_settings = {
    :address => 'mail.mysite.com',
    :domain => 'mysite.com',
    :port => 587,
    :authentication => :plain,
    :enable_starttls_auto => true,
    :openssl_verify_mode  => 'none',
    :user_name => '[email protected]',
    :password => 'test'
}
config.action_mailer.raise_delivery_errors = true
config.action_mailer.perform_deliveries = true

Here is the result from dovecot -n:

# 2.2.9: /etc/dovecot/dovecot.conf
# OS: Linux 4.2.0-23-generic x86_64 Ubuntu 14.04.4 LTS
auth_mechanisms = plain login
mail_location = maildir:~/Maildir
mail_privileged_group = mail
managesieve_notify_capability = mailto
managesieve_sieve_capability = fileinto reject envelope encoded-character vacation subaddress comparator-i;ascii-numeric relational regex imap4flags copy include variables body enotify environment mailbox date ihave
namespace inbox {
  location =
  mailbox Drafts {
    special_use = \Drafts
  }
  mailbox Junk {
    special_use = \Junk
  }
  mailbox Sent {
    special_use = \Sent
  }
  mailbox "Sent Messages" {
    special_use = \Sent
  }
  mailbox Trash {
    special_use = \Trash
  }
  prefix =
}
passdb {
  args = /etc/dovecot/dovecot-sql.conf.ext
  driver = sql
}
plugin {
  sieve = ~/.dovecot.sieve
  sieve_dir = ~/sieve
}
protocols = imap pop3 sieve
service auth-worker {
  user = vmail
}
service auth {
  unix_listener /var/spool/postfix/private/auth {
    group = postfix
    mode = 0666
    user = postfix
  }
  unix_listener /var/spool/postfix/private/dovecot-auth {
    group = postfix
    mode = 0660
    user = postfix
  }
  unix_listener auth-userdb {
    mode = 0600
    user = vmail
  }
  user = dovecot
}
service imap-login {
  inet_listener imap {
    port = 0
  }
}
service lmtp {
  unix_listener /var/spool/postfix/private/dovecot-lmtp {
    group = postfix
    mode = 0600
    user = postfix
  }
}
ssl_cert = </etc/dovecot/dovecot.pem
ssl_cipher_list = ALL:!LOW:!SSLv2:ALL:!aNULL:!ADH:!eNULL:!EXP:RC4+RSA:+HIGH:+MEDIUM
ssl_key = </etc/dovecot/private/dovecot.pem
userdb {
  args = uid=vmail gid=vmail home=/var/mail/vhosts/%d/%n
  driver = static
}
protocol pop3 {
  mail_max_userip_connections = 10
  pop3_client_workarounds = outlook-no-nuls oe-ns-eoh
}
protocol imap {
  imap_client_workarounds = delay-newmail
  mail_max_userip_connections = 10
}
protocol lda {
  deliver_log_format = msgid=%m: %$
  mail_plugins = sieve
  postmaster_address = postmaster
  quota_full_tempfail = yes
  rejection_reason = Your message to <%t> was automatically rejected:%n%r
}

And postconf -n:

alias_database = hash:/etc/aliases
alias_maps = hash:/etc/aliases
append_dot_mydomain = no
biff = no
config_directory = /etc/postfix
inet_interfaces = all
mailbox_command = /usr/lib/dovecot/deliver -c /etc/dovecot/dovecot.conf -m "${EXTENSION}"
mailbox_size_limit = 0
mydestination = localhost
myhostname = mysite.com
mynetworks = 127.0.0.0/8 [::ffff:127.0.0.0]/104 [::1]/128
myorigin = /etc/mailname
queue_directory = /var/spool/postfix
readme_directory = no
recipient_delimiter = +
relayhost =
smtpd_banner = $myhostname ESMTP $mail_name (Ubuntu)
smtpd_recipient_restrictions = permit_sasl_authenticated
smtpd_relay_restrictions = permit_sasl_authenticated, permit_mynetworks, reject_unauth_destination
smtpd_sasl_auth_enable = yes
smtpd_sasl_path = private/auth
smtpd_sasl_type = dovecot
smtpd_tls_auth_only = yes
smtpd_tls_cert_file = /etc/nginx/ssl/ssl-bundle.crt
smtpd_tls_key_file = /etc/nginx/ssl/server.key
smtpd_use_tls = yes
virtual_alias_maps = mysql:/etc/postfix/mysql-virtual-alias-maps.cf
virtual_mailbox_domains = mysql:/etc/postfix/mysql-virtual-mailbox-domains.cf
virtual_mailbox_maps = mysql:/etc/postfix/mysql-virtual-mailbox-maps.cf
virtual_transport = dovecot