Connection refused HTTPS on Apache

I'm currently trying out HTTPS on one of my sites, and I got a trial certificate from a trusted CA. I've gone through the following checklist:

• Copied all the cert files according to the CA's instructions
• Enabled mod_ssl on apache with a2enmod ssl
• Checked PHP has OpenSSL enabled
• Made a new virtual host in Apache listening to 443

• Inputted the SSL directives:

  SSLEngine on

  SSLCertificateKeyFile /etc/ssl/ssl.key/server.key

  SSLCertificateFile /etc/ssl/ssl.crt/api_my_site_com.crt

  SSLCertificateChainFile /etc/ssl/ssl.crt/apimysite.com-bundle

• Checked only apache was listening to port 443 with lsof

• Check locally and from my own PC with telnet if I could connect to 443 (to the IP of the server, not domain; trying to connect to my-site.com:443 gave me connect failed)

However, when I try to browse to https://my-site.com (obviouslly not the real domain), I get a 'Connection Refused' error. This is what Apache logs:

[Sat Jul 20 22:50:34 2013] [info] Loading certificate & private key of SSL-aware server
[Sat Jul 20 22:50:34 2013] [info] Configuring server for SSL protocol
[Sat Jul 20 22:50:34 2013] [info] RSA server certificate enables Server Gated Cryptography (SGC)
[Sat Jul 20 22:50:34 2013] [info] [client ::1] Connection to child 0 established (server my-site.com:443)
[Sat Jul 20 22:50:34 2013] [info] Seeding PRNG with 656 bytes of entropy
[Sat Jul 20 22:50:34 2013] [info] [client ::1] SSL library error 1 in handshake (server my-site:443)
[Sat Jul 20 22:50:34 2013] [info] SSL Library Error: 336027900 error:140760FC:SSL routines:SSL23_GET_CLIENT_HELLO:unknown protocol speaking not SSL to HTTPS port!?
[Sat Jul 20 22:50:34 2013] [info] [client ::1] Connection closed to child 0 with abortive shutdown (server my-site:443)

Any ideas why this is happening?

Configuration files:

ports.conf

Listen 443
NameVirtualHost *:80
Listen 80

virtualhost config

<VirtualHost *:443>
DocumentRoot /var/www/mysite/sandbox/api
ServerName api.my-site.com
RewriteEngine on
RewriteRule ^/v1/* /v1/api.php [L]
RewriteRule ^/* /index.php [L]
<Directory "/var/www/mysite/sandbox/api">
allow from all
</Directory>
Options -MultiViews
ErrorDocument 404 /404.html
AddDefaultCharset utf-8
<IfModule mod_mime> 
AddCharset utf-8 .atom .css .js .json .rss .vtt .webapp .xml
</IfModule> 
<IfModule mod_rewrite> 
Options +FollowSymlinks
RewriteCond %{HTTPS} !=on
RewriteCond %{HTTP_HOST} ^www\.(.+)$ [NC]
RewriteRule ^ http://%1%{REQUEST_URI} [R=301,L]
</IfModule> 
<IfModule mod_autoindex> 
Options -Indexes
</IfModule> 
<IfModule mod_rewrite> 
RewriteCond %{SCRIPT_FILENAME} -d [OR]
RewriteCond %{SCRIPT_FILENAME} -f
RewriteRule "(^|/)\." - [F]
</IfModule> 
<FilesMatch "(^#.*#|\.(bak|config|dist|fla|inc|ini|log|psd|sh|sql|sw[op])|~)$">
Order allow,deny
Deny from all
Satisfy All
</FilesMatch>
FileETag None
SSLEngine on
SSLCertificateKeyFile /etc/ssl/ssl.key/server.key
SSLCertificateFile /etc/ssl/ssl.crt/api_my_site_com.crt
SSLCertificateChainFile /etc/ssl/ssl.crt/apimysite.com-bundle
ErrorLog /var/www/mysite/api.log
LogLevel info
</VirtualHost>