Python HTTPS requests (urllib2) to some sites fail on Ubuntu 12.04 without proxy

This appears to be related to the addition of TLS 1.1 and 1.2 support to the version of OpenSSL found in 12.04. The connection failure can be reproduced with the OpenSSL command line tool:

$ openssl s_client -connect www.mediafire.com:443
CONNECTED(00000003)
140491065808544:error:140790E5:SSL routines:SSL23_WRITE:ssl handshake failure:s23_lib.c:177:
---
no peer certificate available
---
No client certificate CA names sent
---
SSL handshake has read 0 bytes and written 320 bytes
---
New, (NONE), Cipher is (NONE)
Secure Renegotiation IS NOT supported
Compression: NONE
Expansion: NONE
---

The connection succeeds if I force the connection to use TLS 1.0 with the -tls1 command line argument.

I would suggest you file a bug report about this problem here:

https://bugs.launchpad.net/ubuntu/+filebug

For python novices like me, here is the way to override httplib the easiest way. At the top of your python script, include these lines:

import httplib
from httplib import HTTPConnection, HTTPS_PORT
import ssl

class HTTPSConnection(HTTPConnection):
    "This class allows communication via SSL."
    default_port = HTTPS_PORT

    def __init__(self, host, port=None, key_file=None, cert_file=None,
            strict=None, timeout=socket._GLOBAL_DEFAULT_TIMEOUT,
            source_address=None):
        HTTPConnection.__init__(self, host, port, strict, timeout,
                source_address)
        self.key_file = key_file
        self.cert_file = cert_file

    def connect(self):
        "Connect to a host on a given (SSL) port."
        sock = socket.create_connection((self.host, self.port),
                self.timeout, self.source_address)
        if self._tunnel_host:
            self.sock = sock
            self._tunnel()
        # this is the only line we modified from the httplib.py file
        # we added the ssl_version variable
        self.sock = ssl.wrap_socket(sock, self.key_file, self.cert_file, ssl_version=ssl.PROTOCOL_TLSv1)

#now we override the one in httplib
httplib.HTTPSConnection = HTTPSConnection
# ssl_version corrections are done

From here on, you can use urllib or whatever you use just like you normally would.

Note: This is for python 2.7. For a python 3.x solution, you need to override the HTTPSConnection class found in http.client. I leave that as an exercise for the reader. :-)

You can avoid modifying the httplib.py file by modifying your HTTPSConnection object:

import httplib, ssl, socket

conn = httplib.HTTPSConnection(URL.hostname)
sock = socket.create_connection((conn.host, conn.port), conn.timeout, conn.source_address)
conn.sock = ssl.wrap_socket(sock, conn.key_file, conn.cert_file, ssl_version=ssl.PROTOCOL_TLSv1)
conn.request('POST', URL.path + URL.query)

The request method creates a new socket only if connection.sock is not defined. Creating your own one adding the ssl_version parameter will make the request method use it. Then everything else works as usual.

I was having the same issue and this works for me.

Regards

The problem is in ssl, it has nothing to do with HTTP, so why patching httplib if you can patch ssl. The following code should fix all SSL sockets including, but not limited to HTTPS, for Python 2.6+ (built in ssl, did not try with pyopenssl).

import functools
import ssl

old_init = ssl.SSLSocket.__init__

@functools.wraps(old_init)
def ubuntu_openssl_bug_965371(self, *args, **kwargs):
  kwargs['ssl_version'] = ssl.PROTOCOL_TLSv1
  old_init(self, *args, **kwargs)

ssl.SSLSocket.__init__ = ubuntu_openssl_bug_965371

EDIT httplib.py (/usr/lib/pythonX.X/httplib.py on Linux)

FIND HTTPSConnection class declaration

  class HTTPSConnection(HTTPConnection):
....

Inside class code CHANGE line

self.sock = ssl.wrap_socket(sock, self.key_file, self.cert_file)

TO

self.sock = ssl.wrap_socket(sock, self.key_file, self.cert_file, ssl_version=ssl.PROTOCOL_TLSv1)

Then httplib HTTPS request should work

import httplib
from urlparse import urlparse
url = XXX
URL = urlparse(url)
connection = httplib.HTTPSConnection(URL.hostname)
connection.request('POST', URL.path + URL.query)
response = connection.getresponse()

This issue is likely due to SSLv2 being disabled on the web server, but Python 2.x tries to establish a connection with PROTOCOL_SSLv23 by default.

Here's the link to my answer for a similar issue on Stack Overflow - https://stackoverflow.com/a/24166498/41957

Update: this is functionally the same as @temoto's answer above.

A simple fix that worked for me was to override SSL's default protocol:

import ssl
ssl.PROTOCOL_SSLv23 = ssl.PROTOCOL_TLSv1