Post

5 followers Follow
0
Avatar

How To: Configure certificate trust, hostname verification and other SSL options for HttpRequest

Version 4.0.10 of XL Release introduced HttpRequest, a new underlying base class for HTTP requests which is used in webhooks and is available in custom tasks. HttpRequest is based on Apache HTTP Components (Apache HC). This offers more configuration options around HTTP than Jython's implementation of httplib, which was used previously (usually via the deprecated XLRequest base class).

Apache HC's default SSL configuration aims for security, failing to accept things like hostname mismatches between a target host and its certificate (some of the default settings cause slightly different behaviour when compared to the old XLRequest class, which we're looking into).

Here are some examples of how to configure the Apache HC client in HttpRequest with different SSL options. These are based on a fresh 4.0.10 installation of XL Release for which I have enabled SSL and generated a self-signed certificate for "localhost".

Extract HttpRequest to the extensions directory

First, create a directory SERVER_HOME/ext/pythonutil if it does not already exist. Then, extract the HttpRequest.py class (in SERVER_HOME/lib/xl-release-server-<version>.jar) into that directory. An easy way to do this is to copy the JAR into a temporary directory, open it using a ZIP file browser (change the file extension to .zip if necessary), and extract the file from the pythonutil folder. Copy the extracted file into SERVER_HOME/ext/pythonutil, where it will override the provided HttpRequest.py class in the JAR file.

Set up a sandbox template with a test task

I created a simple "sandbox" release template with a single script task to test the various settings:

test-script-task.png

This fails on two counts, as expected: first, it's a self-signed certificate and, secondly, the certificate is for "localhost", not "nb-aphillips" (my laptop's hostname):

ssl-handshake-error.png

Edited on 18th Nov 2015 to add: if you are running TLS 1.2 in your environment and you see javax.net.ssl.SSLHandshakeException: Remote host closed connection during handshake errors after making the changes in this post, see Michiel Sens' comment further down for a solution. Thanks, Michiel!

Configure HttpRequest to trust self-signed certificates

As a first step, we'll configure HttpRequest to trust the self-signed certificate. This boils down to changing HttpRequest to not use Apache HC's default HttpClient object, but one that is configured with the required SSL options. Here, we have commented-out the original client = HttpClients.createDefault() call and added a createHttpClient method that returns a client configured to trust all certificates:

from org.apache.http.conn.ssl import SSLContextBuilder, SSLConnectionSocketFactory, TrustSelfSignedStrategy  
    ...  
    def executeRequest(self, request):  
        client = None  
        response = None  
        try:  
            # replace the original default client with a call to our new method  
            #client = HttpClients.createDefault()  
            client = self.createHttpClient()  
            ...  
    # we're adding this method to the class  
    def createHttpClient(self):  
        # see the Javadoc for SSLConnectionSocketFactory for more options  
        builder = SSLContextBuilder()  
        builder.loadTrustMaterial(None, TrustSelfSignedStrategy())  
        socketfactory = SSLConnectionSocketFactory(builder.build())  
        # print 'DEBUG: Created custom HttpClient to trust all certs\n'  
        return HttpClients.custom().setSSLSocketFactory(socketfactory).build()

Note that we've also added an additional import statement for the required classes to the initial section of HttpRequest.py.

With this change, we're now down to the hostname mismatch:

handshake-error.png

Configure HttpRequest to ignore hostname mismatches

Configuring the HTTP client to also ignore hostname mismatches is simply a matter of adding an additional configuration option in the new createHttpClient method:

def createHttpClient(self):  
        builder = SSLContextBuilder()  
        builder.loadTrustMaterial(None, TrustSelfSignedStrategy())  
        # now also adding ALLOW_ALL_HOSTNAME_VERIFIER here  
        socketfactory = SSLConnectionSocketFactory(builder.build(), SSLConnectionSocketFactory.ALLOW_ALL_HOSTNAME_VERIFIER)  
        # print 'DEBUG: Created custom HttpClient to trust all certs and allow hostname mismatches\n'  
        return HttpClients.custom().setSSLSocketFactory(socketfactory).build()

You can, of course, also configure the client to only ignore hostnames (but not trust self-signed certs):

def createHttpClient(self):  
        socketfactory = SSLConnectionSocketFactory(SSLContextBuilder().build(), SSLConnectionSocketFactory.ALLOW_ALL_HOSTNAME_VERIFIER)  
        # print 'DEBUG: Created custom HttpClient to allow hostname mismatches\n'  
        return HttpClients.custom().setSSLSocketFactory(socketfactory).build()

In this case, you can also remove TrustSelfSignedStrategy from the from org.apache.http.conn.ssl import ... line, since we don't use it here.

With this change, our test task can call the server successfully:

https-call-200.png

Code samples are based on this useful Stack Overflow post. Thanks, mavroprovato!

PS: In the light of the POODLE SSL vulnerability, another SSL configuration option you may wish to consider is to allow only TLS connections. See the Javadoc for SSLConnectionSocketFactory for details on how to construct the socket factory to achieve this.

And as a final formality, please bear in mind that the code here and in all other forum posts is sample code only and is not officially supported by XebiaLabs. Having said that, we of course will help wherever we can!

Andrew Phillips

Please sign in to leave a comment.

10 comments

0
Avatar

Hi Andrew ... we are running into issues using deprecated XLRequest object (httblib) as well and were unable to move over to HttpRequest object due to certificate issues at Ops. Chicken-egg. Due to your article there opens up a possibility to move over to HttpRequest while certificate issues are not fixed yet. Looks promising. (note: in the meantime pushing ops to fix this certificate issue at its core). Thanks. Michiel.

Michiel Sens 0 votes
0
Avatar

Hoi Michiel

Due to your article there opens up a possibility to move over to HttpRequest while certificate issues are not fixed yet. Looks promising.

I hope this ends up working for you - if you get stuck, please comment here or create a new forum post to see if we can help!

Regards

Andrew

Andrew Phillips 0 votes
0
Avatar

Hi Andrew, using HttpRequest solves a lot of issues for me. Updated parts of xlrcalisa (nolio) plugins (+ some company internal plugins) so they now use this object. Also updated HttpRequest as per your blog to relax some security restrictions as ops still does not have certificates in shape. As soon as they fix this, I'll put it back to what is was. I Will send pull request to community xlrcalisa git soon. Newer versions of this plugin will then use HttpRequest.

Michiel Sens 0 votes
0
Avatar

I Will send pull request to community xlrcalisa git soon.

Great, thanks Michiel! Glad to hear this seems to be making things easier for you...

Regards

Andrew

Andrew Phillips 0 votes
0
Avatar

Hello Andrew. This worked perfectly for me upto the point that ops forced everybody to use tlsv1.2, which presented me with following error message in my particular plugin:

[18-11-2015 13:46] Sens, M.A.C. (Michiel): 
2015-11-18 13:45:55.914 [pool-1-thread-8] {task=Applications/Release7350862/Phase8734751/Task3148125} WARN  c.x.xlrelease.script.ScriptService - Exception during execution
javax.script.ScriptException: javax.net.ssl.SSLHandshakeException: javax.net.ssl.SSLHandshakeException: Remote host closed connection during handshake in <script> at line number 42

Fixed this by adding variable stating tls version and using different constuctor for SSLConnectionSocketFactory. in CreateHttpClient(self), basically use:

#socketfactory = SSLConnectionSocketFactory(builder.build(),SSLConnectionSocketFactory.ALLOW_ALL_HOSTNAME_VERIFIER)
tlsVersions = ["TLSv1", "TLSv1.1", "TLSv1.2"]
socketfactory = SSLConnectionSocketFactory(builder.build(), tlsVersions, None, SSLConnectionSocketFactory.ALLOW_ALL_HOSTNAME_VERIFIER)

Cheers, Michiel

Michiel Sens 0 votes
0
Avatar

Fixed this by adding variable stating tls version and using different constuctor for SSLConnectionSocketFactory

Thanks for the fix, Michiel! I've edited the post to add a link to your comment for other users that run into the same issue.

Regards

Andrew

Andrew Phillips 0 votes
0
Avatar

Hi Andrew,

I came across this very useful post meanwhile building my XLD plugin. Initially I used requests module for executing my calls, however stumbled upon the problem of self signed certificates. Obviously a verify=False worked in python on my local machine meanwhile testing my scripts, then I was stuck on XLD with SSLHandshakeException. As I understood in Jython execution context, Java libraries are used for SSL verification and requests settings do not influence them.

Then I came across this post and I rewrote my calls using your example. Still I'm getting the following exception:

javax.net.ssl.SSLHandshakeException: javax.net.ssl.SSLHandshakeException: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target

Judging by error I would say that the certificate that is validated is not present in the key store.

Now, is there a way to ignore certificates completely and accept everything?

I found a post on StackOverflow that may help http://stackoverflow.com/questions/24720013/apache-http-client-ssl-certificate-error but I'm not sure on how to translate it into python (I'm sorry I'm new to it).

Do you know of any way of completely ignoring certificates?

Mario Majcica 0 votes
0
Avatar

After a bit of struggle I defined the following:


class TrustAllStrategy(TrustStrategy):
   def isTrusted(self, chain, authType):
       return True

and then used this strategy insetad of TrustSelfSignedStrategy.

It actually worked. Now certificates verification is disabled.

 

Mario Majcica 0 votes
0
Avatar

Mario, that sounds like the proper approach. Good that you got it to work!

Kind regards,

Hes Siemelink 0 votes