Blog » URL Checking and SSL Versions

Code Monkey by melodymcfadden on DeviantArt

URL Checking and SSL Versions

A recent project required that we check if links in content were not only valid but also existed on the remote server. Since we already had a RegEx to handle the validity check I figured, how hard could it be? Well let me tell you, it wasn’t as easy as you’d think.

I mean, sure, just fire up HttpWebRequest, set the method to HEAD and then get the response. Well, I quickly learned a few things about this.

First is that you need to be VERY careful about closing your responses if the method is being called frequently. Of course one always wants to do that but I’ve never seen a server spiral out of control so quickly when we missed one page. Resolved this by wrapping in a using.. block.

Next, I found out that many servers return a 404 or 402 error when making a HEAD request. This coupled with discovering that HttpWebResponse throws an exception rather than just returning the error code in the .StatusCode parameter makes responding to this a bit more complicated. Problem solved by adding a GET request inside the CATCH statement when we get an exception on GetResponse().

Well, this all worked fine until recently when a site with a secure (https://) URL was closing the connection immediately rather than returning an error. How rude! I mean, this was the equivalent flipping us the bird and slamming the door in our face. Url works fine in a browser and it responded quite nicely when making an insecure request but that wasn’t an answer to the problem.

Many hours of Google-Fu later lead me to the ServicePointManager.SecurityProtocol parameter. It turned out that the server was ONLY accepting TLSv1.1 or higher connections and the .net HttpWebRequest was requesting no higher than TLSv1 by default.

So, in order to cover all bases, we now use the following for this whole method:

Snippet

public static bool RemoteFileExists(string url)
        {
            if (common.IsValidURL(url))
            {
                //Creating the HttpWebRequest
                HttpWebRequest request = WebRequest.Create(url) as HttpWebRequest;
                //Setting the Request method HEAD, you can also use GET too. Trying HEAD first to save time and space.
                request.Method = "HEAD";                
                if (url.StartsWith("https://"))
                {
                    ServicePointManager.SecurityProtocol = SecurityProtocolType.Tls12 | SecurityProtocolType.Tls11 | SecurityProtocolType.Tls | SecurityProtocolType.Ssl3;
                }
                try
                {
                    //Getting the Web Response.
                    using (HttpWebResponse response = request.GetResponse() as HttpWebResponse)
                    {
                        //Returns TRUE if the Status code == 200
                        if (response.StatusCode == HttpStatusCode.OK)
                            return true;
                    }
 
                }
                catch (WebException wex)
                {
                    using (HttpWebResponse exresp = wex.Response as HttpWebResponse)
                    {
                        try
                        {
                            HttpWebRequest request2 = WebRequest.Create(url) as HttpWebRequest;
                            request2.Method = "GET";
                            using (HttpWebResponse response2 = request2.GetResponse() as HttpWebResponse)
                            {
                                if (response2.StatusCode == HttpStatusCode.OK)
                                    return true;
                            }
                        }
                        catch (WebException ex)
                        {
                            return false;
                        }
                    }                    
                }
            }
 
            return false;
        }

Hopefully my hours of searching will help someone else resolve their problem when trying to connect to a remote server using SSL and the HttpWebRequest/HttpWebResponse objects (works similarly with WebClient) in Asp.Net.

Note: TLS1.2 is only available in .Net 4.5 I believe.

Code Monkey image courtesy of MelodyMcFadden on DeviantArt. Shared via CC by-sa 3.0.

Michael Gibbs
Stalk Me...
Latest posts by Michael Gibbs (see all)