URL Checking and SSL Versions

URL Checking and SSL Versions

URL Checking and SSL Versions

A recent project required that we check if links in content were not only valid but also existed on the remote server. Since we already had a RegEx to handle the validity check I figured, how hard could it be? Well let me tell you, it wasn’t as easy as you’d think.

I mean, sure, just fire up HttpWebRequest, set the method to HEAD and then get the response. Well, I quickly learned a few things about this.

First is that you need to be VERY careful about closing your responses if the method is being called frequently. Of course one always wants to do that but I’ve never seen a server spiral out of control so quickly when we missed one page. Resolved this by wrapping in a using.. block.

Next, I found out that many servers return a 404 or 402 error when making a HEAD request. This coupled with discovering that HttpWebResponse throws an exception rather than just returning the error code in the .StatusCode parameter makes responding to this a bit more complicated. Problem solved by adding a GET request inside the CATCH statement when we get an exception on GetResponse().

Well, this all worked fine until recently when a site with a secure (https://) URL was closing the connection immediately rather than returning an error. How rude! I mean, this was the equivalent flipping us the bird and slamming the door in our face. Url works fine in a browser and it responded quite nicely when making an insecure request but that wasn’t an answer to the problem.

Many hours of Google-Fu later lead me to the ServicePointManager.SecurityProtocol parameter. It turned out that the server was ONLY accepting TLSv1.1 or higher connections and the .net HttpWebRequest was requesting no higher than TLSv1 by default.

So, in order to cover all bases, we now use the following for this whole method:


Hopefully my hours of searching will help someone else resolve their problem when trying to connect to a remote server using SSL and the HttpWebRequest/HttpWebResponse objects (works similarly with WebClient) in Asp.Net.

Note: TLS1.2 is only available in .Net 4.5 I believe.

Code Monkey image courtesy of MelodyMcFadden on DeviantArt. Shared via CC by-sa 3.0.

Stalk Me...

Michael Gibbs

I'm a technologist at heart, assisting companies best utilize the Internet to market themselves and sell their products and services online. This includes organizations as large as Google and as small as one or two individuals.
At Over The Top and Exhibit A Communications, I've programmed solutions for Google as well as at least one other company that was later acquired by Google.

I've been CTO of an Internet SaaS company and spent my time pretty evenly between guiding the future technical strategy of the company, architecting software solutions for my dev teams, designing and running a data center to service our clients world-wide as well as being a technical evangalist/sales engineer to our media clients large and small.

I've also been Chief Photographer of the Daily Sun/Post newspaper back when it was a 5-day a week daily newspaper.

I also spent a great many years as a beach lifeguardfor the City of San Clemente as well as Jr. Lifeguard instructor and then as it's program coordinator.
Bragging rights
Private Pilot with Instrument rating and proud husband of soon to be 25 years (and counting).
Stalk Me...

Latest posts by Michael Gibbs (see all)

Leave a Reply