Getting the Correct HTTP Status Codes out of ASP.NET Custom Error Pages

If you are using the default ASP.NET custom error pages, chances are your site is returning the incorrect HTTP status codes for the errors that your users are experiencing (hopefully as few as possible!). Sure, your users see a pretty error page just fine, but your users aren’t always flesh and blood. Search engine crawlers are also your users (in a sense), and they don’t care about the pretty pictures and funny one-liners on your error pages; they care about the HTTP status codes returned. For example, if a request for a page that was removed consistently returns a 404 status code, a search engine will remove it from its index. However, if it doesn’t and instead returns the wrong error code, the search engine may leave the page in its index.

This is what happens if your non-existent pages don't return the correct status code!

Unfortunately, ASP.NET custom error pages don’t return the correct error codes. Here’s your typical ASP.NET custom error page configuration that goes into the Web.config:

<customErrors mode="On" defaultRedirect="~/Pages/Error">
    <error statusCode="403" redirect="~/Pages/Error403" />
    <error statusCode="404" redirect="~/Pages/Error404" />
</customErrors>

And here’s a Fiddler trace of what happens when someone request a page that should simply return 404:

A trace of a request that should have just 404'd

As you can see, the request for /ThisPageDoesNotExist returned 302 and redirected the browser to the error page specified in the config (/Pages/Error404), but that page returned 200, which is the code for a successful request! Our human users wouldn’t notice a thing, as they’d see the error page displayed in their browser, but any search engine crawler would think that the page existed just fine because of the 200 status code! And this problem also occurs for other status codes, like 500 (Internal Server Error). Clearly, we have an issue here.

The first thing we need to do is stop the error pages from returning 200 and instead return the correct HTTP status code. This is easy to do. In the case of the 404 Not Found page, we can simply add this line in the view:

<% Response.StatusCode = (int)HttpStatusCode.NotFound; %>

We will need to do this to all views that handle errors (obviously changing the status code to the one appropriate for that particular error). This not only includes the pages you’ve configured in the customErrors element in the Web.config, but also any views you are using with ASP.NET MVC HandleError attributes (if you’re using ASP.NET MVC). After making these changes, our Fiddler trace looks like this:

A trace of a request that is 404ing, but still redirecting

We’ve now got the correct status code being returned, but there’s still an unnecessary redirect going on here. Why redirect when we can just return 404 and the error page HTML straight up? If we are using vanilla ASP.NET Forms, this is super easy to do with a quick configuration change; just set redirectMode to ResponseRewrite in the Web.config (this setting is new since .NET 3.5 SP1):

<customErrors mode="On" defaultRedirect="~/Error.aspx" redirectMode="ResponseRewrite">
    <error statusCode="403" redirect="~/Error403.aspx" />
    <error statusCode="404" redirect="~/Error404.aspx" />
</customErrors>

Unfortunately, this trick doesn’t work with ASP.NET MVC, as the method by which the response is rewritten using the specified error page doesn’t play nicely with MVC routes. One workaround is to use static HTML pages for your error pages; this sidesteps MVC routing, but also means you can’t use your master pages, making it a pretty crap solution. The easiest workaround I’ve found is to defenestrate ASP.NET custom errors and handle the errors manually through a bit of trickery in the Global.asax.

Firstly, we’ll handle the Error event in our Global.asax HttpApplication-derived class:

protected void Application_Error(object sender, EventArgs e)
{
    if (Context.IsCustomErrorEnabled)
        ShowCustomErrorPage(Server.GetLastError());
}

private void ShowCustomErrorPage(Exception exception)
{
    HttpException httpException = exception as HttpException;
    if (httpException == null)
        httpException = new HttpException(500, "Internal Server Error", exception);

    Response.Clear();
    RouteData routeData = new RouteData();
    routeData.Values.Add("controller", "Error");
    routeData.Values.Add("fromAppErrorEvent", true);

    switch (httpException.GetHttpCode())
    {
        case 403:
            routeData.Values.Add("action", "AccessDenied");
            break;

        case 404:
            routeData.Values.Add("action", "NotFound");
            break;

        case 500:
            routeData.Values.Add("action", "ServerError");
            break;

        default:
            routeData.Values.Add("action", "OtherHttpStatusCode");
            routeData.Values.Add("httpStatusCode", httpException.GetHttpCode());
            break;
    }

    Server.ClearError();

    IController controller = new ErrorController();
    controller.Execute(new RequestContext(new HttpContextWrapper(Context), routeData));
}

In Application_Error, we’re checking the setting in Web.config to see whether custom errors have been turned on or not. If they have been, we call ShowCustomErrorPage and pass in the exception. In ShowCustomErrorPage, we convert any non-HttpException into a 500-coded error (Internal Server Error). We then clear any existing response and set up some route values that we’ll be using to call into MVC controller-land later. Depending on which HTTP status code we’re dealing with (pulled from the HttpException), we target a different action method, and in the case that we’re dealing with an unexpected status code, we also pass across the status code as a route value (so that view can set the correct HTTP status code to use, etc). We then clear the error and new up our MVC controller, then execute it.

Now we’ll define that ErrorController that we used in the Global.asax.

public class ErrorController : Controller
{
    [PreventDirectAccess]
    public ActionResult ServerError()
    {
        return View("Error");
    }

    [PreventDirectAccess]
    public ActionResult AccessDenied()
    {
        return View("Error403");
    }

    public ActionResult NotFound()
    {
        return View("Error404");
    }

    [PreventDirectAccess]
    public ActionResult OtherHttpStatusCode(int httpStatusCode)
    {
        return View("GenericHttpError", httpStatusCode);
    }

    private class PreventDirectAccessAttribute : FilterAttribute, IAuthorizationFilter
    {
        public void OnAuthorization(AuthorizationContext filterContext)
        {
            object value = filterContext.RouteData.Values["fromAppErrorEvent"];
            if (!(value is bool && (bool)value))
                filterContext.Result = new ViewResult { ViewName = "Error404" };
        }
    }

This controller is pretty simple except for the PreventDirectAccessAttribute that we’re using there. This attribute is an IAuthorizationFilter which basically forces the use of the Error404 view if any of the action methods was called through a normal request (ie. if someone tried to browse to /Error/ServerError). This effectively hides the existence of the ErrorController. The attribute knows that the action method is being called through the error event in the Global.asax by looking for the “fromAppErrorEvent” route value that we explicitly set in ShowCustomErrorPage. This route value is not set by the normal routing rules and therefore is missing from a normal page request (ie. requests to /Error/ServerError etc).

In the Web.config, we can now delete most of the customErrors element; the only thing we keep is the mode switch, which still works thanks to the if condition we put in Application_Error.

<customErrors mode="On">
    <!-- There is custom handling of errors in Global.asax -->
</customErrors>

If we now look at the Fiddler trace, we see the problem has been solved; the redirect is gone and the page correctly returns a 404 status code.

A trace showing the correct 404 behaviour

In conclusion, we’ve looked at a way to solve ASP.NET custom error pages returning incorrect HTTP status codes to the user. For ASP.NET Forms users the solution was easy, but for ASP.NET MVC users some extra manual work needed to be done. These fixes ensure that search engines that trawl your website don’t treat any error pages they encounter (such as a 404 Page Not Found error page) as actual pages.