During the OpenForce Conference, I heard from several designers that the DNNMenu had problems with Search Engine Optimization (SEO) because it rendered menu items that used JavaScript for navigation.  While I assured them that this was not the case and that the menu rendered a down-level version for search bots, I thought I would perform further testing and document the exact behavior.  In the process, I found that while the DNNMenu performed as expected, ASP.Net did not.

To observe the behavior of the menu, I used Firefox 3 with the User Agent Switcher plug-in.  This plug-in allows you to simulate any user agent string you wish.  For testing purposes you can find a complete list of user agent strings at useragentstring.com.

Below is the DNNMenu as rendered in the standard Firefox browser.

MozillaMenu

To change user agents, create a set of new user-agent using the Tools>>User Agent Switcher>>Options>>Options screen and add the string for the browser or web crawler you wish to emulate.  If you are emulating a web crawler, you should also disable JavaScript.  With the GoogleBot user-agent enabled and JavaScript disabled, Firefox will render the menus as seen by Google.  You can see this behavior in the picture below.

GoogleMenu

 

Opening Firebug, you can easily verify that the menu is rendered as a series of hyperlinks as shown below.

GoogleMenuRendering

Everything appeared to work as I expected.  Unfortunately, when I tested the user-agents for Yahoo and ask.com, I found that the menu did not behave the same as for the Google and MSN/Live bots. 

The DNNMenu IsDownLevel property determines whether to render the full JavaScript menu or to render a down-level version that doesn’t require JavaScript.  If the “browser” is a web crawler, then the menu should always render as down-level.  Looking at the IsDownLevel code below we see that the key is the IsCrawler property which further delegates the crawler check to the ASP.Net framework. 

Public ReadOnly Property IsDownLevel() As Boolean
    Get
        If ForceDownLevel OrElse Me.IsCrawler OrElse _
         DotNetNuke.UI.Utilities.ClientAPI.BrowserSupportsFunctionality(Utilities.ClientAPI.ClientFunctionality.DHTML) = False OrElse _
         DotNetNuke.UI.Utilities.ClientAPI.BrowserSupportsFunctionality(Utilities.ClientAPI.ClientFunctionality.XML) = False _
        Then
            Return True
        Else
            Return False
        End If
    End Get
End Property
 
Public Property IsCrawler() As Boolean
    Get
        If Len(ViewState("IsCrawler")) = 0 Then
            Return System.Web.HttpContext.Current.Request.Browser.Crawler
        Else
            Return CBool(ViewState("IsCrawler"))
        End If
 
    End Get
    Set(ByVal Value As Boolean)
        ViewState("IsCrawler") = Value
    End Set
End Property

Looking at this code, I see that for some reason ASP.Net is not properly detecting any bots other than GoogleBot and MSN.  After doing further research I found that browser detection is controlled by the .Net 2.0 .browser files that are located at %FrameworkDir%\v2.0.50727\CONFIG\Browsers.  A quick WinGrep determined that the only browser file that sets the Crawler property is the gateway.browser file.  The relevant portion of the file is:

<gateway id="Crawler" parentID="Default">
    <identification>
        <userAgent match="crawler|Crawler|Googlebot|msnbot" />
    </identification>
 
    <capture>
    </capture>
 
    <capabilities>
        <capability name="crawler" value="true" />
    </capabilities>
</gateway>

Clearly, this is only detecting Google and MSN.  Further research indicated that this is a known problem with the default ASP.net behavior.  ASP.Net 2.0 added the ability to define .browser files in the App_Browsers folder of your web application.  By defining custom .browser files we should be able to correctly identify targeted search bots.

The below slurp.browser file will detect the Yahoo crawler.

<browsers>
    <browser id="Slurp" parentID="Mozilla">
        <identification>
            <userAgent match="Slurp" />
        </identification>
        <capabilities>
            <capability name="browser" value="Yahoo!Slurp" />
            <capability name="crawler" value="true" />
            <capability name="cookies" value="false" />
            <capability name="css1" value="true" />
            <capability name="css2" value="true" />
            <capability name="javascript" value="false" />
            <capability name="tables" value="true" />
            <capability name="w3cdomversion" value="1.0" />
            <capability name="xml" value="true" />
            <capability name="tagwriter" value="System.Web.UI.HtmlTextWriter" />
        </capabilities>
    </browser>
</browsers>

Most of the crawlers use a Mozilla compatible string like Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp) and so you need to make sure to set the parentID to “Mozilla” so that the browser detection works correctly.

As you can see, DotNetNuke has good support for search engines in the default DNNMenu.  There are still optimizations you must make if you wish to cater to search engines beyond Google and Windows Live/MSN.  One of the great aspects of DotNetNuke is that if you don’t care for the tradeoffs that the DNNMenu makes, then you can choose to use a third-party CSS menu like the HouseMenu or Snapsis CSS NavMenu

All of this works just fine in DotNetNuke 4.4 and beyond, however, in DotNetNuke 5.0 SEO will get a slight boost.  The down-level rendering in previous versions of the DNNMenu was not very lightweight nor was did it use semantic markup.  The next version of DNNMenu in DotNetNuke 5.0 allows the menu to be rendered as an unordered list which should help search engines to better understand the menu structure.  I will have a separate post to discuss this version of the menu.

EDIT:  Updated DotNetNuke 4.9.1 and DotNetNuke 5.0.0 to include the Crawler and Webkit browser files from http://owenbrady.net/