Consider this scenario:
You stand MOSS up as a Web Content Management (WCM) site for your company’s public internet site. You site is a hierarchy of sites to create the structure your information architect designed. You create pages within these sites – at least 1 for each site. But in certain site – you have more than one page.
Enter the MOSS Indexer
When the MOSS Indexer comes along to index the sites it will only index the page within each site which is flagged as the “default” page for that site. Now if this is a site which has more than one page – they are usually stored in the /pages library along with the default page. When the indexer his this site – it still only indexes the default page and leaves all other pages in that library alone.
Reason
MOSS Publishing Sites (not the collaboration portal) are enabled in “Lockdown Mode” which means that there are a number of crawler rules that do not index pages like newitem.aspx or allitems.aspx. This is prudent as you don’t want to show these types of /Forms pages to anonymous users.
Resolution
You have to add an INCLUDE rule to your crawler rules to include any and all content within the /pages directory. This rule looks like this:
*://*/pages/
Now start a fresh crawl and enjoy all your new content!
BTW: if you want to enable “lockdownmode” on other types of sites, check out this post from Sezai
