When you build your website, you should have not only your readers, but also the “web spiders” in mind. You need to create a website that flows and is easy to navigate for both your readers and the search engine crawlers.

Here is an important article from SubmitEaze about planning your website’s page structure.

Ideally you should think about the structure of your website before you start coding your web pages, as once you’ve implemented one method it is tricky to change things around at a later date.
I recommend that when you design your structure you try to keep the structure as flat as possible. This does not mean that you should put everything in your root directory for best results, but just make sure the structure goes no deeper than two or three levels deep. Below is an example of a flat directory structure with two levels: –

Flat Structure.

This would mean the URL for page three on theme three would be:

http://www.mydomain.com/theme3/page3.html

You should not however design your structure like this: –

Vertical Structure

The URL to get to theme 3, page 3 will now look like this: –

http://www.mydomain.com/Theme1/Theme2/Theme3/page3.html

When you organize your structure you should try to maintain similarly themed pages in the same directory.

The main reasons for keeping a flat directory structure is because search engines might not deep-crawl your site. That is, when a search engine visits your website it might be set to crawl only to a certain depth of your website. So say for example you have a new website, the search engine might only crawl to a depth of 2, so in the case of the vertical page structure above only Page1, Page2, and Page3 of Theme 1 will be indexed by the search engine. The other pages might as well not exist! Whereas if you’d have the flat directory structure then all of the pages on your site would have been crawled.

Sitemaps

For more on this, you can read our older article entitled, “Why are Sitemaps Important?”

When you’ve finished your website, it is a very good idea to create a sitemap. A sitemap provides a link to all of the different pages on your website from one single page. This means that the search engine (or a person) can jump to any other page on your website from this one single page.

If you link to the sitemap from every page on your website then it has the effect of interlinking all of your pages with each other. This means that the search engine will attempt to crawl every page on your website.

So now you’re thinking, ‘well if I make a sitemap then I don’t need to be concerned about my page structure’ Wrong! If you remember, search engines often only crawl to a certain depth, so when it finds a link on your sitemap to
http://www.mydomain.com/Theme1/Theme2/Theme3/page3.html
it will disregard it as the depth of the web page is greater than its maximum crawl depth.

There are different types of sitemaps: the Conventional ones, which are in either .txt or HTML format; and the Special one, which is in XML form. The difference between the two is that conventional sitemaps are for both human and spider use while the latter is specifically for the Googlebot spiders. This is why the XML Sitemaps are called Google Sitemaps. Yahoo and MSN still keep the HTML format. Both the Conventional and Special forms can be used on your website and it will not be penalized by Google for the duplication. XML Sitemaps follow the Google Sitemap Protocols.

How do Search Engine robots use sitemaps? They will follow every single link on your sitemap that leads to the pages of your site and thus, the pages will get crawled and then indexed. SEO experts believe that best place to put the sitemap link for the conventional sitemap is on the homepage at the top right side of the page.

XML Sitemaps are made especially for Google. The conventional ones do not give Googlebot spiders any restriction on what pages they can crawl. There are some parts of the website that cannot easily be reached by the Google crawlers. Examples of these are the dynamic elements behind a search form. But Google sitemaps provide web developers with a way to direct Google to the pages on their site that they want to be crawled and give details on what order these pages should be crawled, how important they are in comparison with one another and how often they are updated or revised.

You can also use Sitemap Generators to create sitemaps for you. Sitemap Generators crawl the URL you provide and generate sitemap details and the number of pages in that website. You can just download the generated file and save it as “sitemap.xml”. You can try the Sitemap Generator at http://www.sitemapz.com/

Once the sitemap is created, the next step is to submit it to Google.

Using a sitemap might not bring results too easily but keep in mind that search engines dynamically improve their sitemap indexing algorithms and in time, they will surely index websites faster through sitemaps.