Search engines use algorithms to determine where a web page ranks for a particular keyword used for a search query. A search query is the search term you might type in to obtain information from a search engine like Google.
Algorithms are complex programs that determine how a search engine indexes content fetched by spiders and how it displays results to users. The search engine strives to provide quality search results that ensures its users return.
Two main factors search engines use to determine ranking is the content on a web page and “off- page-factors” (such as the text used along with links that point to a web page). The latter is become increasingly important and neglected by most companies.
However, we recently start to wonder if links or more importantly backlinks provide large companies and those who can afford to buy their way to the top of search engine results pages an unfair advantage over small and poorly resourced businesses that often provide consumers with excellent information, products and services.
The factors such as content ‘scored’ on a web page vary with differing weights given by different search engines. However, some things such as the positioning of keywords and frequency we consider universally important. For instance, keywords or phrases that occur in headings, page titles and other key places on the web page are more significant than keywords that appear in a more random fashion in the middle of a page.
When considering “offsite” factors, the number of quality pages linking to a web page is significant to a web page’s ranking. Good websites have many other sites that link to them. More recently, the advent of social networking and forums adds a new and very important dynamic – consumer buzz – and consumer preference and referral. These types of links are of critical importance.
At the same time, the relevance of linking is important too and determines “link relevance” and weighting. In short, this means that a link to your page from a similar website or a page having the same keywords as the page linked to is more important than a link to your web page from totally unrelated pages or pages with different content. The text in the link itself is also important, known as “anchor text” or “link reputation”. We will cover link reputation later.
Link reputation is widely accepted as a main factor for Google’s PageRank system. Again, we will discuss this in more detail in Chapters 6 and 8.
In short, PageRank reflects the quality of a web page. The number of links pointing to that page has a direct influence on PageRank, but abuse forces Google to change its algorithm and thus you to modify strategy to demonstrate websites are deserving of their popularity.
At the same time, the higher the importance of a website that links to a web page, the better it is for the pages the linked.
It is important to note that each web page has an assigned rank, which divides amongst the links going out from that page. The more links that point out from a web page the less important each link becomes. Therefore, PageRank is not only about a link coming from a popular site, it is also proportional to the number of links going out from that site.
Chapter 6 will discuss in more detail how to make the most of how PageRank flows within a site, which can help one control the profile of the most important pages on a website.
Robots.txt and Sitemaps
The first thing a spider does when visiting a website is to look for a file entitled “robots.txt”. This file instructs a spider which parts of a website to index and which parts to ignore. Chapters 6 and 11 will cover “robots.txt” in more detail.
Sitemaps are an easy way for webmasters to inform search engines what pages on a website are available for crawling by the spiders. In short, a sitemap is a file that lists URLs for each page along with additional data about each page, such as when it was last updated how often updated and its relative importance to other pages on the website. This information helps search engines crawl a website more intelligently.
We cover Sitemaps in more detail in Chapter 11.