We all know that the practice of Search Engine Optimization (SEO) keeps on evolving along with the underlying algorithms of the respective search engines such as Bing, DuckDuckGo, and Google. Webmasters and website owners continuously need to be watchful of their website’s indexing health with every new update in the crawling criteria of the respective search engines. However, despite all this uncertainty, certain fundamental SEO concepts remain the same for all websites irrespective of size, type, and functionality. If implemented correctly, these SEO concepts make the process of managing additional SEO requirements and compliances a breeze. This includes:
It is a text document placed in the same hosted directory as the website. This document is a de facto standard that defines a convention for webmasters to enable/disable the crawling of different pages in a website by search engine crawlers from different service providers. For instance, in the example below, the robots.txt file disallows the ‘/cgi-bin/’, ‘/tmp/’, and ‘/~joe/’ directories not to be indexed by any search engines.
User-agent: *
Disallow: /cgi-bin/
Disallow: /tmp/
Disallow: /~joe/
Alternatively, robot directives for specific pages can be given via meta-tags as well.
<html>
<head>
<title>...</title>
<META NAME="ROBOTS" CONTENT="NOINDEX, NOFOLLOW">
</head>
A typical HTML document is a form of markup language that is usually unstructured with little to no info about the semantics of the underlying content.Founded by Google, Microsoft, Yahoo, and Yandex, Schema.org provides vocabularies that are developed by an open community process. These vocabularies add schemas to the data available on an HTML page so that this data can be semantically understood, and classified by search engine crawlers. Schema vocabularies can be defined on a page in 3 ways: microdata, RDFa, and JSON-LD. The example below showcases microdata (itemscope, itemtype, and itemprop) attributes applied to the HTML markup of a page on an eCommerce website selling an LCD monitor:
Defining vocabularies and adding schemas to a website helps search engines to better understand the website’s content and deliver rich snippets which users see on search engine result pages after entering a search query.
Open Graph (OG) meta tags are used to implement the open graph protocol that converts a web page into a rich object in a social graph. This means open graph meta tags enable social media networks such as Facebook and Twitter to treat web pages as rich objects just like any other object within a social media network.So every time a Facebook user posts or shares a link to a web page with OG meta tags, the rich information highlighted by the tags is presented in the form of snippets. While snippets can be automatically generated, OG tags enable webmasters to exercise greater control over how a web page is perceived on social media networks.For instance, check out the OG tags of an IMDB movie in the example below from the Open Graph Protocol website:
the following is the Open Graph protocol markup for The Rock on IMDB:
<html prefix="og: https://ogp.me/ns#">
<head>
<title>The Rock (1996)</title>
<meta property="og:title" content="The Rock" />
<meta property="og:type" content="video.movie" />
<meta property="og:url" content="https://www.imdb.com/title/tt0117500/" />
<meta property="og:image" content="https://ia.media-imdb.com/images/rock.jpg" />
...
</head>
...
</html>
These tools from respective search engines enable website owners and webmasters to request website crawls, check for any indexing errors, consolidate duplicate content, submit sitemaps, conduct keyword research, conduct search traffic analysis, and execute other miscellaneous analyses. The basic idea behind integrating these tools with websites is to get a search engine’s perspective on the indexing performance and identify possible areas of improvement.
SEO is an endless discipline with an infinite scope of improvement and iterations. However, taking care of these aforementioned four interventions provides a strong base for architecting a healthy ongoing SEO practice.


Leave a Comment