Let’s try to understand these terms with an example:
You created a website named “Blackberryswitch.com,” with a landing page, and some product pages, and boom you got a well-running, ready-to-go live website. Congratulations!
You send this website to everyone and promote it, but did you think about whether Google knows about your website?
Yes it does! Google’s software continuously explores webpages and looks for new websites so that it has the information of all the websites that are running on its platform.
Otherwise, it’s like illegally running a website.
So Google named the process of finding new web pages as “URL discovery“
There is another way of telling Google that there is a new website in the market by submitting a list of pages. These are called Sitemaps.
Sitemaps:
A sitemap is a file with all the important information about a website, any updates made, or new pages added. This is specifically generated by the owner of the website to help the search engines find useful information about the website and make it easier to find them.
My website freshmarketers.com has 5-6 pages till now and only one of the page has been crawled yet but it is not necessary for me to create sitemaps as my website is very small. I can do it or leave it. Google will automatically find my pages via internal links.
Crawling
First Google searches for new or updated pages. It discovers web pages by following different links (this is also why link building helps), reading sitemaps etc. This is called crawling.
Indexing
When Google’s algorithm finds new pages. It stores the content and location (URL) of that page in its directory containing information of all the pages.
Google indexes it by seeing a page, reading the content, and adding it to the index. This is called Indexing.
The software (Google bot) that does this entire process of crawling and indexing is called Crawler.
Check if site is indexed?
In order for your site to appear on the Search engine result page (SERP), the Crawler must find and scan it. If the crawler has not crawled and indexed your site, no one will be able to see your site when they search online for the keywords.
To check if your site has been indexed or not:
Type in – “site:yourwebsite/
eg. If you don’t own a website, you can test with my website, copy this in Google- site:freshmarketers.com/
You will see my website with a couple of web pages, it will show the pages that has been crawled and indexed by Google.
If the search engine do not return any results, that means your website has not been crawled yet, most probably you need to wait.
Crawling and indexing is an automated task. It means website owners don’t need to do anything. Google automatically reaches the websites and adds them to their index while crawling the web.
But there could be chances that Google can miss the site.
What could be the reasons of not crawling? Let’s see :-
- Poor link network-If your site is not well connected with other sites. Google discovers new pages by following links from known pages. This is where backlinks help.
- New website- Crawling new websites takes time considering there are billions of pages submitted to Google for crawling.
- There is an error on the website which is why crawler could not scan the website
- The design of the website is making it difficult for crawler to scan the webpage.