Free-Reprint Article Written by: Kim Roach
See Terms of Reprint Below.
*****************************************************************
*
* This email is being delivered directly to members of the group:
*
* internet_marketing_articleblaster@yahoogroups.com
*
*****************************************************************
We have moved our TERMS OF REPRINT to the end of the article.
Be certain to read our TERMS OF REPRINT and honor our TERMS
OF REPRINT when you use this article. Thank you.
This article has been distributed by:
http://Article-Distribution.com
Helpful Link:
The Digital Millennium Copyright Act - Overview
http://www.gseis.ucla.edu/iclp/dmca1.htm
---------------------------------------------------------------------
Article Title:
==============
Today's Google Bots and What They Do
Article Description:
====================
Google currently indexes over 8 billion web pages. However,
before these pages were placed in the index, they were each
crawled by a special spider known as the GoogleBot. Unfortunately, many web masters do not know about the internal workings of this virtual robot. This article will attempt to
reveal some of the most important Google spiders, their function,
and how they affect you as a web master. We'll start with the
well-known GoogleBot.
Additional Article Information:
===============================
1061 Words; formatted to 65 Characters per Line
Distribution Date and Time: 2006-08-02 09:48:00
Written By: Kim Roach
Copyright: 2006, All Rights Reserved
Contact Email: mailto:kim@seo-news.com
For more free-reprint articles by Kim Roach, please visit:
http://thePhantomWriters.com/free_content/d/index.shtml#Kim_Roach
=============================================
Special Notice For Publishers and Webmasters:
=============================================
TEXT Copy-and-Paste and HTML Copy-and-Paste
Versions Of Article Are Available at:
http://thePhantomWriters.com/free_content/db/r/todays-google-bots.shtml#get_code
If you use this article on your website or in your ezine,
We Want To Know About It. Use the following URL to let
us know where you have used this article, and we will
include a link to your website on thePhantomWriters.com:
http://thephantomwriters.com/notify.php?id=3251&p=load
---------------------------------------------------------------------
Today's Google Bots and What They Do
Copyright � 2006 Kim Roach, All Rights Reserved
SEO-News
http://www.seo-news.com
Google currently indexes over 8 billion web pages. However,
before these pages were placed in the index, they were each
crawled by a special spider known as the GoogleBot.
Unfortunately, many web masters do not know about the internal
workings of this virtual robot.
In fact, Google actually uses a number of spiders to crawl the
Web. You can catch these spiders by examining your log files.
This article will attempt to reveal some of the most important
Google spiders, their function, and how they affect you as a web
master. We'll start with the well-known GoogleBot.
GoogleBot
Googlebot, as you probably know, is the search bot used by Google
to scour the web for new pages. Googlebot has two versions,
deepbot and freshbot. Deepbot is a deep crawler that tries to
follow every link on the web and download as many pages as it can
for the Google index. It also examines the internal structure of
a site, giving a complete picture for the index.
Freshbot, on the other hand, is a newer bot that crawls the web
looking for fresh content. The Google freshbot was implemented to
take some of the pressure off of the GoogleBot. The freshbot
recalls pages already in the index and then crawls them for new,
modified, or updated pages. In this way, Google is better
equipped to keep up with the ever-changing Web.
This means that the more you update your web site with new,
quality content, the more the Googlebot will come by to
check you out.
If you�d like to see the Googlebot crawling around your web
property more often, you need to obtain quality inbound links.
However, there is also one more step that you should take. If you
haven't already done so, you should create a Google Sitemap for
your site.
Creating a Google sitemap allows you to communicate with Google,
telling them about your most important pages, new pages, and
updated pages. In return, Google will provide you with some
valuable information as well. Google Sitemaps will tell you
about pages it was unable to crawl and links it was unable to
follow. This allows you to pinpoint problems and fix them so that
you can gain increased exposure in the search results.
The next Google bot in our lineup is known as the MediaBot.
MediaBot - used to analyze Adsense pages
useragent: Mediapartners-Google
MediaBot is the Google crawler for Adsense Publishers. Mediabot
is used to determine wich ads Google should display on Adsense
pages.
Google recommends that webmasters specifically add a command in
their robots.txt file that grants Mediabot access to their entire
site. To do this, simply enter the following code into your
robots.txt file:
User-agent: Mediapartners-Google*
Disallow:
This will ensure that the MediaBot is able to place relevant
Adsense ads on your site.
Keep in mind that ads can still be shown on a page if the
MediaBot has not yet visited. If that is the case, the ads chosen
will be based on the overall theme of the other pages on the site.
If no ads can be chosen, the dreaded public service announcements
are displayed instead.
There is an ongoing debate over whether or not the MediaBot is
giving websites with Adsense an advantage in the search engines.
Even Matt Cutts has confirmed that the Adsense Mediabot has
indexed webpages for Google's main index.
He states,"Pages with AdSense will not be indexed more
frequently. It's literally just a crawl cache, so if e.g. our
news crawl fetched a page and then Googlebot wanted the same
page, we'd retrieve the page from the crawl cache. But there's no
boost at all in rankings if you're in AdSense or Google News. You
don't get any more pages crawled either."
Matt Cutts claims that your website does not get any advantage
by using Adsense. However, in my mind, simply getting your site
updated in and of itself is an advantage.
This is very similar to Google Analytics, which also promotes a
slightly higher degree of spider activity.
Those who run Google Analytics on their site can expect
additional spider activity.
However, you certainly shouldn't depend on any of these tools
for getting your site indexed. The key to frequent spidering is
having quality inbound links, quality content, and frequent
updates.
Have images on your site? If so, you have likely been visited by
our next Google spider, the ImageBot.
ImageBot - used to crawl for the Image Search
useragent: GoogleBot-Image
The Imagebot prowls the Web for images to place in Google's image
search. Images are ranked based upon their filename, surrounding
text, alt text, and page title.
If you have a website that is primarily image based, then you
would definitely want to optimize your images to receive some
extra Google traffic.
On the other hand, some web sites may not benefit from Google
image search. In most cases, the traffic from the Image search
engine is very low quality and rarely converts into buyers. Many
people are often just looking for images that they can swipe. So,
if you want to save some bandwidth, use your robots.txt file to
block ImageBot from accessing your image directory.
One of the few exceptions might be if you have a site dedicated
to downloadable images.
Our final bot is completely dedicated to the Google Adwords
program.
AdsBot - Checks Adwords landing pages for quality
useragent: AdsBot-Google
AdsBot is one of Google's newest spiders. This new crawler is
used to analyze the content of advertising landing pages, which
helps determing the Quality score that Google assigns to your
ads.
Google uses this Quality score in combination with the amount
you bid to determine the position of your ads. Therefore, ads
with a high quality score can rank higher even if other
advertisers are paying more than you.
This is one of Google's many efforts to ensure that they are
delivering the best results to their users.
Can you still block being spidered? Of course, but it will lower
your overall Adwords quality score, which could end up lowering
the positioning of your ads. If possible, it is best to give
AdsBot complete access to your site.
Today's Google bots are becoming more advanced all the time.
However, nothing beats relevant, quality, updated content.
Deliver that and the search engines will eat it up.
---------------------------------------------------------------------
Kim Roach is a staff writer and editor for the SiteProNews
(http://www.sitepronews.com) & SEO-News (http://www.seo-news.com)
newsletters. You can contact Kim at: kim@seo-news.com
This article may be freely distributed without modification and
provided that the copyright notice and author information remain
intact.
--- END ARTICLE ---
Get TEXT or HTML Copy-and-Paste Versions Of This Article at:
http://thePhantomWriters.com/free_content/db/r/todays-google-bots.shtml#get_code
.....................................
TERMS OF REPRINT - Publication Rules
(Last Updated: May 11, 2006)
Our TERMS OF REPRINT are fully enforcable under the terms of:
The Digital Millennium Copyright Act
http://thomas.loc.gov/cgi-bin/query/z?c105:H.R.2281.ENR:
.....................................
*** Digital Reprint Rights ***
* If you publish this article in a website/forum/blog,
You Must Set All URL's or Mailto Addresses in the body
of the article AND in the Author's Resource Box as
Hyperlinks (clickable links).
* Links must remain in the form that we published them.
Clean links should point to the Author's links without
redirects having been inserted into the copy.
* You are not allowed to Change or Delete any Words or
Links in the Article or Resource Box. Paragraph breaks
must be retained with articles. You can change where
the paragraph breaks fall, but you cannot eliminate all
paragraph breaks as some have chosen to do.
* Email Distribution of this article Must be done through
Opt-in Email Only. No Unsolicited Commercial Email.
* You Are Allowed to format the layout of the article for
proper display of the article in your website or in your
ezine, so long as you can maintain the author's interests
within the article.
* You may not use sentences from this article as an input
for any software that steals sentences from others in
order to build an article with software. The copyright on
this article applies to the "WHOLE" article.
*** Author Notification ***
We ask that you notify the author of publication of his
or her work. Kim Roach can be reached at:
kim@seo-news.com
*** Print Publication Reprint Rights ***
If you desire to publish this article in a PRINT
publication, you must contact the author directly
for Print Permission at:
mailto:kim@seo-news.com
.....................................
If you need help converting this text article for proper
hyperlinked placement in your webpage, please use this
free tool: http://thephantomwriters.com/link-builder.pl
=====================================================================
ABOUT THIS ARTICLE SUBMISSION
http://thePhantomWriters.com is a paid article distribution
service. thePhantomWriters.com and Article-Distribution.com
are owned and operated by Bill Platt of Stillwater, Oklahoma USA.
The content of this article is solely the property
and opinion of its author, Kim Roach
http://www.seo-news.com
---------------------------------------------------------------------
XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
---------------------------------------------------------------------
Yahoo! Groups Links
<*> To visit your group on the web, go to:
http://groups.yahoo.com/group/internet_marketing_articleblaster/
<*> To unsubscribe from this group, send an email to:
internet_marketing_articleblaster-unsubscribe@yahoogroups.com
<*> Your use of Yahoo! Groups is subject to:
http://docs.yahoo.com/info/terms/
No comments:
Post a Comment