Your Website And Smart “Spiders”

Not too long ago, to keep your search engine rankings for your website up there, you merely had to put some content up and add a few links or a video. Not so much anymore because the spiders have been getting smarter with every passing month.

First, let’s assume you don’t know what a spider (or bot) is in search engine terms. Google, Yahoo, Bing (Microsoft) and all other search engines send out little programs that, in effect, roam the web and “crawl” over every single website they can find all over the world, and then send info on what they find back “home.” How many web pages is that?

Hold on to your hat: Google has indexed over 40 Billion web pages (not sites, but all website pages). Your site is in there, too. Google will re-index all those pages every 30-90 days, too, so, yes, you are being watched … sort of. How do they do it?

First, Google has over 1,000,000 servers! Mind boggling, and they can find anything on them almost instantly so that you can have it when you do a search. Millions of spiders are coursing over pages and sending a constant stream of info back to the servers where every jot and tittle is cataloged and made ready for search.

As if this wasn’t enough, the spiders are getting smarter every day. One of the things they “measure” is relevancy, which is based on (among other things): The sources you quote; whether it looks, sounds or “smells” like another article (meaning it might be nothing more than a copy); how reliable your sources are (popular? oft quoted?), and; how many sources you have used. Bewildered? Discouraged?

Well, don’t be. It simply means that, when you put articles and content up there, you write shorter pieces on which you do more synthesis. Having longer articles isn’t always better because the spiders are looking “into” your pieces and judging the relevancy and originality. Step up your creativity a bit, cast a bit wider net for sources and you might wind up better than before.