Damn you InktomiSearch! Damn you to hell!!!!!!!

February 26, 2006 Posted by Tyler Cruz

So I’m checking my traffic logs as I usually do once every day or two, and I notice traffic has been a bit extra high the past few days. No mystery here. The bots must be at it again. I get these 3-5 day mini-traffic spikes usually about once every 10-12 days or so. However, I decided to look a bit closer, and I didn’t like what I saw.

Those are unique hits by day. I average around 6000 per day for both PokerForums and Movie Vault, but obviously something is amiss. I already know it’s from bots, but again, I wanted to check further.

Referral logs didn’t show anything out of the ordinary. 90% from direct type-ins/bookmarks, 5% from search engines, and 5% from other sites.

However, when I went to check the Domains and Users demographics for the past 6 days, on both sites, InktomiSearch accounted for 50% on each site! Fuck!

The browser and robots section wasn’t nearly as bad, with only a few percent being taken by bots for Movie Vault, and 10% by Gigabot for PokerForums.

So I go to check the month of January. Browsers and robots were down to minimal numbers, but damn InktomiSearch was still 50% on both sites!

So now fellow brethren, I have the following questions for you:

1. What does the InktomiSearch.com user agent crawl for? What search engines?
2. What does the GigaBot.com user agent crawl for? What search engines?
3. Is this normal? To have such a dramatically high percentage of users being annexed by a spider?
4. Check your logs, and let me know what % of your userbase and traffic are spiders. Do not check your referral area, check your domain/users area.

So this is saddening. Because as far as I can tell, this means that my traffic is really only about 50% as I thought it was. And as my sites grow in content size, the balance will fall more in favour of the spiders, as more will be needed to recrawl the large portion of pages.

Moving on…

I’ve commissioned a presentational video, made in flash, to be used as a motivational and inspirational tool for other webmasters.

It will be around 5-6 minutes long, and based on my meagre successes, encoporating custom commissioned music, video, photos, and text.

However, to help underline the point that this isn’t an isolated case, and to help score the point of the video to help inspire, near the end I’d like to have a quick flashing of names and income of other successful internet entrepreneurs as well.

I was looking for the following information:

Name (First name, Last name)
URL of 1-3 sites (optional)
Income (either monthly average profits or yealy profit range or daily range)

Name, Age, and Income are mandatory.

The video right now is around 60% done and is looking good. So if you could reply here with that information, that’d be great. Also, if you could spread the word to your other web entrepreneur friends, that’d help out too,.


If you enjoyed this post, please consider leaving a comment below, subscribing to my RSS feed, or following me on Twitter.
Posted: February 26th, 2006 under My Websites  

4 Responses to “Damn you InktomiSearch! Damn you to hell!!!!!!!”

  1. Saadh says:

    Hey there Tyler,

    What do you use for checking all of your logs? Currently, I only look at awstats, although I have analog + webalizer on my server as well.

    What should I be using? Anything else, or am I just not using what I already have properly?

    Thanks, and I’ll try to send the one or two of the young entrepeneurs I know who are making good money as well 🙂

  2. Tyler says:

    I use Urchin, which was recently bought out by Google and turned into Google Analytics (http://www.google.com/analytics/) which is currently not accepting any new sign-ups, and is free, but tracks statistics through javascript/clientside (EW!)

    However, Urchin is still available for demo at http://www.google.com/analytics/urchin_software.html

  3. benjamin says:

    I can’t wait for that movie thing to come out, will be a great thing to have

  4. Dave Starr says:


    Regarding your question about Inktomi search it appears the take info only from Yahoo and send info to no other engine. See:


    (this is an excellent semi-live page that gives the relationships between all common serach engines .. hover over the logos for detailed info).

    I haven’t seen any big increases in traffic from Inktomi, but then my tiny blogs are nearly hidden and not of any mainstream interest as a ‘going concern’ poker site would be.

    Best reagrds


Leave a Reply