User vs. Crawler

[Gomez]

Hi,
Is there a way to know if a session on my web server is from an actual user or an automated crawler.

please advise.
G

 

[Ray at]

Perhaps by looking at the user-agent that is making the request, although I
can't say for certain that crawling software doesn't send browser-looking
user-agent info, or that this isn't possible if the crawler software writer
wants it to. That would be pretty snakey though, IMO.

Load a page with this code. Note the http_user_agent variable,
specifically.

<%
For each x in Request.serverVariables
Response.Write x & " = " & Request.Servervariables(x)
Response.Write "<hr>"
Next
%>

Ray at work

 

[Evertjan.]

Hi, Is there a way to know if a session on my web server
is from an actual user or an automated crawler.

Perhaps by looking at the user-agent that is making the request,
although I can't say for certain that crawling software doesn't send
browser-looking user-agent info, or that this isn't possible if the
crawler software writer wants it to.

Yes

Google puts "googlebot in Request.ServerVariables("HTTP_USER_AGENT")

More:

1
using
r = Request.ServerVariables("HTTP_REFERER")

you can check for:

"64.68." crawler googlebot.com
"216.239." crawler googlebot.com
"209.237.237." Alexa
"209.237.238." Alexa
"62.69.162." ingrid.ilse.nl
"216.39." AltaVista
etc., etc.

2
In hindsight, if a Request.ServerVariables("HTTP_REFERER") single IP
looks at many pages in a timeframe surpassing the maximum
vingerleftmousclick and if every page is a seperate session, it is
probably eiter a crawlerbot or [if there is also a strange supect
querystring] a hacker.