Home All Groups Group Topic Archive Search About

Need to block Web Spider software like Teleport pro

Author
15 Apr 2005 6:10 AM
Rijesh
Hi,

Recently our website crashed due to malicious activities by a stranger by
using web spider software Teleport pro. Teleport Pro is the software that
does this web capture activity, the client that was repeatedly hitting us
every few seconds with the ‘web capture’ and ultimately caused for the web
server crashes.

I did some research on how to block web spider software like Teleport pro. I
couldn’t find any effective way to block it as it works like a browser and
does the trick with HTTP functions. One way to block such activities is by
blocking IP address, the way which we did. Yes, if they start it from another
IP address again we will be in trouble.  So blocking IP address is just a
temporary solution.

I would like to know an effective way to prevent such software from
spidering my site and making the content available off line. Can anyone
provide some ideas to block such software activities? Is there any patch,
method to block it with in IIS?

There are some software available to protect the content that capable to
block Teleport pro , The one is TagsLock,
http://www.tagslock.com/tagslock_pro.htm . But I am not sure how efficient
this software and how it works. Did anyone tested it and give me the opinion?
Is that cause to decrease the performance?

Ultimately I want a solution to block similar software activities in an
effective way and any advice from anyone would be grateful.

Thanks,
Rijesh.

Author
15 Apr 2005 12:39 PM
Tom Kaminski [MVP]
Show quote Hide quote
"Rijesh" <Rij***@discussions.microsoft.com> wrote in message
news:DA560DB6-C7B5-4F09-9927-D1076DD0816A@microsoft.com...
> Hi,
>
> Recently our website crashed due to malicious activities by a stranger by
> using web spider software Teleport pro. Teleport Pro is the software that
> does this web capture activity, the client that was repeatedly hitting us
> every few seconds with the 'web capture' and ultimately caused for the web
> server crashes.
>
> I did some research on how to block web spider software like Teleport pro.
I
> couldn't find any effective way to block it as it works like a browser and
> does the trick with HTTP functions. One way to block such activities is by
> blocking IP address, the way which we did. Yes, if they start it from
another
> IP address again we will be in trouble.  So blocking IP address is just a
> temporary solution.
>
> I would like to know an effective way to prevent such software from
> spidering my site and making the content available off line. Can anyone
> provide some ideas to block such software activities? Is there any patch,
> method to block it with in IIS?
>
> There are some software available to protect the content that capable to
> block Teleport pro , The one is TagsLock,
> http://www.tagslock.com/tagslock_pro.htm . But I am not sure how efficient
> this software and how it works. Did anyone tested it and give me the
opinion?
> Is that cause to decrease the performance?
>
> Ultimately I want a solution to block similar software activities in an
> effective way and any advice from anyone would be grateful.

Because it has this feature:

"Configurable Agent Identity allows Teleport Pro to impersonate popular
browsers; gets data from even the stingiest servers"

it doesn't appear that you can do anything about it, other than monitor and
block IP addresses.  Why is your server crashing though?  That shouldn't
happen ...

Author
5 May 2005 7:20 AM
Rijesh
Our server has VCMS plug-in configured, so each time webserver checks the URL
though plug-in and eventually i think the plug-in cause the problem. Not sure
why?

Thanks,
Rijesh.

Show quoteHide quote
"Tom Kaminski [MVP]" wrote:

> "Rijesh" <Rij***@discussions.microsoft.com> wrote in message
> news:DA560DB6-C7B5-4F09-9927-D1076DD0816A@microsoft.com...
> > Hi,
> >
> > Recently our website crashed due to malicious activities by a stranger by
> > using web spider software Teleport pro. Teleport Pro is the software that
> > does this web capture activity, the client that was repeatedly hitting us
> > every few seconds with the 'web capture' and ultimately caused for the web
> > server crashes.
> >
> > I did some research on how to block web spider software like Teleport pro.
> I
> > couldn't find any effective way to block it as it works like a browser and
> > does the trick with HTTP functions. One way to block such activities is by
> > blocking IP address, the way which we did. Yes, if they start it from
> another
> > IP address again we will be in trouble.  So blocking IP address is just a
> > temporary solution.
> >
> > I would like to know an effective way to prevent such software from
> > spidering my site and making the content available off line. Can anyone
> > provide some ideas to block such software activities? Is there any patch,
> > method to block it with in IIS?
> >
> > There are some software available to protect the content that capable to
> > block Teleport pro , The one is TagsLock,
> > http://www.tagslock.com/tagslock_pro.htm . But I am not sure how efficient
> > this software and how it works. Did anyone tested it and give me the
> opinion?
> > Is that cause to decrease the performance?
> >
> > Ultimately I want a solution to block similar software activities in an
> > effective way and any advice from anyone would be grateful.
>
> Because it has this feature:
>
> "Configurable Agent Identity allows Teleport Pro to impersonate popular
> browsers; gets data from even the stingiest servers"
>
> it doesn't appear that you can do anything about it, other than monitor and
> block IP addresses.  Why is your server crashing though?  That shouldn't
> happen ...
>
> --
> Tom Kaminski IIS MVP
> http://www.microsoft.com/windowsserver2003/community/centers/iis/
> http://mvp.support.microsoft.com/
> http://www.iistoolshed.com/ - tools, scripts, and utilities for running IIS
>
>
>
Author
5 May 2005 8:18 AM
Rijesh
What is Configurable Agent Identity?

Show quoteHide quote
"Tom Kaminski [MVP]" wrote:

> "Rijesh" <Rij***@discussions.microsoft.com> wrote in message
> news:DA560DB6-C7B5-4F09-9927-D1076DD0816A@microsoft.com...
> > Hi,
> >
> > Recently our website crashed due to malicious activities by a stranger by
> > using web spider software Teleport pro. Teleport Pro is the software that
> > does this web capture activity, the client that was repeatedly hitting us
> > every few seconds with the 'web capture' and ultimately caused for the web
> > server crashes.
> >
> > I did some research on how to block web spider software like Teleport pro.
> I
> > couldn't find any effective way to block it as it works like a browser and
> > does the trick with HTTP functions. One way to block such activities is by
> > blocking IP address, the way which we did. Yes, if they start it from
> another
> > IP address again we will be in trouble.  So blocking IP address is just a
> > temporary solution.
> >
> > I would like to know an effective way to prevent such software from
> > spidering my site and making the content available off line. Can anyone
> > provide some ideas to block such software activities? Is there any patch,
> > method to block it with in IIS?
> >
> > There are some software available to protect the content that capable to
> > block Teleport pro , The one is TagsLock,
> > http://www.tagslock.com/tagslock_pro.htm . But I am not sure how efficient
> > this software and how it works. Did anyone tested it and give me the
> opinion?
> > Is that cause to decrease the performance?
> >
> > Ultimately I want a solution to block similar software activities in an
> > effective way and any advice from anyone would be grateful.
>
> Because it has this feature:
>
> "Configurable Agent Identity allows Teleport Pro to impersonate popular
> browsers; gets data from even the stingiest servers"
>
> it doesn't appear that you can do anything about it, other than monitor and
> block IP addresses.  Why is your server crashing though?  That shouldn't
> happen ...
>
> --
> Tom Kaminski IIS MVP
> http://www.microsoft.com/windowsserver2003/community/centers/iis/
> http://mvp.support.microsoft.com/
> http://www.iistoolshed.com/ - tools, scripts, and utilities for running IIS
>
>
>
Author
5 May 2005 2:33 PM
Tom Kaminski [MVP]
There's a user agent string that gets sent by a web browser that identifies
the browser type and version as well as client OS.  Teleport pro can be
configured to send any user agent info you desire so it appears to be just
another browser.

Show quoteHide quote
"Rijesh" <Rij***@discussions.microsoft.com> wrote in message
news:62CFCECA-99E4-46C6-97AA-C01EDB0054D9@microsoft.com...
> What is Configurable Agent Identity?
>
> "Tom Kaminski [MVP]" wrote:
>
>> "Rijesh" <Rij***@discussions.microsoft.com> wrote in message
>> news:DA560DB6-C7B5-4F09-9927-D1076DD0816A@microsoft.com...
>> > Hi,
>> >
>> > Recently our website crashed due to malicious activities by a stranger
>> > by
>> > using web spider software Teleport pro. Teleport Pro is the software
>> > that
>> > does this web capture activity, the client that was repeatedly hitting
>> > us
>> > every few seconds with the 'web capture' and ultimately caused for the
>> > web
>> > server crashes.
>> >
>> > I did some research on how to block web spider software like Teleport
>> > pro.
>> I
>> > couldn't find any effective way to block it as it works like a browser
>> > and
>> > does the trick with HTTP functions. One way to block such activities is
>> > by
>> > blocking IP address, the way which we did. Yes, if they start it from
>> another
>> > IP address again we will be in trouble.  So blocking IP address is just
>> > a
>> > temporary solution.
>> >
>> > I would like to know an effective way to prevent such software from
>> > spidering my site and making the content available off line. Can anyone
>> > provide some ideas to block such software activities? Is there any
>> > patch,
>> > method to block it with in IIS?
>> >
>> > There are some software available to protect the content that capable
>> > to
>> > block Teleport pro , The one is TagsLock,
>> > http://www.tagslock.com/tagslock_pro.htm . But I am not sure how
>> > efficient
>> > this software and how it works. Did anyone tested it and give me the
>> opinion?
>> > Is that cause to decrease the performance?
>> >
>> > Ultimately I want a solution to block similar software activities in an
>> > effective way and any advice from anyone would be grateful.
>>
>> Because it has this feature:
>>
>> "Configurable Agent Identity allows Teleport Pro to impersonate popular
>> browsers; gets data from even the stingiest servers"
>>
>> it doesn't appear that you can do anything about it, other than monitor
>> and
>> block IP addresses.  Why is your server crashing though?  That shouldn't
>> happen ...
>>
>> --
>> Tom Kaminski IIS MVP
>> http://www.microsoft.com/windowsserver2003/community/centers/iis/
>> http://mvp.support.microsoft.com/
>> http://www.iistoolshed.com/ - tools, scripts, and utilities for running
>> IIS
>>
>>
>>
Author
9 May 2005 5:11 AM
Rijesh
Currently the Teleport Pro have user agent label "Teleport Pro/1.29.1718”, so
how do I block access on the server level, if user agent is above? I know it
is not feasible, but it would be an approach towards blocking of such browser
access. Apache have directive to control access based on user agent, but I
don't find anything in IIS. Some times these type requirement are very much
needed.

Thanks,
Rijesh.

Show quoteHide quote
"Tom Kaminski [MVP]" wrote:

> There's a user agent string that gets sent by a web browser that identifies
> the browser type and version as well as client OS.  Teleport pro can be
> configured to send any user agent info you desire so it appears to be just
> another browser.
>
> "Rijesh" <Rij***@discussions.microsoft.com> wrote in message
> news:62CFCECA-99E4-46C6-97AA-C01EDB0054D9@microsoft.com...
> > What is Configurable Agent Identity?
> >
> > "Tom Kaminski [MVP]" wrote:
> >
> >> "Rijesh" <Rij***@discussions.microsoft.com> wrote in message
> >> news:DA560DB6-C7B5-4F09-9927-D1076DD0816A@microsoft.com...
> >> > Hi,
> >> >
> >> > Recently our website crashed due to malicious activities by a stranger
> >> > by
> >> > using web spider software Teleport pro. Teleport Pro is the software
> >> > that
> >> > does this web capture activity, the client that was repeatedly hitting
> >> > us
> >> > every few seconds with the 'web capture' and ultimately caused for the
> >> > web
> >> > server crashes.
> >> >
> >> > I did some research on how to block web spider software like Teleport
> >> > pro.
> >> I
> >> > couldn't find any effective way to block it as it works like a browser
> >> > and
> >> > does the trick with HTTP functions. One way to block such activities is
> >> > by
> >> > blocking IP address, the way which we did. Yes, if they start it from
> >> another
> >> > IP address again we will be in trouble.  So blocking IP address is just
> >> > a
> >> > temporary solution.
> >> >
> >> > I would like to know an effective way to prevent such software from
> >> > spidering my site and making the content available off line. Can anyone
> >> > provide some ideas to block such software activities? Is there any
> >> > patch,
> >> > method to block it with in IIS?
> >> >
> >> > There are some software available to protect the content that capable
> >> > to
> >> > block Teleport pro , The one is TagsLock,
> >> > http://www.tagslock.com/tagslock_pro.htm . But I am not sure how
> >> > efficient
> >> > this software and how it works. Did anyone tested it and give me the
> >> opinion?
> >> > Is that cause to decrease the performance?
> >> >
> >> > Ultimately I want a solution to block similar software activities in an
> >> > effective way and any advice from anyone would be grateful.
> >>
> >> Because it has this feature:
> >>
> >> "Configurable Agent Identity allows Teleport Pro to impersonate popular
> >> browsers; gets data from even the stingiest servers"
> >>
> >> it doesn't appear that you can do anything about it, other than monitor
> >> and
> >> block IP addresses.  Why is your server crashing though?  That shouldn't
> >> happen ...
> >>
> >> --
> >> Tom Kaminski IIS MVP
> >> http://www.microsoft.com/windowsserver2003/community/centers/iis/
> >> http://mvp.support.microsoft.com/
> >> http://www.iistoolshed.com/ - tools, scripts, and utilities for running
> >> IIS
> >>
> >>
> >>
>
>
>
Author
9 May 2005 2:18 PM
Tom Kaminski [MVP]
"Rijesh" <Rij***@discussions.microsoft.com> wrote in message
news:D1FAF495-F159-4C94-BDAB-A0CE29C486EE@microsoft.com...
> Currently the Teleport Pro have user agent label "Teleport Pro/1.29.1718",
> so
> how do I block access on the server level, if user agent is above? I know
> it
> is not feasible, but it would be an approach towards blocking of such
> browser
> access. Apache have directive to control access based on user agent, but I
> don't find anything in IIS. Some times these type requirement are very
> much
> needed.

How about URLScan and the [DenyHeaders] section?
http://support.microsoft.com/default.aspx?scid=kb;%5BLN%5D;326444