webcrawler (bot) in javascript?

[bdy120602]

I have a few programs that search an entire site for a keyword, and I
run
them from my machine: "Light Web Searcher" and "Teleport Pro" are
just
a few. My objective is to create a bare-bones software application
identical to the functions available in the aforementioned pieces of
software.

What language would I be able to accomplish that with?


Thanks,


Danny

 

[Evertjan.]

wrote on 20 feb 2008 in comp.lang.javascript:

I have a few programs that search an entire site for a keyword, and I
run
them from my machine: "Light Web Searcher" and "Teleport Pro" are
just
a few. My objective is to create a bare-bones software application
identical to the functions available in the aforementioned pieces of
software.

What language would I be able to accomplish that with?

The Q is mainly off topic,
you can only ask if it can be done using Javascript.

Answer: Yes.

 

[bdy120602]

 wrote on 20 feb 2008 in comp.lang.javascript:



The Q is mainly off topic,
you can only ask if it can be done using Javascript.

Answer: Yes.

Great. Can you get me started on writing a web crawler (bot) in
Javascript. What tools do I need?

 

[Nick Fletcher]

Great.

The answer may be yes, but JavaScript would still be a poor choice. A
web crawler requires a lot of bandwidth and lots of text parsing. A
server running Java or C++ would do far better than your web browser
running JavaScript.

Can you get me started on writing a web crawler (bot) in
Javascript. What tools do I need?

Your brain and a text editor.

 

[Thomas 'PointedEars' Lahn]

wrote on 20 feb 2008 in comp.lang.javascript:
I have a few programs that search an entire site for a keyword, and I
run
them from my machine: "Light Web Searcher" and "Teleport Pro" are
just
a few. My objective is to create a bare-bones software application
identical to the functions available in the aforementioned pieces of
software.
What language would I be able to accomplish that with?
The Q is mainly off topic,
you can only ask if it can be done using Javascript.
Answer: Yes.
[...]

Great.

The answer may be yes, but JavaScript would still be a poor choice. A
web crawler requires a lot of bandwidth and lots of text parsing. A
server running Java or C++ would do far better than your web browser
running JavaScript.

You are overlooking the simple fact that where a program
runs is not a matter of the programming language at all.


PointedEars

 

[Bart Van der Donck]

Can you get me started on writing a web crawler (bot) in
Javascript. What tools do I need?

I don't want to toot my own horn, but you might be interested in a
cross-domain AJAX application. This way you can crawl your desired
remote resources while receiving and parsing the outcome in
javascript.

http://www.ajax-cross-domain.com

http://www.ajax-cross-domain.com/source/ACD-source.txt could be
adapted as follows:
- Comment code lines 90-96 in order to allow any URI's (only in
secured environments!).
- Line 239 might can be moved to the main caller file (before calling
ACD.js), so that the properties of the ACD-object are overwritten each
time.

Hope this helps,