...How to parse search engine results fast?

[VB]

Hi,

I'm building a metaseach engine based on data mining techniques....but
this is not important...

My question is about performances of the activity of scraping search
engine results from an HTML response page.

I see that some metasearch engines (Mamma, DogPile, Vivisimo & C.)
present top 50 results of 3-5 search engines in about 1 seconds.

With my perl script I am able to retrieve top 100 results of Google in
about 1,5 seconds, but from only one search engine!

Somebody (very much skilled in Perl) can tell me some advanced
technique (parallelism, thread...bo?) to retrieve from 3-5 search
engines very fast? (Hardware not included in this issue, I have a fast
hardware)


Excuse me for my english (I'm italian) and for my poor Perl skills.

Thanks,

VB

 

[phaylon]

My question is about performances of the activity of scraping search
engine results from an HTML response page.

Maybe they /asked/ and used the API provided by some SE's?

Somebody (very much skilled in Perl) can tell me some advanced technique
(parallelism, thread...bo?) to retrieve from 3-5 search engines very fast?
(Hardware not included in this issue, I have a fast hardware)

- Very fast hardware with enough resources.
- Internet Connection
- Interfaces, see above.

There may be much more, but I can't see the Perl relation (I've written in
Perl may be not enough, this group is more on coding Perl, not about
technologies which can be coded with Perl, would be a wide field.);

hth,
p

 

[Anno Siegel]

Hi,

I'm building a metaseach engine based on data mining techniques....but
this is not important...

Then why mention it?

My question is about performances of the activity of scraping search
engine results from an HTML response page.

Looks like you use "scraping" to mean, roughly, parsing.

The answer would depend on the format of the response page. Since you
don't mention which search engines you are tapping into, there is
nothing we can say about that.

Except that the time needed to parse the results will most likely
be small compared to the time taken to retrieve them.

I see that some metasearch engines (Mamma, DogPile, Vivisimo & C.)
present top 50 results of 3-5 search engines in about 1 seconds.

With my perl script I am able to retrieve top 100 results of Google in
about 1,5 seconds, but from only one search engine!

Well, Perl isn't the fastest of languages. If you want super-fast,
don't use Perl.

Somebody (very much skilled in Perl) can tell me some advanced
technique (parallelism, thread...bo?) to retrieve from 3-5 search
engines very fast? ...

Finally you get to the core of your question. The one thing you can
do to arrive at results faster is to handle multiple queries in
parallel. See "perldoc perlipc" for general techniques, and "perldoc
-f fork" and "perldoc -f open" for the basic methods. Also check out
LWP:arallel on CPAN, it could be useful.

... (Hardware not included in this issue, I have a fast hardware)

Again, your local processing speed will not be the limiting factor. Net
delay and the vastly more extensive processing on the actual search engine
will.

Anno

 

[Gregory Toomey]

Hi,

I'm building a metaseach engine based on data mining techniques....but
this is not important...

My question is about performances of the activity of scraping search
engine results from an HTML response page.

Use C or possibly Perl, but that's not your problem.

The problem is the copyright lawsuit thats heading your way.

gtoomey