Python Search Engine app

[Harlin Seritt]

Hi,

Is anyone aware of an available open-source/free search engine app
(something similar to HTDig) written in Python that is out there?
Googling has turned up nothing. Thought maybe I'd mine some of you
guys' minds on this.

thanks,

Harlin Seritt
Internet Villa: www.seritt.org

 

[Alan Meyer]

Hi,

Is anyone aware of an available open-source/free search engine app
(something similar to HTDig) written in Python that is out there?
Googling has turned up nothing. Thought maybe I'd mine some of you
guys' minds on this.

thanks,

Harlin Seritt
Internet Villa: www.seritt.org

I'm not aware of such a thing.

I stand ready to be corrected, but I think Python would not be a
good language for writing search engines. In the ones I've written
for custom projects - in C or PL/1, it has been necessary to
perform very high speed operations on highly compressed binary
structures - which is not Python's forte.

You might be able to put a Python interface over an engine written
in another language.

Alan

 

[gene tani]

Yes, there's a bunch. Google for "query parser" + python, "porter
stemming" "stopwords" "text indexer". Maybe lucene has some python
bindings, hmm?

 

[Fredrik Lundh]

Is anyone aware of an available open-source/free search engine app
(something similar to HTDig) written in Python that is out there?
Googling has turned up nothing. Thought maybe I'd mine some of you
guys' minds on this.

http://divmod.org/ has a couple of alternatives:

http://divmod.org/projects/lupy
http://divmod.org/projects/pyndex
http://divmod.org/projects/xapwrap

</F>

 

[Will McGugan]

I'm not aware of such a thing.

I stand ready to be corrected, but I think Python would not be a
good language for writing search engines. In the ones I've written
for custom projects - in C or PL/1, it has been necessary to
perform very high speed operations on highly compressed binary
structures - which is not Python's forte.

You might be able to put a Python interface over an engine written
in another language.

Wasn't Google's first search engine actualy written in Python?


Will McGugan

 

[Kent Johnson]

Yes, there's a bunch. Google for "query parser" + python, "porter
stemming" "stopwords" "text indexer". Maybe lucene has some python
bindings, hmm?

At least two Python versions of Lucene:
http://pylucene.osafoundation.org/
http://divmod.org/projects/lupy

Kent

 

[Alan Meyer]

....

I stand ready to be corrected, but I think Python would not be a
good language for writing search engines. In the ones I've written
for custom projects - in C or PL/1, it has been necessary to
perform very high speed operations on highly compressed binary
structures - which is not Python's forte.

You might be able to put a Python interface over an engine written
in another language.

It looks like the engines cited by the other responders to the query
are indeed mostly Python wrappings for C++ engines.

One that might not be, "lupy", notes under performance in the FAQ
the simple statement that "Java is faster".

As between Java and C++, I like Java much more as a programmer
friendly programming language, but I think I'd still choose C++ for
a search engine if the number of users or the size of the data was
large.

Alan

 

[Peter Hansen]

Wasn't Google's first search engine actualy written in Python?

Not unless it was some kind of quickie prototype that was immediately
discarded in favour of an optimized C version once it was built. The
rumour you're probably referring to is the *web spider* part of the
system, while the indexing/searching stuff is written in C or
something close (memory fails me on that point, but I'm sure Google
itself could easily tell you more).

There's an interesting paper about the early architecture/history of
Google which was a published by the authors (when they were still
in figurative diapers) and which was referenced recently in this
newsgroup. The archives would quickly pull it up for you.

-Peter

 

[Thomas Guettler]

Am Wed, 14 Sep 2005 07:39:35 -0700 schrieb Harlin Seritt:

Hi,

Is anyone aware of an available open-source/free search engine app
(something similar to HTDig) written in Python that is out there?
Googling has turned up nothing. Thought maybe I'd mine some of you
guys' minds on this.

Hi,

I use ZODB (not ZOPE) with ZCTextIndex.

HTH,
Thomas

 

[Jim]

I am finding this discussion illuminating. Can I ask: does it
possibly make sense to use a dB? For an application of mine, I had
been looking at putting the text into PostGres, and using their
OpenFTS, at //http://openfts.sourceforge.net/ (there is a Python module
to talk to OpenFTS, although it is in an early stage of development).

Jim

 

[asmlinkage]

Wasn't Google's first search engine actualy written in Python?

I would'nt be surprised if they actually used python to do some
prototyping of their algorithms but I cannot imagine the search engine
actually being written in python.

Search engines consist of many sub-systems like spidering, tokenizing,
indexing, compression, ranking etc ... I think the main portion where
google or as a matter of fact many search companies would use python
is probably for spidering and for all the glue that supports the system.