T
thunk
Hello,
Some months ago I showed up here after I had just finished a big part
of a project that has been morphing for 4 years. It all started with
the simple concept of using a DSL for experts to author directly
using a garden variety script language. The complexity "exploded" on
us because of the low quality of the data we were web-scraping.
One thing had led to another and another and at one point I just
decided to start testing a concept that said "the data will be
carefully controlled and there will be a module of code controlling
and serving the data of the store, and meta data on the store that can
make the DSL stuff simpler. We found such a domain... (and there are
many more).
The early project was successful. The complexity of writing "scripts"
was reduced to largely just picking from lists. The inference engine
part was made almost trivial by using multiple sections, each having
its own DSL. A concept of "Helper_Library"s (as also used by Ruleby)
helped simplify the "work" of the agents further. They became
"supervisors" that had only to keep track of a few variables, and
state a useful set of "if thens" to set things into motion in the
Helper_Library. Most of the time spent creating a single agent, in
fact, turned out to be the messages that are attached to each
"branch" (pass/fail).
I hope that makes some sense. The agents have been reduced to about
40-50 lines - even with blank lines and comments. An agent is created
into a browser and the file that results from this session, is pops
out at the Server end - thus the basis for "distributed" - there is
more but that is beyond the scope of this posting.
When I showed up trying to determine what the above system (that was
running nicely on machine) was I was met in a friendly and open manner
at first, but when I would not "show my source" the project was
pronounced "uninteresting" by some authorities here and generally
things went down hill from there.
Its not a totally simple system to describe, I am self taught, the
university training in programming I received was Assembler and
Fortran, I do not need or use the public forums as I find Ruby to be
entirely simple, I'm not too concerned on performance yet, and the
pick-ax book answers 99% of my questions.
Since that time I spent only a couple weeks out of 6 months tuning the
"core" of the system which totals only about 6,000 lines of ruby, as I
remember. No dependencies(!) Its all pretty simple, exactly the way
I like it. However it can't really "do" anything until there is a
working Helper_Module - and the Helper module by its nature is highly
"domain dependent".
Making public the "engine" never bothered me particularly, I'd love to
do this if it helped the Ruby community somehow and so on. Making the
public Helper_Library was a problem. However now I have one that I
could give away because it was just a testing stage. The code has
been removed from any web issues also - the input is strictly
simulated - meaning that at the highest level, in this domain - a set
of 200 agents can be fired at 100's of simulated medical records in a
loop and the results studied without a single character of input.
Fun.
Another way of saying this is that there is now a "bundle" of some 5
modules that I could make public that actually demonstrates the whole
system and would allow folks to go in whatever directions with it.
It's clocking - on my 1/2 state of the art system - about 2,500 agents
fired per second. (That's not a massive layoff at the FBI - that is
the running of 2,500 little script files which often abort for missing
info, negative inference and so on before they would complete.)
I know "it works on my machine". I'm up to my eyebrows in "domain
work" and can't really see giving it away only to have people write in
about it doesn't work this way or that way on my Windows machine or
whatever. I will be willing to work with one person to kick things
off and if that figures there is no value, or its all been done, or
whatever then C'est la vie. And ditto, in this spirit, for Docs.
I think this means that somebody needs to be just looking for an
'interesting project' in Ruby that has extra time, or somebody has to
see there are real commercial applications for this ball of concepts
and code. The latter seems true blatantly clear to me.
Sincerely,
George
Some months ago I showed up here after I had just finished a big part
of a project that has been morphing for 4 years. It all started with
the simple concept of using a DSL for experts to author directly
using a garden variety script language. The complexity "exploded" on
us because of the low quality of the data we were web-scraping.
One thing had led to another and another and at one point I just
decided to start testing a concept that said "the data will be
carefully controlled and there will be a module of code controlling
and serving the data of the store, and meta data on the store that can
make the DSL stuff simpler. We found such a domain... (and there are
many more).
The early project was successful. The complexity of writing "scripts"
was reduced to largely just picking from lists. The inference engine
part was made almost trivial by using multiple sections, each having
its own DSL. A concept of "Helper_Library"s (as also used by Ruleby)
helped simplify the "work" of the agents further. They became
"supervisors" that had only to keep track of a few variables, and
state a useful set of "if thens" to set things into motion in the
Helper_Library. Most of the time spent creating a single agent, in
fact, turned out to be the messages that are attached to each
"branch" (pass/fail).
I hope that makes some sense. The agents have been reduced to about
40-50 lines - even with blank lines and comments. An agent is created
into a browser and the file that results from this session, is pops
out at the Server end - thus the basis for "distributed" - there is
more but that is beyond the scope of this posting.
When I showed up trying to determine what the above system (that was
running nicely on machine) was I was met in a friendly and open manner
at first, but when I would not "show my source" the project was
pronounced "uninteresting" by some authorities here and generally
things went down hill from there.
Its not a totally simple system to describe, I am self taught, the
university training in programming I received was Assembler and
Fortran, I do not need or use the public forums as I find Ruby to be
entirely simple, I'm not too concerned on performance yet, and the
pick-ax book answers 99% of my questions.
Since that time I spent only a couple weeks out of 6 months tuning the
"core" of the system which totals only about 6,000 lines of ruby, as I
remember. No dependencies(!) Its all pretty simple, exactly the way
I like it. However it can't really "do" anything until there is a
working Helper_Module - and the Helper module by its nature is highly
"domain dependent".
Making public the "engine" never bothered me particularly, I'd love to
do this if it helped the Ruby community somehow and so on. Making the
public Helper_Library was a problem. However now I have one that I
could give away because it was just a testing stage. The code has
been removed from any web issues also - the input is strictly
simulated - meaning that at the highest level, in this domain - a set
of 200 agents can be fired at 100's of simulated medical records in a
loop and the results studied without a single character of input.
Fun.
Another way of saying this is that there is now a "bundle" of some 5
modules that I could make public that actually demonstrates the whole
system and would allow folks to go in whatever directions with it.
It's clocking - on my 1/2 state of the art system - about 2,500 agents
fired per second. (That's not a massive layoff at the FBI - that is
the running of 2,500 little script files which often abort for missing
info, negative inference and so on before they would complete.)
I know "it works on my machine". I'm up to my eyebrows in "domain
work" and can't really see giving it away only to have people write in
about it doesn't work this way or that way on my Windows machine or
whatever. I will be willing to work with one person to kick things
off and if that figures there is no value, or its all been done, or
whatever then C'est la vie. And ditto, in this spirit, for Docs.
I think this means that somebody needs to be just looking for an
'interesting project' in Ruby that has extra time, or somebody has to
see there are real commercial applications for this ball of concepts
and code. The latter seems true blatantly clear to me.
Sincerely,
George