modularity and sophisticated filtering

[Eric Dean Tribble <tribble>]

I just realized some great benefits of the modularization in the
filtering algorithms.  The power results for the design.  The
modularity screams with new ideas for more sophisticated filtering
possibilities (ATTENTION frontend guys).  Caveat: I believe the claims
below are true, but I won't try to really prove them till they are necessary.

Remember that filtering information currently is disjunctive.  We can
represent endorsements of many different people and filter for
documents with any one of a set of endorsements.  When I figured out
how to filter based on link types and endorsements, I actually
discovered two things.  The first is the ability to attach filtering
information to sections of documents rather than only full documents.
MarkM and I only designed enough of this to expect it to be
straightforward.  The second that I only hit me today is that filters
can be conjuctive.  We would filter for link-type AND endorsements.  

I noticed this after realizing that negative filters are equally
trivial: give me all the things NOT endorsed by Rifkin.  The "naive"
implementation of all this is log in the size of the search space and
linear in the number of filters (naive indeed!).  Since we have cheap
disjuctive filters, a 20 test filter would still be feasiable (even
before we get clever about them!)  

Now add filters for all endorsements within the last three years.  Or
have un-endorsements.  Then I could search for all documents endorsed
by X but not unendorsed by X!  Or search for all documents with more
than 50 references to tham that pass your other filters.  Or....

Essentially, the current scheme could support searches based on a
general criterion and test the criterion O(n*log(n)) times (n = size
of the solution)!  Different criterion require widding the appropriate
information through the canopies, but it works for all cases I can
think of.  This is the claim that spurred my excitement!

Or did I forget to mention that all this works just fine for sensors,
too.  The thing that triggerred this realization was redoing bert
hopping.  The idea of ringing a sensor when a bert moved OFF of a
document tickled me.

Just incidentally Marcs, this just falls out of the current
implementation (in theory :-) and won't take any extra time.  Adding
the extensions will take time, but I don't intend to do them until
someone really astounds us with uses for such filtering.  And for
endorsing sections of documents (subBerts, links).

Whew!  I'm calming down now....
Hee hee.  ha ha.  Giggle, gasp.  Chuckle, hiccup, snort...
Whippeee...

This orignally started out as a vindication of well structured
programming.  I invented the wierd filter stuff on the fly.  Hmm. I
think I'll go to sleep now....

dean