 | | | | | |
Todd,
I have a couple of requests for the web interface search function.
Would it be feasable to institute some kind of search language, to
search for articles within a date range, within a certain newsgroup, or
from a certain poster?
Also, the results page currently lists: something(?), the article
number, and words found on each line. I would also like to be able to
see the articles' subject lines and dates posted. You know, to make ego
surfing easier. :-,
Cheers,
- Sproat
--
Jeremy H. Sproat - jsproat@geocities.com
http://www.geocities.com/SiliconValley/Horizon/5249/
| | | | | | | | | | | | | | | |
| |
|
Jeremy Sproat <jsproat@geocities.com> writes:
> Would it be feasable to institute some kind of search language, to
> search for articles within a date range, within a certain newsgroup, or
> from a certain poster?
I'll need to make something that indexes all the articles by date/time stamp
and by poster, but it can be done. I assume you'd want to search by
someone's real-life name rather than their news-posting name/address, right?
-- because some people post from multiple addresses. OTOH, some people have
the same name as other people; maybe being able to search both ways is
needed.
Searching within a certain newsgroup, however, is already there (it's been
there from day 1 of the search function) -- check out:
http://www.lugnet.com/news/search/
For the "in Groups" field you can type things like
lugnet.*
market.*
loc.uk.*
robotics
(Note that it automatically prepends "lugnet." onto the front for you if you
omit it.)
> Also, the results page currently lists: something(?), the article
> number, and words found on each line. I would also like to be able to
> see the articles' subject lines and dates posted. You know, to make ego
> surfing easier. :-,
I didn't have a little library for news articles when I wrote the search
function the first time, so that's why it's so bare-bones. :) But let me
see what I can do now. The other filterings will take a more work, but this
is probably pretty straightforward -- I'll allocate a 1/2 hour for it today
-- seems like a pretty useful thing now that the searches are returning many
more results than they used to.
--Todd
| | | | | | | | | | | | | | | | | | |
| |
|
lehman@javanet.com (Todd Lehman) writes:
> [...]
>
> Jeremy Sproat writes:
> > Also, the results page currently lists: something(?), the article
> > number, and words found on each line. I would also like to be able to
> > see the articles' subject lines and dates posted. You know, to make ego
> > surfing easier. :-,
>
> I didn't have a little library for news articles when I wrote the search
> function the first time, so that's why it's so bare-bones. :) But let me
> see what I can do now. The other filterings will take a more work, but this
> is probably pretty straightforward -- I'll allocate a 1/2 hour for it today
> -- seems like a pretty useful thing now that the searches are returning many
> more results than they used to.
OK, give it a whirl now -- it's a lot friendlier, but unforunately it's also
a *lot* slower -- because it's got to go fetch details about the article
which aren't part of the raw words-to-article index.
Generally, the more words you type (that is, the more specific your search),
the faster the results come back, because the crummy matches can be
separated from the good matches much more easily.
So, for example, if you search for a very common single word "cool" or
"lego", it takes a loooong time. But if you search for a very uncommon
single word like "atlas", then it's lickety-split. Something a little more
complex like "the best place to get linux" takes longer, but it's still
tolerable.
I may have to tweak the thresholding on what gets chopped off the bottom
when weeding out crummy matches -- it's probably returning too many matches
in most cases. It erred on the side of "too many" before when the results
didn't give any real info except the ng name.
Lemme know what you think.
--Todd
p.s. If what you're looking for is something that scours all the ng's for
any mention of your name, I think that's probably left as a separate
special-purpose thingie which does it all automagically like an agent and
sends you incremental information. I'd love to offer that as a service but
it's a pretty low priority.
| | | | | | | | | | | | | | | | | | | | |
Todd Lehman wrote:
> OK, give it a whirl now -- it's a lot friendlier, but unforunately it's also
> a *lot* slower -- because it's got to go fetch details about the article
> which aren't part of the raw words-to-article index.
> Lemme know what you think.
Yah, I like.
Todd, you're the CGI Lord. I hope you're making some money off if it!
Cheers,
- Sproat
--
Jeremy H. Sproat - jsproat@geocities.com
http://www.geocities.com/SiliconValley/Horizon/5249/
| | | | | | | | | | | | | | | | | | | |
| |
|
This is an append to a thread started about 8 months ago...
In lugnet.admin.general, Todd Lehman writes:
> Jeremy Sproat <jsproat@geocities.com> writes:
> > Would it be feasable to institute some kind of search language, to
> > search for articles within a date range, within a certain newsgroup, or
> > from a certain poster?
>
> I'll need to make something that indexes all the articles by date/time
> stamp and by poster, but it can be done.
OK, Jeremy!
What I did recently as a background task was rebuilt the news search
database from scracth, this time including article timestamps and increased
emphasis on the poster's name.
> I assume you'd want to search
> by someone's real-life name rather than their news-posting name/address,
> right? -- because some people post from multiple addresses. OTOH, some
> people have the same name as other people; maybe being able to search
> both ways is needed.
If you go to the root homepage and type someone's name, you'll see that
the first (best) matches are typically articles posted by that person,
followed by lesser matches which only happen to contain the person's name
in the body of the article. It's a hybrid interleaved weighted matcher
which gives words a higher emphasis the closer they are to the beginning
of the document (sum of 1/(10+n) where n is a hit) but which also takes
the age of the article into account.
With the timestamp ordering, you'll notice that the topmost articles also
tend to be the most recent ones. This is because the "target" timeframe is
set to "very recent" by default and the ranker is told that the timeframe is
moderately important (important enough to bias the results toward newer-
first, but not so important as to make the relative term hits irrelevant).
The data structures produced by the indexer do contain enough information
to filter articles on hard date boundaries, but while date ranges are
popular at DejaNews, I always found them incredibly frustrating. So I set
the timestamp ranker up to be a _fuzzy_ match. In an advanced search,
you'll be able to say: find articles with "orange halloween buckets" that
were posted "about a year ago," and that will cause the ranker to favor
older articles over than newer articles. If there were articles 12 months
old that matched, then great! -- but if not, then any matching articles 8
months old would still show up (but with a lower score).
Similarly, you might want to look something up that you remember a couple
keywords from going back about 2 or 3 weeks. If you can't remember the
exact date, you could tell the search engine to produce results with a
target of "about 3 weeks ago" and any matches in that approximate time
range would be ranked higher than older or newer matches.
To achieve a smooth match maching function, the fuzzy timestamp ranking
function uses a bell-shaped curve
y = exp(-.5 * ((delta / sigma) ^ 2)
where 'delta' is the difference between the "target" time and a given
article's timestamp and 'sigma' is the "time uncertainty factor." So a
relatively large sigma produces wide time spreads around the target, and
a small sigma produces fine time spreads. Both infinitely large and
infinitely small sigma cause the timestamps to be a non-factor.
The default target time is "right now" (i.e., right when you invoke the
search function) and the default sigma is 1 week, which gives extremely
fresh articles a score boost of 1.0, and articles 1 week old a boost of
.6065, articles 2 weeks old .1353, articles 3 weeks old .1111, etc. The
extra "umph" kicked in by the target-time-matching approaches zero
asymptotically as the time differential approaches infinity. It seems to
work pretty nicely with sigma set to 1 week by default, although I'd like
to make this configurable in an advanced search later, as well as possibly
an option for hard (non-fuzzy) date filtering.
> Searching within a certain newsgroup, however, is already there (it's been
> there from day 1 of the search function) -- check out:
>
> http://www.lugnet.com/news/search/
>
> For the "in Groups" field you can type things like
>
> lugnet.*
> market.*
> loc.uk.*
> robotics
>
> (Note that it automatically prepends "lugnet." onto the front for you if
> you omit it.)
The /news/search/ page went away with the new website reorg last month, but
it might be useful again if it came back as an advanced search page. Note
that searching within categories is now as easy as being in a category and
initiating a search right from there, rather than having to go to a special
search page and type in gunky regex pattern. (And it was never a real full-
blown regex matcher anyway -- all it understood was a single '*' since its
only purpose in life was to help filter categories.) Of course this part
could always also come back for an advanced search, since the low-level
group filter uses a regex internally. It's just a matter of identifying
which options are priorities and which ones are merely feeping creaturitis.
A recent change in the group filters, BTW, was to fix the searching of
crossposted articles. Before, if something was crossposted as
Newsgroups: lugnet.loc.us.ca.sf,lugnet.general,lugnet.announce
then it would only show up in searches of lugnet.loc.us.ca.sf or higher --
not in lugnet.general or lugnet.announce (d'Oh!). This misfeature was the
result a misguided attempt at saving space in the index database long before
the notion of restricting the searches to subgroups came up. The indexer now
uses a clever and efficient encoding (space, time, and code!) mechanism to
include crossposting information. :-)
> > Also, the results page currently lists: something(?), the article
> > number, and words found on each line. I would also like to be able to
> > see the articles' subject lines and dates posted. You know, to make ego
> > surfing easier. :-,
>
> I didn't have a little library for news articles when I wrote the search
> function the first time, so that's why it's so bare-bones. :) But let
> me see what I can do now. The other filterings will take a more work,
> but this is probably pretty straightforward -- I'll allocate a 1/2 hour
> for it today -- seems like a pretty useful thing now that the searches
> are returning many more results than they used to.
I forget exactly what happened after the above, but whatever it was, it was
tossed out when the new "real" interface was moved into place a few weeks
ago (the one which encases articles in 250-word snippet/preview boxes).
BTW, you can also now use '+' and '-' like AltaVista. (I probably mentioned
this a few weeks ago already, but oh well, here goes again.)
If you enter "+foo" then it means that "foo must exist in the matches; don't
show me things that don't contain foo." If you enter "-foo" then it means
that "foo mustn't exist in the matches; don't show me anything that contains
foo." For example:
jeremy sproat droideka (1354 matches)
jeremy sproat -droideka (1346 matches)
jeremy sproat +droideka (8 matches)
+droideka -jeremy -sproat (2 matches)
Welp, there you have it. Lemme know if it rocks or if it sucks.
--Todd
| | | | | | | | | | | | | | | | | |
| |
|
Todd Lehman wrote:
> This is an append to a thread started about 8 months ago...
> In lugnet.admin.general, Todd Lehman writes:
> > Jeremy Sproat <jsproat@geocities.com> writes:
> > > Would it be feasable to institute some kind of search language, to
> > > search for articles within a date range, within a certain newsgroup, or
> > > from a certain poster?
> > I'll need to make something that indexes all the articles by date/time
> > stamp and by poster, but it can be done.
> What I did recently as a background task was rebuilt the news search
> database from scracth, this time including article timestamps and increased
> emphasis on the poster's name.
> [...]
> The indexer now
> uses a clever and efficient encoding (space, time, and code!) mechanism to
> include crossposting information. :-)
Todd, you are a lean, mean, re-coding machine! :-, I like the date
indexing, but I haven't been able to figure out how to use it; e.g., what is
the syntax to, say, modify the query to prefer articles from about four
months ago?
> The /news/search/ page went away with the new website reorg last month, but
> it might be useful again if it came back as an advanced search page.
That'd be really cool, but take a break first. :-,
> To achieve a smooth match maching function, the fuzzy timestamp ranking
> function uses a bell-shaped curve
> y = exp(-.5 * ((delta / sigma) ^ 2)
> where 'delta' is the difference between the "target" time and a given
> article's timestamp and 'sigma' is the "time uncertainty factor." So a
> relatively large sigma produces wide time spreads around the target, and
> a small sigma produces fine time spreads. Both infinitely large and
> infinitely small sigma cause the timestamps to be a non-factor.
Whoa. All that went way over my head. It's intriguing though -- do you
have any recommendations for off-line reading on this subject? It appears
that a rudimentary understanding of statistics may also be useful...
> Welp, there you have it. Lemme know if it rocks or if it sucks.
It rocks, man. Thanks bunches! (FYI, I haven't been a welp since grade
nine... :-)
Cheers,
- jsproat
--
Jeremy H. Sproat <jsproat@io.com>
http://www.io.com/~jsproat
Darth Maul Lives
| | | | | | | | | | | | | | | | |
In lugnet.admin.general, Jeremy Sproat writes:
> [...] I haven't been able to figure out how to use it; e.g., what is
> the syntax to, say, modify the query to prefer articles from about four
> months ago?
It appears that for now we just have the recent (default) target time and
1-week sigma values. If there is a syntax for overriding these, Todd hasn't
chosen to tell us about it. I would guess that it isn't yet ready for general
use.
- Robert Munafo http://www.mrob.com/
LEGO: TC+++(8480) SW++ #+ S-- LS++ Hsp M+ A@ LM++ YB64m IC13
| | | | | | | | | | | | | | | | | |
| |
|
In lugnet.admin.general, "Robert Munafo" <munafo@gcctechNO.SPAMcom> writes:
> In lugnet.admin.general, Jeremy Sproat writes:
> > [...] I haven't been able to figure out how to use it; e.g., what is
> > the syntax to, say, modify the query to prefer articles from about four
> > months ago?
>
> It appears that for now we just have the recent (default) target time and
> 1-week sigma values. If there is a syntax for overriding these, Todd hasn't
> chosen to tell us about it. I would guess that it isn't yet ready for general
> use.
A syntax vis-à-vis URLs isn't yet defined, but how does this sound?--
There's currently q= for the query string:
http://www.lugnet.com/?q=jeremy+sproat
and qn= for subsequent pages of results:
http://www.lugnet.com/?q=jeremy+sproat&qn=60
so there are still 25 possibilities in the q*= namespace. How about qt= for
the target time and qs= for the sigma (provided that some other non-greek
mnemonic can be identified for 's'):
http://www.lugnet.com/?q=jeremy+sproat&qt=-10519200&qs=2629800
4 months is about 10519200 seconds, and 1 month is about 2629800 seconds, so
that query would mean "prefer articles from about four months ago, with a
preferential spread of about a week or two either way."
For clipping absolute time ranges, how do qt0= and qt1= sound?--
http://www.lugnet.com/?q=jeremy+sproat&qt0=-1814400&qt1=-0
with the sign of qt0-qt1 specifying the sort order, chronological or reverse
chronological.
As far as the numerical values themselves go, they could absolute times in
the standard epoch format (seconds since 1 Jan 1970 00:00:00 GMT) or
relative times or both, or expressed in hours or days instead of seconds.
I'd favor seconds and standard epoch time, with a transparent query
generator page. That is, say the advanced search page is at /news/search/,
and you specify "about 4 months ago." Then it should calculate the current
time minus 10519200 seconds for you automatically, and do a double-CGI call:
first to process the form and convert "4 months" into 10519200 and second to
respond to the query itself. (I wouldn't want to litter the URL line with
all of the possible options present on the actual human-interface form
because it's important to keep the URLs short and to separate (abstract) the
query interface layers.)
So when you click the Go button on the advanced search page, it would do
that via HTTP POST (not GET), and that script would interpret the query, and
rewrite it into a URL which gets spit out via a Location: header to enact
the actual query. (It would be 99% transparent -- all you'd actually see in
your browser is "contacting host..." twice in rapid succession.)
In the URL parser, positive values for time could be interpreted as absolute
values and negative values for time could be interpreted as relative values
(relative to the current time).
--Todd
| | | | | | | | | | | | | | | | | | |
| |
|
Todd Lehman wrote:
> In lugnet.admin.general, "Robert Munafo" <munafo@gcctechNO.SPAMcom> writes:
> > In lugnet.admin.general, Jeremy Sproat writes:
> > > [...] I haven't been able to figure out how to use it; e.g., what is
> > > the syntax to, say, modify the query to prefer articles from about four
> > > months ago?
> > It appears that for now we just have the recent (default) target time and
> > 1-week sigma values. If there is a syntax for overriding these, Todd hasn't
> > chosen to tell us about it. I would guess that it isn't yet ready for general
> > use.
> A syntax vis-à-vis URLs isn't yet defined, but how does this sound?--
> [snippage of URL syntax]
> I'd favor seconds and standard epoch time, with a transparent query
> generator page. That is, say the advanced search page is at /news/search/,
> and you specify "about 4 months ago."
Sure, sounds okay. I'm a little nervous about handling raw seconds; the big
numbers make my puny earthling brain hurt. :-, How about using the same
syntax used in the traffic report page; e.g.
http://www.lugnet.com/news/traffic/custom.cgi?hours=6
Shows what's happened in the past 6 hours. How about something like
http://www.lugnet.com/?q=jeremy+sproat&qtmonths=6&qsdays=7
to ask for the results to prefer articles submitted a month ago, with s
spread of 7 days? Hmmm... There would be lost flexibility then, as you can
only ask for integral values of one time metric. Arg. I'll have to
surrender to the intellectual superiority of my browser. :-P
> (I wouldn't want to litter the URL line with
> all of the possible options present on the actual human-interface form
> because it's important to keep the URLs short and to separate (abstract) the
> query interface layers.)
I don't think it matters, really. With the URL for something more or less
static (like an article), this is true, but we're talking about search
queries here. With regards to time, searches are very volatile stuff.
> So when you click the Go button on the advanced search page, it would do
> that via HTTP POST (not GET), and that script would interpret the query, and
> rewrite it into a URL which gets spit out via a Location: header to enact
> the actual query.
I like this idea. It'd let you totally separate the query interface from
the actual engine, allowing for gobs of changes to either end, needing
probably minimal changes to the glue-layer CGI.
Cheers,
- jsproat
--
Jeremy H. Sproat <jsproat@io.com>
http://www.io.com/~jsproat
Darth Maul Lives
| | | | | | | | | | | | | | | | | | | | | |
| |
|
In lugnet.admin.general, Sproaticus <jsproat@io.com> writes:
> Sure, sounds okay. I'm a little nervous about handling raw seconds; the big
> numbers make my puny earthling brain hurt. :-, How about using the same
> syntax used in the traffic report page; e.g.
>
> http://www.lugnet.com/news/traffic/custom.cgi?hours=6
>
> Shows what's happened in the past 6 hours. How about something like
>
> http://www.lugnet.com/?q=jeremy+sproat&qtmonths=6&qsdays=7
>
> to ask for the results to prefer articles submitted a month ago, with s
> spread of 7 days?
Apart from the URLs getting longer and more clutterred with parameters and
being harder to share via cut & paste into messages, it would probably also
mean two separate URL interpretation mechanisms (more code bloat, even if
it's broken into a separate library) rather than just one which converts
human time to epoch time along with everything else and feeds the result to
a lower layer.
BTW, how would you write the following range using the multi-parameter
approach?--
From: 31 Dec 1998 23:45:00 EST
to: 1 Jan 1999 00:30:00 EST
(Let's say you're looking for Happy New Year messages posted by people in
the New York City time zone last year.)
http://www.lugnet.com/loc/us/ny/?q=happy+new+year&........
What goes in the ....'s ?
> Hmmm... There would be lost flexibility then, as you can
> only ask for integral values of one time metric. Arg. I'll have to
> surrender to the intellectual superiority of my browser. :-P
Well, not necessarily; they could be floating point just as easily as they
could be integer (not that I don't still prefer seconds).
They numbers on the custom traffic report page are non-integer, for example:
http://www.lugnet.com/news/traffic/custom.cgi?hours=.5
http://www.lugnet.com/news/traffic/custom.cgi?hours=1.5&days=2.33333
--Todd
| | | | | | | | | | | | | | | | | | | |
Do we REALLY need the precision that seconds bring us? :-)
I'd vote for days or at least hours. Even for a non human readable API
like this one, go with data that's understandable to humans and let the
machines keep the books straight.
--
Larry Pieniazek larryp@novera.com http://my.voyager.net/lar
- - - Web Application Integration! http://www.novera.com
fund Lugnet(tm): http://www.ebates.com/ Member ref: lar, 1/2 $$ to
lugnet.
NOTE: I have left CTP, effective 18 June 99, and my CTP email
will not work after then. Please switch to my Novera ID.
| | | | | | | | | | | | | | | | | | | | | |
| |
|
In lugnet.admin.general, Larry Pieniazek <lar@voyager.net> writes:
> Do we REALLY need the precision that seconds bring us? :-)
>
> I'd vote for days or at least hours.
"Admiral, if we go by the book, like Lietennant Saavik, hours would seem
like days."[1]
> Even for a non human readable API like this one, go with data that's
> understandable to humans and let the machines keep the books straight.
Days are certainly human-readable and intuitive and okie-fine for relative
times expressed counting backward from "now," but what about absolute times?
What if you wanted to restrict a search between April 1, 1999 and April 7,
1999 inclusive?
Seconds since Days since
Time/Date stamp the epoch the epoch
-------------------------- ------------ --------------
April 1, 1999 00:00:00 GMT 922924800 10682
April 7, 1999 12:59:59 GMT 923529599 10688.99998843
April 8, 1999 00:00:00 GMT 923529600 10689
If the range is given as [a,b), then whole number days work fine. If the
range is given as [a,b], then fractional days like 10688.99998843 would be
needed. Or if you wanted to search from 6:00am to 6:00pm on some day, then
you'd also need fractional days.
In any case, is a number like 10682 really any more meaningful than
922924800? Sure, it's shorter...but that actually makes it more confusing
because it looks like something that's supposed to make sense but doesn't.
At least 922924800 doesn't look like it's supposed to make sense.
--Todd
[1] STII:TWOK ;-)
| | | | | | | | | | | | | | | | | | | | | | |
In lugnet.admin.general, Todd Lehman writes:
> STII:TWOK
Just the sound that was made as I released the bow string and the arrow dug
itself into the concrete wall behind the target...
Cheers,
- jsproat
| | | | | | | | | | | | | | | | | | | |
In article <37c311a6.412616938@lugnet.com>, Todd Lehman
<lehman@javanet.com> writes
>
> A syntax vis-à-vis URLs isn't yet defined, but how does this sound?--
<snip>
>
> http://www.lugnet.com/?q=jeremy+sproat&qt=-10519200&qs=2629800
>
> 4 months is about 10519200 seconds, and 1 month is about 2629800 seconds, so
> that query would mean "prefer articles from about four months ago, with a
> preferential spread of about a week or two either way."
>
> For clipping absolute time ranges, how do qt0= and qt1= sound?--
>
> http://www.lugnet.com/?q=jeremy+sproat&qt0=-1814400&qt1=-0
>
> with the sign of qt0-qt1 specifying the sort order, chronological or reverse
> chronological.
<snip>
> --Todd
Man, you *really* need to cut down on the coffee ;-)
--
Tony Priestman
| | | | | | | | | | | | | | | | | |
In lugnet.admin.general, Todd Lehman writes:
> [much useful stuff deleted]
> How about qt= for
> the target time and qs= for the sigma (provided that some other non-greek
> mnemonic can be identified for 's'):
How about a non-GEEK mnemonic? Oh, sorry, I forgot who I'm talking with (-:
> [much more deleted, including an intensely elaborate and feindishly clever way
for the server to talk to itself via HTTP EQUIV reload directives -- whew!]
- Robert
| | | | | | | | | | | | | | | | | |
| |
|
In lugnet.off-topic.fun, "Robert Munafo" <munafo@NOgcctech.SPAMcom> writes:
> In lugnet.admin.general, Todd Lehman writes:
> > [much useful stuff deleted]
> > How about qt= for
> > the target time and qs= for the sigma (provided that some other non-greek
> > mnemonic can be identified for 's'):
>
> How about a non-GEEK mnemonic? Oh, sorry, I forgot who I'm talking with (-:
LOL! Yes! :) It shouldda said non-g[r]eek or non-gr?eek then, eh? :-)
> > [much more deleted, including an intensely elaborate and feindishly clever
> > way for the server to talk to itself via HTTP EQUIV reload directives --
> > whew!]
Oh, not via the <META HTTP-EQUIV=...> directive, but via the 'Location:'
header right in the HTTP response. You can see an example of this in action
(sans a complex HTML form) if you try to pull up an old article URL:
http://www.lugnet.com/news/display.cgi?lugnet.off-topic.fun:2696
Note that it automatically redirects this (via the 'Location:' response) to
the new URL:
http://www.lugnet.com/off-topic/fun/?n=2696
without having to mess with sending back an intermediate HTML document.
--Todd
| | | | | | |
