 | | News search function temporarily disabled
|
|
The system crashed again. Either it's running out of memory and that's causing a downward spiral or it's running out of CPU cycles and starving enough processes to build up and cause a meltdown. Either way some tuning needs to be done. This is going (...) (23 years ago, 11-Dec-00, to lugnet.admin.general, lugnet.off-topic.geek, lugnet.announce) !
|
| |
| | Re: News search function temporarily disabled
|
|
<nod nod> Thanks for letting us know. I'm no geek in any means, but first things first, I would limit the search. Sometimes a single search brings up thousands of posts. I suggest if the search is too broad, say it and make the searcher narrow it (...) (23 years ago, 13-Dec-00, to lugnet.admin.general, lugnet.off-topic.geek)
|
| |
| | Re: News search function temporarily disabled
|
|
(...) Hi Todd, Did you ever consider a google search box? Like this one: www.sis.pitt.edu/~dist I don't know for sure, but I think that the search is done on google's server. Toki (23 years ago, 17-Dec-00, to lugnet.admin.general, lugnet.off-topic.geek)
|
| |
| | Re: News search function temporarily disabled
|
|
(...) That's an intriguing thought. It looks like it requires Google to be able to crawl a site completely, though. I wonder if they play well with dynamically generated content... (...) Yes, it is. --Todd (23 years ago, 17-Dec-00, to lugnet.admin.general, lugnet.off-topic.geek)
|
| |
| | Re: News search function temporarily disabled
|
|
(...) I've had nothing but problems with www.hort.net/gallery/, which is all dynamically generated. Every month at about the same time we disappear entirely out of Google's search engine, which they claim is due to updates. Although it usually only (...) (23 years ago, 17-Dec-00, to lugnet.admin.general, lugnet.off-topic.geek)
|
| |
| | Re: News search function temporarily disabled
|
|
(...) Certain bottlenecks are indeed standing out like sore thumbs. One big one is the dynamic generation of the /shop/ pages on guide.lugnet.com, and another is the dynamic generation of member-specific set lists. An even bigger one is (was) the (...) (23 years ago, 17-Dec-00, to lugnet.admin.general)
|
| |
| | Re: News search function temporarily disabled
|
|
(...) How about breaking down the member pages into pages split by first letter of first or last name, and by country/state? I'm not sure if your "caching" means keeping the indexes by each sort but it would seem this need not be true dynamic (...) (23 years ago, 17-Dec-00, to lugnet.admin.general)
|
| |
| | Re: News search function temporarily disabled
|
|
(...) I think it's more of a problem of having to have the page in memory (all 300k of it) while a slow client downloads it. Nothing much to do about that except make the page smaller - perhaps move all the FONT FACE entries to an all encompassing (...) (23 years ago, 18-Dec-00, to lugnet.admin.general)
|
| |
| | Re: News search function temporarily disabled
|
|
(...) It's (currently) a half-gigabyte index that gets indexed in realtime (once per minute). --Todd (23 years ago, 18-Dec-00, to lugnet.admin.general)
|
| |
| | Re: News search function temporarily disabled
|
|
(...) Splitting the list into a page for each letter (or perhaps groups of 2-4 letters) or country/state would cut the page size down dramatically was my idea. Almost every time I've gone to look at the list, I'm either looking for a specific (...) (23 years ago, 18-Dec-00, to lugnet.admin.general)
|
| |
| | Re: News search function temporarily disabled
|
|
(...) How long does it take to generate the index? Can you subdivide the index at all (and do something like backup where you have incremental indices plus once a day or once an hour regenerate the complete index)? If some sort of incremental index (...) (23 years ago, 18-Dec-00, to lugnet.admin.general)
|
| |
| | Re: News search function temporarily disabled
|
|
(...) What about using a style sheet? Would that help? I admit that I am not sure exactly what level of browser for the big two correctly supports style sheets (I ought to know this since I have been using them a lot in recent web development (...) (23 years ago, 18-Dec-00, to lugnet.admin.general)
|
| |
| | Re: News search function temporarily disabled
|
|
(...) Zero time. It's done continuously as a background process. Once per minute, any new article is added into the mix. (...) This kind of index is more efficient to do as soon as something new appears, as opposed to a kind of index where content (...) (23 years ago, 18-Dec-00, to lugnet.admin.general)
|
| |
| | Re: News search function temporarily disabled
|
|
(...) Yikes! Holy speculation, Batman! Thanks, but no thanks! Guys, this is nice and all, but you're guessing way off in left field. On the pages I was talking about, the actual webpage is not held in memory -- ever -- it's written directly to a (...) (23 years ago, 18-Dec-00, to lugnet.admin.general)
|
| |
| | Re: News search function temporarily disabled
|
|
(...) No need to get snippy. We're just trying to help. We don't have access to the source or even the design documentation so all we can do is speculate based on what you say, and if our speculation is wide of the mark, so be it. It's not our (...) (23 years ago, 18-Dec-00, to lugnet.admin.general)
|
| |
| | Re: News search function temporarily disabled
|
|
(...) I'm sorry. I know. You're right. It just always happens. Someone says something vague, someone else guesses something about it, someone else adds another guess, someone else adds another guess, and before you know it, everything's way off (...) (23 years ago, 18-Dec-00, to lugnet.admin.general)
|
| |
| | Re: News search function temporarily disabled
|
|
(...) Sorry for having contributed to wasted bandwidth, though I do hope you will give some thought to the ideas of splitting up the members list as I had suggested, and some kind of "recentness" limit to the search, though I realize that the (...) (23 years ago, 18-Dec-00, to lugnet.admin.general)
|
| |
| | Re: News search function temporarily disabled
|
|
(...) Yes, definitely! And the flags can link to country pages that list just the people in the country. (...) It actually won't incur extra overhead (amazingly, it will actually reduce overhead) to restrict things to hard or soft time ranges, so (...) (23 years ago, 18-Dec-00, to lugnet.admin.general)
|
| |
| | News search function reactivated (was: News search function temporarily disabled)
|
|
The LUGNET News search function is now re-enabled. I completely revamped the index data structures and list-merge algorithm and rewrote the core query engine in C. It's a much more solid implementation. Everyone's patience during the outage is much (...) (23 years ago, 2-Jan-01, to lugnet.admin.general, lugnet.off-topic.geek, lugnet.announce) !!
|
| |
| | Re: News search function reactivated (was: News search function temporarily disabled)
|
|
(...) Very good! Although the new search doesn't return most recent articles first like it used to. Is that how it should work? Now I can't see most recent posts that contain the keyword I want to search for, which makes the search pretty much (...) (23 years ago, 2-Jan-01, to lugnet.admin.general, lugnet.off-topic.geek)
|
| |
| | Re: News search function reactivated (was: News search function temporarily disabled)
|
|
(...) Could I suggest some amendments to the "To do someday" list (for an 'advanced search' only)? - Search by author - Search by subject line contents - Search by date range (or open-ended-- i.e. after date X or before date X) - Search for articles (...) (23 years ago, 2-Jan-01, to lugnet.admin.general)
|
| |
| | Re: News search function reactivated (was: News search function temporarily disabled)
|
|
(...) Could you tell us the URL syntax for those of us willing to modify URLs? (23 years ago, 2-Jan-01, to lugnet.admin.general, lugnet.off-topic.geek)
|
| |
| | Re: News search function reactivated (was: News search function temporarily disabled)
|
|
(...) Not yet, no. See "to-do soon" section in previous post. (...) For now, if you don't mind URL mucking, you can manually append &qs=<number> to the URL and it will use that number (in seconds) as a time delta. For example, to limit posts to the (...) (23 years ago, 2-Jan-01, to lugnet.admin.general, lugnet.off-topic.geek)
|
| |
| | Re: News search function reactivated (was: News search function temporarily disabled)
|
|
(...) <snip> (...) I would have to concur. (...) < goes away and tries it... > Umm - Would it not make sense to simply include the appropriate qualifier on the system side? (I tried it and got two year old results for "qs", but I'm probably doing (...) (23 years ago, 2-Jan-01, to lugnet.admin.general, lugnet.off-topic.geek)
|
| |
| | Re: News search function reactivated (was: News search function temporarily disabled)
|
|
(...) Sure! (...) These are mostly covered already due to the way the indexing works -- words closer to the beginning of a document are given higher weights than words occurring later in a document. When the indexer chews on a news article, it first (...) (23 years ago, 2-Jan-01, to lugnet.admin.general)
|
| |
| | Re: News search function reactivated (was: News search function temporarily disabled)
|
|
(...) Pretty much... actually, I was thinking more along the lines of an advanced search form though: Search for: ___...___ (uses +'s and -'s as is) Search for text in subject line [] (checkbox) Posted by: ___...___ (uses +'s and -'s... or no (...) (23 years ago, 2-Jan-01, to lugnet.admin.general)
|
| |
| | Re: News search function reactivated (was: News search function temporarily disabled)
|
|
(...) One thing which would generally make it pretty easy to find URLs is to index "http". When dealing with special characters, definitely treat "/" and "\" as word separators. Probably ":" also. (23 years ago, 2-Jan-01, to lugnet.admin.general)
|
| |
| | Re: News search function reactivated (was: News search function temporarily disabled)
|
|
(...) Do you index the name in "X-real-life-name"? One thought, index the special strings "from:" and "subject:". The the search: from: ffilz Should rank my posts highly due to proximity. Of course it would be better to index the real life name as (...) (23 years ago, 2-Jan-01, to lugnet.admin.general)
|
| |
| | RE: News search function reactivated (was: News search function temporarily disabled)
|
|
(...) I think a multi-field advanced search is the way to go...much easier to use, IMO--something like the DejaNews power search: (URL) actual search keywords are in one field, and then there are numerouse ways to limit the search. --Bram Bram (...) (23 years ago, 2-Jan-01, to lugnet.admin.general)
|
| |
| | Re: News search function reactivated (was: News search function temporarily disabled)
|
|
(...) Ya, let's see...as it assembles the text to index, first it grabs X-Real-Life-Name:, then it grabs either Original-From: or From:, then Subject:, then Keywords:, then Summary:, and then finally the non-quoted and non-sig parts of the body. So, (...) (23 years ago, 3-Jan-01, to lugnet.admin.general)
|
| |
| | Re: News search function reactivated (was: News search function temporarily disabled)
|
|
(...) Ya, something like that'd be good to slap on top after the base functionality. :-) Nobody wants to *have* to remember how all the squiggly and square brace thingums in a search box work. :-) (...) Ya, precisely. It was originally (summer of (...) (23 years ago, 3-Jan-01, to lugnet.admin.general)
|
| |
| | Article bit-flags (was: Re: News search function reactivated)
|
|
(...) Oh, one other thing...planning ahead: Another potential application of article bit-flags is read/unread lists on a person-by-person basis via the web interface. I know this is something that people have been asking for for a long time. When (...) (23 years ago, 3-Jan-01, to lugnet.admin.general)
|
| |
| | Re: Article bit-flags (was: Re: News search function reactivated)
|
|
(...) Ooh ooh ooh... One thing I really really want is to be able to put messages into folders (if anyone knows of a decent newsreader which allows such - please let me know - it would be preferable that it do so without requiring me to store the (...) (23 years ago, 3-Jan-01, to lugnet.admin.general)
|
| |
| | Re: News search function reactivated (was: News search function temporarily disabled)
|
|
(...) It works great! ... but the &qs doesn't carry over to the next page of results. So if I want to see more pages, I have to edit the querystring on each page. Since you already have the inner workings of this in place, it would be really easy to (...) (23 years ago, 3-Jan-01, to lugnet.admin.general, lugnet.off-topic.geek)
|
| |
| | Re: News search function reactivated (was: News search function temporarily disabled)
|
|
(...) oops, doy! I didn't put in the propagation of that URL term. I don't consider it 100% "documented" yet (it's still subject to change without notice), but I still shouldn't have missed that. Thanks. I'll fix that. The reason it's subject to (...) (23 years ago, 3-Jan-01, to lugnet.admin.general, lugnet.off-topic.geek)
|
| |
| | Re: News search function reactivated (was: News search function temporarily disabled)
|
|
(...) Wow! So you have terms for the ampersand options in a URL? My standpoint on this would be to put everything in a form and kill 2 birds with 1 stone - not having to think of how to name URL terms (unless you enjoy doing that) and having the (...) (23 years ago, 3-Jan-01, to lugnet.admin.general, lugnet.off-topic.geek)
|
| |
| | Re: News search function reactivated (was: News search function temporarily disabled)
|
|
(...) Ya, exactly -- first name the URL components carefully and then put a user- friendly level on top of it. Best of both worlds. (...) Ah, I see. Yeah, that could be helpful in certain cases, if you're scouring tons of results! I've needed to (...) (23 years ago, 3-Jan-01, to lugnet.admin.general, lugnet.off-topic.geek)
|
| |
| | Re: News search function reactivated (was: News search function temporarily disabled)
|
|
(...) As an algorithmical guess, I think I'd probably attempt something a bit different... If someone enters: the I'd probably want to ignore it. But if they entered: the best design I might want to consider the 'the'. Dunno. I'd probably test an (...) (23 years ago, 3-Jan-01, to lugnet.admin.general)
|
| |
| | Re: News search function reactivated (was: News search function temporarily disabled)
|
|
(...) All geeks capitulate sooner or later on perl vs C. Of course Larry (and many others I work with) would tell you to write that stuff in Java but that would be a step backwards. Congratulations! Of course it's stability depends greatly on your (...) (23 years ago, 3-Jan-01, to lugnet.off-topic.geek)
|
| |
| | Re: News search function reactivated (was: News search function temporarily disabled)
|
|
(...) I like Java. But this really needed to be close to the metal and generate code that would fit in the L1 cache for the non-memory-bus-bound portions of the loops. The GNU C compiler is incredible. (...) It's the best C code I've written in 12 (...) (23 years ago, 3-Jan-01, to lugnet.off-topic.geek)
|
| |
| | Re: News search function reactivated (was: News search function temporarily disabled)
|
|
(...) Aah, you definately couldn't have done that in Java. Of course the ability to declare a couple register variables helps too. KL (23 years ago, 3-Jan-01, to lugnet.off-topic.geek)
|
| |
| | Re: News search function reactivated (was: News search function temporarily disabled)
|
|
(...) Of course I've heard of situations where an interpreter outdid hand crafted assembler. This can occur if the portion of the interpreter necessary to run your code fits in the code cache and the byte codes fit in the data cache when the hand (...) (23 years ago, 3-Jan-01, to lugnet.off-topic.geek)
|
| |
| | Re: News search function reactivated (was: News search function temporarily disabled)
|
|
(...) I probably couldn't, no, but a very experienced Java programmer and a good JVM machine could conceivably do better than C. (It's not unheard of for Java to be faster than C for certain types of things.) The big hits would probably be the JVM (...) (23 years ago, 3-Jan-01, to lugnet.off-topic.geek)
|
| |
| | Re: News search function reactivated (was: News search function temporarily disabled)
|
|
(...) Oh -- actually, what search engines typically do on queries (and I just finally added this last week) is downvalue relatively common words and upvalue relatively uncommon words -- what's called "term ranking" or "term weighting." For example, (...) (23 years ago, 4-Jan-01, to lugnet.admin.general)
|
| |
| | Re: News search function reactivated (was: News search function temporarily disabled)
|
|
(...) Todd, this would be really useful. I'll often search for a recent post, only remembering the poster's name and maybe one or two key-words, and that the post was in the past few days. I don't need two year old messages nearly as frequently. (...) (23 years ago, 5-Feb-01, to lugnet.admin.general)
|
