Subject:
|
Re: News search function temporarily disabled
|
Newsgroups:
|
lugnet.admin.general
|
Date:
|
Mon, 18 Dec 2000 00:37:38 GMT
|
Viewed:
|
3892 times
|
| |
|
|
On Sun, Dec 17, 2000 at 11:21:23PM +0000, Frank Filz wrote:
> Todd Lehman wrote:
> > Certain bottlenecks are indeed standing out like sore thumbs. One big one is
> > the dynamic generation of the /shop/ pages on guide.lugnet.com, and another
> > is the dynamic generation of member-specific set lists. An even bigger one
> > is (was) the dynamic generation of member listings (people) such as:
> >
> > http://www.lugnet.com/people/members/byfirstname.cgi
> >
> > That was taking several seconds of CPU and in some cases 200+ seconds of
> > transmission time to people on slow modems. I can't do anything about the
> > transmission time without reducing the page size, but for now I changed it
> > to cache the content for faster page generation. Holding a large chunk of
> > data in memory for 200 elapsed-time seconds while generating a page is bad,
> > even if only a fraction of a CPU second is used.
>
> How about breaking down the member pages into pages split by first
> letter of first or last name, and by country/state? I'm not sure if your
> "caching" means keeping the indexes by each sort but it would seem this
> need not be true dynamic content. You don't get that many updates of
> that content.
I think it's more of a problem of having to have the page in memory (all
300k of it) while a slow client downloads it. Nothing much to do about
that except make the page smaller - perhaps move all the FONT FACE entries
to an all encompassing FONT tag - that should reduce the page size by about
10% or so...
> > Another bottleneck is that there seems to be a strong correlation between the
> > CPU time to generate a page for a LEGO set and the number of people who've
> > given comments or supplied personal data about the set. For example, this
> > page:
> >
> > http://guide.lugnet.com/set/7130
> >
> > consumes 3 times as many CPU cycles as this page:
> >
> > http://guide.lugnet.com/set/1462
> >
> > or about 50 times as many CPU cycles above and beyond the regular overhead
> > for a set page.
> >
> > A quick gnuplot graph with number of people on the x-axis and CPU time on
> > the y-axis showed a nearly straight line correlation. That's good info for
> > me to know. It ought to be is a gradual logarithmic-looking curve rather
> > than a straight line.
> >
> > This isn't anyone's fault[1] and it doesn't mean that people need to stop
> > or entering data -- it just means there's an identifiable inefficiency which
> > I need to find a way to eliminate. A bottleneck like this won't get better
> > with more RAM or a faster CPU -- it's a coding issue in need of algorithmic
> > adjustments rather than quantitative system tuning.
>
> How about making it optional if the personal data is included? Most
> times I don't need it if my purpose for looking up the set is just to
> see what the heck is that set. You can have a button on the bottom of
> the page which says "show comments and personal data".
also, this data isn't that dynamic anyway - couldn't it a static page be
autogenerated every time a member updates their entry?
> > The biggies for now are the set guide pages and, of course, bringing back
> > the news search functionality. I think I can live with the inefficiencies
> > of the set guide pages for a few more weeks, because their worst-case
> > performance is still deterministic. The worst-case performance of the
> > news-search function, however, was non-deterministic.
> >
> > I think I know of a way to make the search much more memory-efficient and
> > later more time-efficient without radically ripping things apart. If a
> > miracle occurs, the news search function might be back sometime this week.
>
> Can you at all conveniently reduce the size of the news search if you
> gave an option to "limit search to previous X posts" (where you pick one
> or perhaps a handful of values for X, which would preferably be by time
> span but could be by number [i.e. "limit search to the past week" would
> be preferable to "limit search to the past 2000 posts"). Such search
> constraints could certainly be useful and might constrain the search
> enough to provide a serious performance improvement.
I agree that have a "search X day back" is a good function... Also, I
don't know how the search is run anyway - if it's indexed beforehand, or
run on the full data every time?
--
Dan Boger / dan@peeron.com / www.peeron.com / ICQ: 1130750
<set:9265_1>: LEGO DACTA Roof Tiles (DACTA/SYSTEM/Supplemental), '98, 250 pcs
|
|
Message has 3 Replies:
 | | Re: News search function temporarily disabled
|
| (...) Splitting the list into a page for each letter (or perhaps groups of 2-4 letters) or country/state would cut the page size down dramatically was my idea. Almost every time I've gone to look at the list, I'm either looking for a specific (...) (26 years ago, 18-Dec-00, to lugnet.admin.general)
| | | Re: News search function temporarily disabled
|
| (...) What about using a style sheet? Would that help? I admit that I am not sure exactly what level of browser for the big two correctly supports style sheets (I ought to know this since I have been using them a lot in recent web development (...) (26 years ago, 18-Dec-00, to lugnet.admin.general)
|
Message is in Reply To:
| | Re: News search function temporarily disabled
|
| (...) How about breaking down the member pages into pages split by first letter of first or last name, and by country/state? I'm not sure if your "caching" means keeping the indexes by each sort but it would seem this need not be true dynamic (...) (26 years ago, 17-Dec-00, to lugnet.admin.general)
|
45 Messages in This Thread:
 
- Entire Thread on One Page:
- Nested:
All | Brief | Compact | Dots
Linear:
All | Brief | Compact
This Message and its Replies on One Page:
- Nested:
All | Brief | Compact | Dots
Linear:
All | Brief | Compact
|
|
|
|
