Subject:
|
Re: News search function temporarily disabled
|
Newsgroups:
|
lugnet.admin.general
|
Date:
|
Sun, 17 Dec 2000 17:34:18 GMT
|
Viewed:
|
778 times
|
| |
|
|
In lugnet.admin.general, Todd Lehman writes:
> [...]
> As a second step, I'm going to install a monitoring log that records the
> elapsed time and CPU time of every dynamically generated webpage. From
> those results, the bottlenecks will stand out like a fistful of sore thumbs.
> (Fixing them is a different matter.)
> [...]
Certain bottlenecks are indeed standing out like sore thumbs. One big one is
the dynamic generation of the /shop/ pages on guide.lugnet.com, and another
is the dynamic generation of member-specific set lists. An even bigger one
is (was) the dynamic generation of member listings (people) such as:
http://www.lugnet.com/people/members/byfirstname.cgi
That was taking several seconds of CPU and in some cases 200+ seconds of
transmission time to people on slow modems. I can't do anything about the
transmission time without reducing the page size, but for now I changed it
to cache the content for faster page generation. Holding a large chunk of
data in memory for 200 elapsed-time seconds while generating a page is bad,
even if only a fraction of a CPU second is used.
Another bottleneck is that there seems to be a strong correlation between the
CPU time to generate a page for a LEGO set and the number of people who've
given comments or supplied personal data about the set. For example, this
page:
http://guide.lugnet.com/set/7130
consumes 3 times as many CPU cycles as this page:
http://guide.lugnet.com/set/1462
or about 50 times as many CPU cycles above and beyond the regular overhead
for a set page.
A quick gnuplot graph with number of people on the x-axis and CPU time on
the y-axis showed a nearly straight line correlation. That's good info for
me to know. It ought to be is a gradual logarithmic-looking curve rather
than a straight line.
This isn't anyone's fault[1] and it doesn't mean that people need to stop
or entering data -- it just means there's an identifiable inefficiency which
I need to find a way to eliminate. A bottleneck like this won't get better
with more RAM or a faster CPU -- it's a coding issue in need of algorithmic
adjustments rather than quantitative system tuning.
Some of these bottlenecks amount to a significant portion of the CPU-cycles
pie. For example, the member listings consume(d) significant CPU cycles but
aren't accessed anywhere near as often as the set guide pages; the bottleneck
in the set guide pages is small, but adds up to a lot of wasted CPU cycles
overall. Across all dynamically generated pages served, the single-set-view
guide pages together account for 23% and multi-set-view guide pages together
account for 18%. I estimate that one to two hours per day of CPU cycles are
currently being wasted due to this bottleneck alone (based on a desired
page-generation time of 0.2 seconds (average) per page).
The news homepage <http://news.lugnet.com/> is also a bottleneck, averaging
a very embarrassing 0.537 seconds of CPU cycles per invocation, but in the
end it actually amounts to only about 1000 seconds of CPU time per day, so
it's actually not a true bottleneck at its current rate of request (~2000
page views per day). It *could* turn out to be a bottleneck someday if the
usage went up significantly from where it is during peak hours, but for now
it's not a real issue.
The main LUGNET homepage <http://www.lugnet.com/> sees two to three times
as much traffic as the LUGNET News homepage but is also three times more
efficient in terms of memory and CPU cycles, so I'm not even remotely
worried about it yet. Everything else falls into group of page types.
The biggies for now are the set guide pages and, of course, bringing back
the news search functionality. I think I can live with the inefficiencies
of the set guide pages for a few more weeks, because their worst-case
performance is still deterministic. The worst-case performance of the
news-search function, however, was non-deterministic.
I think I know of a way to make the search much more memory-efficient and
later more time-efficient without radically ripping things apart. If a
miracle occurs, the news search function might be back sometime this week.
--Todd
[1] except mine, of course
|
|
Message has 1 Reply:
 | | Re: News search function temporarily disabled
|
| (...) How about breaking down the member pages into pages split by first letter of first or last name, and by country/state? I'm not sure if your "caching" means keeping the indexes by each sort but it would seem this need not be true dynamic (...) (23 years ago, 17-Dec-00, to lugnet.admin.general)
|
Message is in Reply To:
45 Messages in This Thread:
 
- Entire Thread on One Page:
- Nested:
All | Brief | Compact | Dots
Linear:
All | Brief | Compact
This Message and its Replies on One Page:
- Nested:
All | Brief | Compact | Dots
Linear:
All | Brief | Compact
|
|
|
|
