| |
Today, I got the 2GB of compressed log files representing Lugnet traffic
over the past approximately-a-year. After crunching for a while, realizing
that I needed to move everything to a faster server, and restarting there, I
discovered that what I *really* need is another gigabyte of RAM.
Oh, and also I found some interesting things about the traffic here.
First, some a few disclaimers. The logs on the site are combined for
news.lugnet.com, guide.lugnet.com, www.lugnet.com, etc. Unfortunately,
there's at this point no way to distinguish which was which. This isn't a
big deal for requests for something like a particular news article or a set
image, since that's pretty clearly from one virtual server or the other. But
it does mean we can't distinguish between the site's main front page and the
root pages of the other subsites. Also, the logs (very sensibly for
performance reasons) don't contain DNS information, so I had to fill that in
now, and it's possible that the addresses have changed underneath us.
Finally, Todd points out that remote sites are allowed to embed ldraw images
from the Lugnet server in their pages, so that skews the statistics a bit.
(Particularly, we can tell that Peeron is very popular -- hi Dan and
Jennifer!)
Anyway, that's plenty of ado. So without any further of it, here's some
summary data and random commentary.
General:
Average hits (that's images, pages, whatever) per day: 430,681
Distinct hosts served: 1,028,820
Data transfered: 715.26 gigabytes
Average data per day: 2.07 gigabytes
Average hits-per-second: 4.98
Weekly:
The busiest week was that of December 14th, with 467,505 page requests.
However, in general, the traffic is basically steady -- there's been
neither a massive decline nor noticeable growth.
Days and hours:
Weekdays are busier than weekends. 11 am to 4 pm EST are the busiest
times of the day, with 1 to 4 am being the slowest. The peak is about
64% busier than the slowest time -- and that "slow" time is still pretty
busy, still averaging 3.84 hits/second.
Big ISPs:
Big US cable companies comcast.net, rr.com, cox.net, and attbi.com
together represent about 9.8% of the traffic in bytes (and 9.5% of
hits). German ISP "t-dialin.net" (Deutsche Telekom) is 2.9%, Japanese
bbtec.net is 1.4%, and wanadoo.fr (France, of course) is 1.1%. Oh, and
our dear AOL is 3.2%.
Top-level domains:
On a broader scale, 26% of traffic (in bytes) came from .net addresses,
23% from .com, and 19% from unresolved addresses. These could be from
anywhere in the world -- .com and .net tend to be US, but are used by
many large international ISPs as well. Oh, and .edu is 2.0%. With those
out of the way, the top 10 country codes are:
6.2% .jp (Japan)
3.9% .nl (Netherlands)
2.3% .ca (Canada)
1.8% .de (Germany)
1.8% .fr (France)
1.6% .uk (United Kingdom)
1.6% .au (Australia)
1.2% .it (Italy)
1.1% .pl (Poland)
1.1% .dk (Denmark)
Referring sites:
By far, the biggest site here is www.peeron.com -- largely due to the
embedded images links above. After that, there's an huge number of
requests from Korean sites cafe.daum.net, dethhend.compuz.com, and
www.brickinside.com -- also embedded links. Anyone know anything about
these sites?
Google is surprisingly low on the list -- for most sites I've seen the
logs for recently (admittedly not my actual job for about 6 years now),
it's near the top.
Having logs where the image site was excluded would make this more
meaningful. I could run the report again with image files excluded,
but not before I buy more RAM. :)
Search Queries:
Top 20 Search requests that led to Lugnet, by number of requests:
49006 lego (duh!)
17848 lugnet (yup)
5461 www.lego.com (hmmm)
2783 legos (see below)
2781 lego.com (hmmm, take two)
2604 www.lugnet.com (URLs go in the location bar, kids)
2192 spy bot (interesting)
1823 lugnet.com (yet another one)
1530 lego pirates (arrh! everyone loves piracy!)
991 cool site (yeah, we should bring that back)
847 cool games (well, close)
701 black seas barracuda (google -> an old fibblesnork page)
647 lego com (sheesh)
617 lego creations (people like MOCs)
602 lego instructions (I have nothing interesting to say)
535 www.legoshop.com (dot dot dot)
527 lego games
521 cool lego site of the week
521 www lego com
504 site of the week
Lugnet doesn't show up in the top 100 search results for "legos".
However, it's clear that many people are searching using this, because
there's a lot of stuff like "pictures of legos" or "paradisa legos" or
"star wars legos" further down the list.
Browser and OS Wars:
Huh, looks like Microsoft is winning. Weird!
87.1% MSIE
MSIE/6 (65.1%)
MSIE/5 (21.4%)
1.7% Old Netscape (4 or older)
Netscape 4.x (1.6%)
5.4% Mozilla (including Netscape 6+)
Mozilla (2.9%)
Netscape 7 (2.1%)
1.1% Safari
1.0% Opera
And for operating systems:
91.1% MS Windows
Windows XP (38.5%)
Windows 98 (22.3%)
Windows 2000 (17.0%)
Windows ME ( 8.3%)
Windows NT ( 3.3%)
Windows 95 ( 1.2%)
4.1% Macintosh (Sorry, no breakdown available.)
1.2% Unix
Linux (0.9%)
File types:
29.00% of traffic is .jpg or .gif graphics.
Nothing else is really interesting in this category.
And there you have it.
--
Matthew Miller mattdm@mattdm.org <http://www.mattdm.org/>
Boston University Linux ------> <http://linux.bu.edu/>
|
|
| |
In lugnet.admin.general, Matthew Miller wrote:
> [...]
> Referring sites:
>
> By far, the biggest site here is www.peeron.com -- largely due to the
> embedded images links above. After that, there's an huge number of
> requests from Korean sites cafe.daum.net, dethhend.compuz.com, and
> www.brickinside.com -- also embedded links. Anyone know anything about
> these sites?
>
> Google is surprisingly low on the list -- for most sites I've seen the
> logs for recently (admittedly not my actual job for about 6 years now),
> it's near the top.
All very interesting statistics! Thanks for crunching the data.
I have always thought that Lugnet should allow Google (and perhaps other search
engines) to crawl the posts on Lugnet. Even though the data would perhaps be a
few weeks old, I think that having the extra search abilities would be both
beneficial, and draw more people to Lugnet. It would certainly give faster
answers for many 'old' questions.
> [...]
>
> Search Queries:
>
> Top 20 Search requests that led to Lugnet, by number of requests:
>
> 49006 lego (duh!)
> 17848 lugnet (yup)
> 5461 www.lego.com (hmmm)
> 2783 legos (see below)
> 2781 lego.com (hmmm, take two)
> 2604 www.lugnet.com (URLs go in the location bar, kids)
> 2192 spy bot (interesting)
> 1823 lugnet.com (yet another one)
> 1530 lego pirates (arrh! everyone loves piracy!)
> 991 cool site (yeah, we should bring that back)
> 847 cool games (well, close)
> 701 black seas barracuda (google -> an old fibblesnork page)
> 647 lego com (sheesh)
> 617 lego creations (people like MOCs)
> 602 lego instructions (I have nothing interesting to say)
> 535 www.legoshop.com (dot dot dot)
> 527 lego games
> 521 cool lego site of the week
> 521 www lego com
> 504 site of the week
>
> Lugnet doesn't show up in the top 100 search results for "legos".
> However, it's clear that many people are searching using this, because
> there's a lot of stuff like "pictures of legos" or "paradisa legos" or
> "star wars legos" further down the list.
However, searching for "Lego" on Google returns Lugnet as the fourth hit, right
after lego.com, lego.com/deu and legomindstorms.com. That's good! (The next few
are the Cool Lego Site of the Week, crynwr.com/lego-robotics/, and then the
BrickTestament!)
I guess the reason that "legos" doesn't return a Lugnet hit (until #102, in
which case it's refering to the legOS subgroup) is because by and large people
here know enough to not call the bricks "legos", and so if they make links to
Lugnet on their own home pages, they'll use a different word.
--
David Schilling
|
|
| |
On Sun, Feb 22, 2004 at 04:55:11AM +0000, Matthew Miller wrote:
> Finally, Todd points out that remote sites are allowed to embed ldraw
> images from the Lugnet server in their pages, so that skews the
> statistics a bit. (Particularly, we can tell that Peeron is very
> popular -- hi Dan and Jennifer!)
heh :) Though, I believe since around the time that LDraw.org moved to
be hosted on Peeron's server, we stopped using partref's pictures, and
started using LDraws - any referrer's from us to LUGNET after, say,
April 2003 or so are either people clicking from Peeron to the Guide, or
people clicking on news on the News Sidebar
http://peeron.com/newsticker.html
> General:
>
> Average hits (that's images, pages, whatever) per day: 430,681
Cool - do you have the stats for page views? Those are often useful to
compare with the hits.
> Distinct hosts served: 1,028,820
> Data transfered: 715.26 gigabytes
> Average data per day: 2.07 gigabytes
> Average hits-per-second: 4.98
Interesting stats!
> Referring sites:
>
> By far, the biggest site here is www.peeron.com -- largely due to
> the embedded images links above.
I am curious to see how many of those are after we stopped using
partsref images.
...
> And there you have it.
Very cool! Maybe we could set up a way to have the server generate and
report these stats monthly in admin.statistics? It's very interesting!
--
Dan Boger
dan@peeron.com
|
|
| |
Dan Boger <dan@peeron.com> wrote:
> Cool - do you have the stats for page views? Those are often useful to
> compare with the hits.
It's hard on Lugnet, since pages aren't all handily suffixed with .html.
But I think it's around 18.4% -- the majority is definitely graphics.
> > Distinct hosts served: 1,028,820
> > Data transfered: 715.26 gigabytes
> > Average data per day: 2.07 gigabytes
> > Average hits-per-second: 4.98
> Interesting stats!
I'd be curious to see the ones for peeron, if only because I'm in a web
server stats geek mode right now. :)
> > Referring sites:
> > By far, the biggest site here is www.peeron.com -- largely due to
> > the embedded images links above.
> I am curious to see how many of those are after we stopped using
> partsref images.
Yeah, me too. As a rough cut, I can run it again for just the last six
months.
> > And there you have it.
> Very cool! Maybe we could set up a way to have the server generate and
> report these stats monthly in admin.statistics? It's very interesting!
Right now, doing so would totally kill the Lugnet server (even using
trimmed historical data). But I'd like to see something like that sometime
in the future.
--
Matthew Miller mattdm@mattdm.org <http://www.mattdm.org/>
Boston University Linux ------> <http://linux.bu.edu/>
|
|
| |
In lugnet.admin.general, Matthew Miller wrote:
> Dan Boger <dan@peeron.com> wrote:
> > Cool - do you have the stats for page views? Those are often useful to
> > compare with the hits.
>
> It's hard on Lugnet, since pages aren't all handily suffixed with .html.
> But I think it's around 18.4% -- the majority is definitely graphics.
Depending on the software you use, I guess - I think that webalizer allows you
to specify what _isn't_ a page :)
> > > Distinct hosts served: 1,028,820
> > > Data transfered: 715.26 gigabytes
> > > Average data per day: 2.07 gigabytes
> > > Average hits-per-second: 4.98
> > Interesting stats!
>
> I'd be curious to see the ones for peeron, if only because I'm in a web
> server stats geek mode right now. :)
Always fun :) Here are the stats from end of March 2003 to the end of January
2004.
Average hits (that's images, pages, whatever) per day: 143,853
Average Pages (no images, JS, etc) per day: 39,362
Distinct hosts served: 388,651
Data transfered: 300.77 gigabytes
Average data per day: 824 megabytes
This is only for Peeron - not counting LDraw, or any of the other sites hosted
on Peeron. It also means this doesn't count all the ldraw generated images (and
hits) from inventory views.
Dan
|
|
| |
Thanks for posting these. Although I have absolutely no use for them I find
statistics facinating.
I had a brief look at the Korean pages you mentioned, all seem very professional
but of course completely unintelligable (even with the Korean fonts installed)!
Interestingly after a couple of random hyperlink clicks I found the Coke train
thats also here http://www.mocpages.com/moc.php/610. I couldn't find anything
that was recognisably from Lugnet
Tim
|
|
|
