Subject:
|
Re: Peeron.com crosses 1000 inventories posted!
|
Newsgroups:
|
lugnet.db.inv
|
Date:
|
Tue, 8 Jan 2002 20:37:03 GMT
|
Viewed:
|
910 times
|
| |
|
|
Dan Boger wrote:
> In lugnet.db.inv, Johannes Koehler wrote:
>
> > Unfortunately MOST German AFOLs can not use this database. And that's
> > really a shame because www.peeron.com is a very useful database.
>
>
> Unfortunatly, we had to block access to t-online.de after repeated
> spiders downloaded the whole site (ignoring robots.txt). We are limited
> in the bandwidth we can serve out a month, and didn't want to have the
> site go down to all, just because of some bad apples. When I get a
> chance, I'll try to put in a system where a user could log in, and get
> access anyway - that way, if a user misbehaves, I could block just that
> one user, instead of the whole domain.
Dan told me as much in a private mail exchange a few weeks ago.
What I'm doing is getting the set list (only the set list, promise; i'm
not spidering the whole site ;-) ) every few days through an anonimyzing
proxy and run that through my own scripts to set up my "private peeron",
but my system is nowhere as feature-rich as the original (for example,
I'm missing the wonderful custom pictures of peeron's). Due to work
overload, I haven't gotten to the point where I can automatically get
the newly added sets and build a look-alike of the real thing... So many
things to do, so few free hours per day...
By the way, Dan: you once said you could provide the "raw" of the
inventories. Could you create an FTP account on your machines and dump
the raw data there? Like what you are doing with the master list of
bricks? That way, we could get the data to feed our engines, and you
could still block the script kiddies with their point-and-click spiders.
If you provide access to the data, I would provide my set of scripts to
allow people to set up their local search engines (which would save you
still more bandwith).
If bandwith still starts to be a problem, maybe you could coordinate
something with lugnet or brickshelf; those sites doesn't seem to have as
much of a problem with spidering... (even though I heard of people
downloading the complete scans from from brickshelf and selling them on
CD....)
Regards,
Hakan
>
> Dan
>
|
|
Message has 2 Replies:
 | | Re: Peeron.com crosses 1000 inventories posted!
|
| (...) Hey, i'm following up myself once again ;-) Just in case someone thinks I'm just out to take advantage of the hard work of Jennifer and Dan, I tip my hat off to them for working up the energy to start the "experimental engine" and continuing (...) (23 years ago, 8-Jan-02, to lugnet.db.inv)
| | | Re: Peeron.com crosses 1000 inventories posted!
|
| Hi, (...) That's why I started contributing to peeron as well, I want everything in my collection inventoried. Then I can just look on peeron and see how many I have of piece X in a particular color. (...) Oh yes, that would be ideal. I would love (...) (23 years ago, 10-Jan-02, to lugnet.db.inv)
|
Message is in Reply To:
| | Re: Peeron.com crosses 1000 inventories posted!
|
| (...) Unfortunatly, we had to block access to t-online.de after repeated spiders downloaded the whole site (ignoring robots.txt). We are limited in the bandwidth we can serve out a month, and didn't want to have the site go down to all, just because (...) (23 years ago, 8-Jan-02, to lugnet.db.inv)
|
10 Messages in This Thread:
 
- Entire Thread on One Page:
- Nested:
All | Brief | Compact | Dots
Linear:
All | Brief | Compact
This Message and its Replies on One Page:
- Nested:
All | Brief | Compact | Dots
Linear:
All | Brief | Compact
|
|
|
|
