To LUGNET HomepageTo LUGNET News HomepageTo LUGNET Guide Homepage
 Help on Searching
 
Post new message to lugnet.admin.generalOpen lugnet.admin.general in your NNTP NewsreaderTo LUGNET News Traffic PageSign In (Members)
 Administrative / General / 518
517  |  519
Subject: 
Router failure; system downtime details
Newsgroups: 
lugnet.admin.general
Date: 
Mon, 26 Oct 1998 08:37:54 GMT
Viewed: 
844 times
  
This evening at approximately 10:30 p.m. EST (25 Oct 98), pair.com experienced
a failure in a Cisco 7508 Router, resulting in most of its network (including
lugnet.com) going offline.  Details are available from Pair Networks at:

   http://support.pair.com/notices/

Pair Networks (pair.com) is the ISP where the lugnet.com server is co-located.



FYI, here is how we were affected by this network outage:

1.  Sometime between 10:30 and 10:35 p.m. all NNTP and HTTP connections were
    dropped (NNTP connections shown below):

        :
10/25 21:45 ''''''''''''''' |        |      |    |   |   |  |  | | | | | |    9
10/25 21:50 ''''''''''''    |        |      |    |   |   |  |  | | | | | |    7
10/25 21:55 ''''''''''''''' |        |      |    |   |   |  |  | | | | | |    9
10/25 22:00 ''''''''''''    |        |      |    |   |   |  |  | | | | | |    7
10/25 22:05 ''''''''''''''  |        |      |    |   |   |  |  | | | | | |    8
10/25 22:10 ''''''''''''''  |        |      |    |   |   |  |  | | | | | |    8
10/25 22:15 ''''''''''''''  |        |      |    |   |   |  |  | | | | | |    8
10/25 22:20 ''''''''''''''  |        |      |    |   |   |  |  | | | | | |    8
10/25 22:25 ''''''''''''''  |        |      |    |   |   |  |  | | | | | |    8
10/25 22:30 ''''''''''''''  |        |      |    |   |   |  |  | | | | | |    8
10/25 22:35                 |        |      |    |   |   |  |  | | | | | |    0
10/25 22:40                 |        |      |    |   |   |  |  | | | | | |    0
10/25 22:45                 |        |      |    |   |   |  |  | | | | | |    0
10/25 22:50                 |        |      |    |   |   |  |  | | | | | |    0
10/25 22:55                 |        |      |    |   |   |  |  | | | | | |    0
10/25 23:00                 |        |      |    |   |   |  |  | | | | | |    0
10/25 23:05                 |        |      |    |   |   |  |  | | | | | |    0
10/25 23:10                 |        |      |    |   |   |  |  | | | | | |    0
        :


2.  The server remained active doing miscellaneous chores during the network
    outage:

        :
10/25 22:27 '                   |           |       |     |     |   |   |  0.03
10/25 22:28 '                   |           |       |     |     |   |   |  0.01
10/25 22:29                     |           |       |     |     |   |   |  0.00
10/25 22:30                     |           |       |     |     |   |   |  0.00
10/25 22:31 ''                  |           |       |     |     |   |   |  0.06
10/25 22:32 ''                  |           |       |     |     |   |   |  0.08
10/25 22:33 '''                 |           |       |     |     |   |   |  0.11
10/25 22:34 '                   |           |       |     |     |   |   |  0.04
10/25 22:35 ''''                |           |       |     |     |   |   |  0.15
10/25 22:36 ''''                |           |       |     |     |   |   |  0.16
10/25 22:37 '''                 |           |       |     |     |   |   |  0.12
10/25 22:38 '                   |           |       |     |     |   |   |  0.04
10/25 22:39 '                   |           |       |     |     |   |   |  0.01
10/25 22:40                     |           |       |     |     |   |   |  0.00
10/25 22:41 ''''''              |           |       |     |     |   |   |  0.22
10/25 22:42 ''                  |           |       |     |     |   |   |  0.08
        :
10/26 01:39 '                   |           |       |     |     |   |   |  0.01
10/26 01:40                     |           |       |     |     |   |   |  0.00
10/26 01:41 ''                  |           |       |     |     |   |   |  0.06
10/26 01:42 '                   |           |       |     |     |   |   |  0.02
10/26 01:43 '                   |           |       |     |     |   |   |  0.01
10/26 01:44 '                   |           |       |     |     |   |   |  0.04
      reboot
10/26 01:47 ''                  |           |       |     |     |   |   |  0.08
10/26 01:48 ''                  |           |       |     |     |   |   |  0.06
10/26 01:49 '                   |           |       |     |     |   |   |  0.05
10/26 01:50 '                   |           |       |     |     |   |   |  0.04
        :


3.  The lugnet.com server was rebooted at approximately 1:45 a.m.  I haven't
    yet heard from Pair Support as to why the reboot was necessary (I'm sure
    they're quite busy tonight) but it's likely that it had something to do
    with reloading router tables, and also likely that they were able to
    perform an orderly system shutdown by hand, since the reboot occurred
    at the end of the outage rather than the beginning.


4.  NNTP connections resumed approximately shortly after the routing problem at
    Pair was fixed:

        :
10/26 01:25                 |        |      |    |   |   |  |  | | | | | |    0
10/26 01:30                 |        |      |    |   |   |  |  | | | | | |    0
10/26 01:35                 |        |      |    |   |   |  |  | | | | | |    0
10/26 01:40                 |        |      |    |   |   |  |  | | | | | |    0
10/26 01:50                 |        |      |    |   |   |  |  | | | | | |    0
10/26 01:55 ''              |        |      |    |   |   |  |  | | | | | |    1
10/26 02:00 ''''            |        |      |    |   |   |  |  | | | | | |    2
10/26 02:05 '''''''''''     |        |      |    |   |   |  |  | | | | | |    6
10/26 02:10 '''''''''''     |        |      |    |   |   |  |  | | | | | |    6
10/26 02:15 ''''''''        |        |      |    |   |   |  |  | | | | | |    4
10/26 02:20 ''''''''        |        |      |    |   |   |  |  | | | | | |    4
        :


5.  HTTP file transfers resumed at exactly 1:46:19.


6.  No data was lost.


--Todd Lehman



1 Message in This Thread:

Entire Thread on One Page:
Nested:  All | Brief | Compact | Dots
Linear:  All | Brief | Compact
    

Custom Search

©2005 LUGNET. All rights reserved. - hosted by steinbruch.info GbR