As of May 29, 2012 - 12:15 am PDT
  • NIF
    • nif-apps[1-2] are up
    • blog and website are up

 

As of May 28, 2012 - 12:00 pm PDT
  • No significant change in status in the last 13 hours
  • EMAIL
    • Delays of up to 12 hours are still being seen on some messages
  • CAMERA
    • Oracle down
    • Portal down
  • CCDB 
    • Oracle down
    • iRODS down because Oracle is down
  • NIF
    • nif-apps[1-2] are down
    • blog and website are down
  • NITRC
    • emailed Butch asking for status, no reply as yet
  • WBC
    • Dave Little reported servers are down
As of May 27, 2012 - 11:00 pm PDT
  • All of our servers have been powered back on, but we are still working on problems with many of the services running on them. 
  • The network issues are mostly resolved, except for one that may be affecting CAMERA.
  • IRODS remains down
  • Oracle dbs are down 
     If you notice a service down please submit a Jira ticket.
As of May 27, 2012 - 1:00 pm PDT
  • Please be advised that the CRBS SysOps Team is short on staff due to the holiday weekend.  As a result, it may be several hours before everything is back online and functioning properly
  • We are currently investigating the possibility that there is a problem with one of the switches in one of our racks at SDSC
  • We are aware that many systems are still offline or unusable.  Please be patient until we post here that we have everything back online and need help with end-to-end testing
As of May 27, 2012 - 10:00 am PDT
  • Campus network issues affecting CRBS appear to have been resolved.  Email is being delivered, though it will take some time for it to catch up with all the messages that were queued up.  No mail should have been lost, but time will tell.
  • We are resuming our recovery efforts.  Many systems that depended on servers at NCMIR or Holly are still offline and we are working on getting them back online.
As of May 27, 2012 - 7:30 am PDT
  • There are campus network routing issues that are preventing access to many of our systems - for example, our CRBS Status Page, NCMIR email, CAMERA systems
  • When the network issues are resolved, we will resume our recovery efforts
    Many thanks to Sean Penticoff, Edmond Negado, and Brandon Carl for working through the night and early this morning to get us this far. 
As of May 27, 2012 - 5:00 am PDT
  • WORKING - some CRBS systems are now back online - for example, this wiki and our support page
As of May 27, 2012 - 1:30 am PDT
  • Power restored
As of May 26, 2012 - ~11:45 am PDT
  • Most of UCSD Main Campus loses electrical power
Additional Info on the outage itself can be found at:

     UCSD Status Page

     SDSC Status Page

  • No labels