The System Is Down [Updated]
Yesterday two systems went down: the hours page and our production server. The hours page is not allowing us to update the current hours, and as a result, is only showing hours for two locations: Mary I and Steelcase. Institutional Marketing is aware of the problem and working on it. This has been updated
As soon as we determined that this would not be a quick fix, I went to our Status app to post an issue with the hours page, but the Status page was unavailable. Whoops.
How do you post the status of the status app when it’s down?
Trying to visit any tool on our production server resulted in the error, so at first I thought our production server was down. Then I realized that I was connected to the server through ssh, doing some work with git, our version control software. How could I be connected to the server if it was down? That’s when I realized that on the web I use a different URL to connect than over ssh.
In an email that morning, Mary had let me know that our production server (hosted by Dreamhost) was no longer using a fixed IP address. The production server on Dreamhost lives at gvsuliblabs.com, but we use a GVSU-supplied domain for labs.library.gvsu.edu. It turns out that when Dreamhost switched how our IP address was configured, it revealed a conflict in how the labs.library.gvsu.edu domain was configured. Basically, we can access the production server directly, through gvsuliblabs.com but not through the gvsu.edu domain.
The downside? Every tool and script we have is linked using labs.library.gvsu.edu. Here is what is currently not working: This has been updated
- Custom scripts and styles for Summon 2.0
- 360Link Reset script for the Link Resolver
- All JavaScript and custom styles for the catalog
- The Custom Scoped Summon Search tool
- The UI Pattern Library
- The Sustainability Game
- Our custom CAS login gateway (which routes using labs.library.gvsu.edu, so I still can’t log in to the status app, etc.)
Mary and I are working with Dreamhost and IT to resolve this issue, and hope to have it up and going shortly. (Mary was up late last night trading emails with folks, so she deserves a medal.) I’ll post updates here and on the status app (once I can access it again).
We’ve been fortunate these past few years to cobble together a system for hosting our custom tools that has been fairly reliable. We don’t have on-campus support for infrastructure for our web projects, so we looked elsewhere and created development ad production environments our of hosted tools. Someday I hope we’ll have in-house infrastructure so these types of things don’t happen (or at least get dealt with quickly, since we don’t have to email a bunch of different vendors to get something straightened out). For now, we’ll be patient and keep working on it. Thanks for being patient!
If you want to know what goes through my head every time we have a server down, you can watch it here.
[Update 10:21am] The hours page is functioning normally again. We’re not really sure what was wrong, but there appeared to be some corrupted data in the database table. Once we removed that, everything went back to normal.
[Update 10:34am] I’ve updated the paths in Summon, the catalog, and the link resolver to pull resources from gvsuliblabs.com rather than labs.library.gvsu.edu. Summon and the catalog are working now, but 360Link takes 24 hours to make changes live. I’ll keep you up-to-date on the status of the Great URL debacle of 2014 as things come up!
[Update 5/15/2014] With the help of GVSU’s IT department, we were able to fix the problems with our CNAME for labs.library.gvsu.edu. All of our systems are back up and running.