Project

General

Profile

Bug #1463

CDB is down

Added by Rogers, Chris about 9 years ago. Updated about 9 years ago.

Status:
Closed
Priority:
Urgent
Assignee:
Martyniak, Janusz
Category:
Server
Start date:
08 May 2014
Due date:
% Done:

100%

Estimated time:

Description

The CDB is down and has been since 17.00 last night.

http://cdb.mice.rl.ac.uk/cdb/

returns

Proxy Error

The proxy server received an invalid response from an upstream server.
The proxy server could not handle the request GET /cdb/.

Reason: Error reading from remote server

Apache/2.2.3 (Scientific Linux) Server at cdb.mice.rl.ac.uk Port 80

Henry is added as watcher because I believe he was mucking about in the MLCR CDB yesterday...

#1

Updated by Rogers, Chris about 9 years ago

Nb: I checked that heplnv156 and heplnm069 are up...

#2

Updated by Rogers, Chris about 9 years ago

  • Priority changed from Normal to Urgent
#3

Updated by Rogers, Chris about 9 years ago

  • Status changed from Open to Closed
  • % Done changed from 0 to 100

I did some digging with Janusz on the other end of the phone. The two machines in question are 69 and 156; 69 hosts postgres; 156 hosts tomcat and apache. apache makes a vhost which maps cdb.mice.rl.ac.uk to localhost port 4443. Note that there is a stopped tomcat instance on 69 which is the hot swap for the MLCR instance of tomcat.

  • checked that Tomcat was running by attempting to wget a wsdl on localhost; we were able to download a wsdl
  • checked that postgres was running by logging in as postgres and using pgsql command

So we figured that the problem was in the apache layer or some networking. Apache config files had not changed since Feb, other Apache services (e.g. redmine) were okay. I tried restarting Apache and this didn't help. I then went into /var/log/tomcat5/catalina.out and found a line like java.lang.OutOfMemoryError. Turns out tomcat had run out of memory. Not sure why I was able to wget a wsdl before. Anyway, restarted tomcat process and that fixed it.

Also available in: Atom PDF