For the context:

Zuul is a python daemon running on gallium. It establish a ssh connection as jenkins-bot on manganese.wikimedia.org and runs the 'gerrit stream-events' command.   It expect JSON object to be returned.

Looking at the Zuul log (fgrep GerritWatcher /var/log/zuul/zuul.log), it does emit a stacktrace every 10 minutes:


2013-04-08 05:50:48,028 ERROR gerrit.GerritWatcher: Exception on ssh event stream:
2013-04-08 06:00:53,176 ERROR gerrit.GerritWatcher: Exception on ssh event stream:
2013-04-08 06:10:58,266 ERROR gerrit.GerritWatcher: Exception on ssh event stream:
2013-04-08 06:21:03,383 ERROR gerrit.GerritWatcher: Exception on ssh event stream:
2013-04-08 06:31:08,472 ERROR gerrit.GerritWatcher: Exception on ssh event stream:
2013-04-08 06:41:13,594 ERROR gerrit.GerritWatcher: Exception on ssh event stream:
2013-04-08 06:51:18,687 ERROR gerrit.GerritWatcher: Exception on ssh event stream:
2013-04-08 07:01:23,782 ERROR gerrit.GerritWatcher: Exception on ssh event stream:

A full trace is:


2013-04-08 00:12:10,016 ERROR gerrit.GerritWatcher: Exception on ssh event stream:
Traceback (most recent call last):
  File "/usr/local/lib/python2.7/dist-packages/zuul-0.0.151.e9d929a-py2.7.egg/zuul/lib/gerrit.py", line 68, in _run
    self._listen(stdout, stderr)
  File "/usr/local/lib/python2.7/dist-packages/zuul-0.0.151.e9d929a-py2.7.egg/zuul/lib/gerrit.py", line 52, in _listen
    self._read(stdout)
  File "/usr/local/lib/python2.7/dist-packages/zuul-0.0.151.e9d929a-py2.7.egg/zuul/lib/gerrit.py", line 39, in _read
    data = json.loads(l)
  File "/usr/lib/python2.7/json/__init__.py", line 326, in loads
    return _default_decoder.decode(s)
  File "/usr/lib/python2.7/json/decoder.py", line 366, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File "/usr/lib/python2.7/json/decoder.py", line 384, in raw_decode
    raise ValueError("No JSON object could be decoded")
ValueError: No JSON object could be decoded


Which is stream-events sending some invalid JSON, most probably something like "Connection timed out after 10 minutes".    It seems Zuul properly reconnect the ssh connection.


Whenever the stream-events for Zuul, it is also blocked for any other user even on a fresh new connection.  For example, I could run the stream-events command using my own account, fill a comment in Gerrit and I get no event on the channel.