Since this bug has been around for a while and has affected quite some
people, I've been asked to give a short explanation of the root issue
and what SSHD-330 does.

Gerrit uses Apache Mina's SSHD [1] as ssh server. When connecting to
gerrit through ssh, this ssh server uses Java's own crypto/security
implementations to negotiate session keys (i.e.: different for each
connection attempt) with the client. Java's default provider yielded
those session keys without leading zero bytes, and Apache Mina's SSHD
relied on no leading zero bytes being present.

But at some point Java [2] changed behaviour and is no longer
stripping leading zero bytes, but Apache Mina SSHD still relied on no
leading zero bytes being present. Hence assumptions mismatched and
caused the issue.

The Java we use at gerrit.wikimedia.org is recent enough to no longer
strip leading zero bytes. So when connecting to our gerrit through
ssh, either

* the negotiated session key starts with a non-zero byte, and
  everything works nicely. This case happens most of the time.

* the negotiated session key starts with a zero byte. Then gerrit's
  built-in Apache Mina SSHD falsely treats the key as if there were no
  leading zero bytes and the connection setup with the client fails.

SSHD-330 adds stripping of leading zero bytes from the session key to
Apache Mina SSHD and thereby fixes the issue we are seeing.

------

There was recently some FUD around OpenSSL generated keys not being
affected. That did not work for me, and I do not see in code how this
would make a difference.

Also, there was some recent discussion around extracting the keys from
the keystore to proper files. I did not get a chance to try that, but
that could do the trick too ... indirectly.
Because in order to get gerrit to use keys from separate files, one
needs to install BouncyCastle libraries to gerrit. BouncyCastle will
act as provider for the needed security/crypto functionality and
get used instead of Java's default providers. As the BouncyCastle
providers (for now) also strip the leading zero bytes, that could
work out.

Regardless, having Apache Mina SSHD to strip leading zero bytes seems
most reliable, so we backported the Apache Mina SSHD's upstream fix to
the version used in our gerrit, and rebuilt gerrit using that custom
built Apache Mina SSHD.

[1] https://mina.apache.org/sshd-project/
[2] I know that OpenJDK versions up to
  OpenJDK Runtime Environment (IcedTea7 2.2.1) (Gentoo build 1.7.0_05-b21)
work and the default providers strip the leading zeros, while the ones from
  OpenJDK Runtime Environment (IcedTea 2.4.7) (7u55-2.4.7-1ubuntu1~0.12.04.2)
do not strip them.


Thanks Krinkle for the pointer to SSHD-330!