(In reply to christian from comment #28)
> Since this bug has been around for a while and has affected quite some
> people, I've been asked to give a short explanation of the root issue
> and what SSHD-330 does.
> 
> Gerrit uses Apache Mina's SSHD [1] as ssh server. When connecting to
> gerrit through ssh, this ssh server uses Java's own crypto/security
> implementations to negotiate session keys (i.e.: different for each
> connection attempt) with the client. Java's default provider yielded
> those session keys without leading zero bytes, and Apache Mina's SSHD
> relied on no leading zero bytes being present.
> 
> But at some point Java [2] changed behaviour and is no longer
> stripping leading zero bytes, but Apache Mina SSHD still relied on no
> leading zero bytes being present. Hence assumptions mismatched and
> caused the issue.
> 
> The Java we use at gerrit.wikimedia.org is recent enough to no longer
> strip leading zero bytes. So when connecting to our gerrit through
> ssh, either
> 
> * the negotiated session key starts with a non-zero byte, and
>   everything works nicely. This case happens most of the time.
> 
> * the negotiated session key starts with a zero byte. Then gerrit's
>   built-in Apache Mina SSHD falsely treats the key as if there were no
>   leading zero bytes and the connection setup with the client fails.
> 
> SSHD-330 adds stripping of leading zero bytes from the session key to
> Apache Mina SSHD and thereby fixes the issue we are seeing.
> 
> ------
> 
> There was recently some FUD around OpenSSL generated keys not being
> affected. That did not work for me, and I do not see in code how this
> would make a difference.
> 
> Also, there was some recent discussion around extracting the keys from
> the keystore to proper files. I did not get a chance to try that, but
> that could do the trick too ... indirectly.
> Because in order to get gerrit to use keys from separate files, one
> needs to install BouncyCastle libraries to gerrit. BouncyCastle will
> act as provider for the needed security/crypto functionality and
> get used instead of Java's default providers. As the BouncyCastle
> providers (for now) also strip the leading zero bytes, that could
> work out.
> 
> Regardless, having Apache Mina SSHD to strip leading zero bytes seems
> most reliable, so we backported the Apache Mina SSHD's upstream fix to
> the version used in our gerrit, and rebuilt gerrit using that custom
> built Apache Mina SSHD.
> 
> [1] https://mina.apache.org/sshd-project/
> [2] I know that OpenJDK versions up to
>   OpenJDK Runtime Environment (IcedTea7 2.2.1) (Gentoo build 1.7.0_05-b21)
> work and the default providers strip the leading zeros, while the ones from
>   OpenJDK Runtime Environment (IcedTea 2.4.7) (7u55-2.4.7-1ubuntu1~0.12.04.2)
> do not strip them.
> 
> 
> Thanks Krinkle for the pointer to SSHD-330!

And thank you for the analysis and the informative summary -- well done!