Adjusting my upgrade list from above, minor tweaks only:

1) Decide/announce a window (ASAP, no time is good though)
2) Stop puppet on `ytterbium`, stop Gerrit processes
3) Do a last snapshot of the Gerrit database, in case we have to roll back
4) Apply puppet / dns changes for lead and CI ([[https://gerrit.wikimedia.org/r/#/c/298673/|298673]], [[https://gerrit.wikimedia.org/r/#/c/299007/|299007]])
5) Run puppet on `lead`, which will issue the schema upgrades to m2-master (will take about 4-5 minutes from testing)
6) Run puppet for zuul nodes, restart
7) Verify installation
8) Point DNS from ytterbium to `lead`

The big points is we can leave puppet on `lead` the whole time and don't need downtime for reindexing. We did most of the prep work so Gerrit 2.12.2 is already running on that host. The index will be slightly out of date, but it can be rebuilt in-place. Searches will be a little wonky for the 45m or so, but it's better than staying down. We'll need a zuul upgrade, but this should be mostly prepared and in place prior to doing it. @hashar: How close are we to having the new zuul ready? It should be mostly done right?

The whole process, ideally, should only take about 45 minutes tops, thanks to the extensive testing we did prior. Looking at the ganglia data, it looks like our lowest traffic times on weekdays are from about 01:00 to 10:00 UTC. Saturday and Sundays are mostly quiet other than the l10n bot updates at around 00:00UTC. So quietest time from my gathering is basically 01:00 to about 10:00 on Mondays, late Sunday night in the US. Basically let l10n bot do its thing and then start the window. The nice thing, in case things go totally wrong the European ops will be awake soon on a workday to help and take over.

But it shouldn't go totally wrong. Roll back plan if things go wrong (and I plan to be very aggressive in rolling back if we don't have full functionality back) is as follows:
* Stop puppet on lead, turn off gerrit
* Restore database to snapshat that we did prior to downtime
* Re-start puppet and gerrit on ytterbium
* Reassess and plan a new window after issues fixed

If this sounds good, I'd like to shoot for this coming Monday, July the 25th for our downtime (Sunday evening my time). I'm thinking 01:00-04:00UTC, that's 6-9pm locally.