The two main implementation options are: a batcher internal to parsoid, and a shared node.js batcher service. The internal batcher would operate roughly as follows:

* Instead of having the token handlers directly dispatch HTTP requests, they would send their requests to a batcher service, which would be local to the current parse operation. The batcher API would be slightly different to the HTTP API, since it needs to be able to easily aggregate titles, and the batcher will take over responsibility for HTTP protocol details such as timeouts.
* If the batch is full, the request can be dispatched on the next attempted queue insertion.
* If Parsoid returns to the event loop with requests queued in the batch but not sent, the request could permanently hang. So in relevant code paths prior to returning to the event loop, Parsoid needs notify the batcher, giving it a last chance to check if there are any batch requests queued so that it can dispatch them.
* Additionally, a timeout would be prudent, allowing Parsoid to recover from accidental return to the event loop without notification.

Gabriel suggested an external batcher which, if I understand his proposal correctly, would operate as follows:

* One cluster-wide batch queue per (wiki, action) tuple. 
* Requests to the proxy would be held in the batch queue until either it fills, or some specified timeout (e.g. 10ms) is reached. Then the batch query would be sent to the MW backend.
* While a request is waiting in the batch queue, and while it is waiting for the backend response, the client connections, one per item, would be held open. Responses would be sent to these clients after the batch query returns from the backend.
* This architecture avoids the need for significant changes inside Parsoid.

The main reason I am leaning towards the internal batcher is because some of the queues will have very little traffic on them, e.g. queues for small wikis. This makes for some uncomfortable tradeoffs when you try to set the dispatch timeout. Consider that parse operations are observed to take ~10s and issue ~100 API requests -- this implies a mean interval between requests on the order of 100ms, and a timeout of at least that much if there is to be any throughput benefit from batching at all. That is quite a long time to wait. From a throughput perspective, the optimal dispatch timeout may be more than 1s.

Explicit notification of the idle event avoids severe tradeoffs between batch size and latency. It can be implemented in either the internal or external design. But the need for idle event notification would reduce the attractiveness of the external batcher -- it would make the service stateful and its advantages in simplicity on the Parsoid side would be reduced. There is a risk that idle events could be lost anyway, and a timeout incurred, due to exhaustion of the client connection limit (globalAgent.maxSockets).

Also, note that Parsoid currently performs API requests with keep-alive disabled. With default socket options, this implies a limit of 470 API requests per second per client server before local port exhaustion occurs, as in T75949. Batching internally, without the involvement of TCP, avoids this limit.