The client doesn't necessarily have to advertise what it wants, but then if it doesn't tell us anything, we would have to be conservative and not serve formats like WEBP or SVG that aren't widely supported. When the client doesn't ask for a specific format, it would be wise to stick to JPG and PNG as the only options, as we do now.

I think that accept headers make the most sense, because the client can use them in two ways. Either provide us with a list of what it supports, and let our backend pick within that selection what we think is best. Or request a specific format.

In either case some knowledge of what we're capable of for a given file type is needed if you're constructing a thumbnail URL from scratch and want a specific output format. But that's the same in your proposal if you ask for a "vector" when the original is a JPG. That request doesn't make any sense. And if we serve a JPG in that case, the client that was expecting a "vector" or an SVG, if the format is asked explicitly, might not be able to process it if the data is sent to a component only designed to render SVGs.

I can't see how you intend to make WEBP and SVGsupport work without the client explicitly advertising its support for these formats. Asking for "bitmap" doesn't mean that you support WEBP, asking for "vector" doesn't mean that you support SVG. At the end of the day the client support list is fine-grained. Which is why I believe that the accept header is the best option, because it can give us a comprehensive list of the client's capabilities, based on which we can decide what makes the most sense for a given file.

Now on the subject of the client needing some knowledge of our original type -> thumbnail type mapping. Like I mentioned above it's unavoidable regardless of your level of abstraction about the kind of output is requested. The question is how does the client know that it can request an SVG for a given file? Or rather, that it can request just SVG and it will get what it wants. I think it's the same issue as the original's dimension, it has to be provided as metadata alongside the rest of the thumbnail information in the APIs that clients consume to construct thumbnail URLs. Think of the Media Viewer case again. Right now the original sized as passed as attributes, because that's metadata that the thumbnail URL (or the original's URL) doesn't tell you. If in the future thumbnail URLs are addressed by hash and not by filename, Media Viewer becomes "blind" to what kind of thumbnails it can request. Therefore it would make sense to advertise either the original's mime type as another attribute, or the list of possible output mime types, if we don't want the client to need to keep a mapping up to date between original mime types and possible thumbnail mime types we support.

Even when you consider the use case of constructing a thumbnail URL "from scratch". Right now you need to know the filename, presumably you got that from an API or from parsing some page. Granted that you know what thumbnail formats we support for each original type, that's enough to request a thumbnail, which might fail if you request something too big. When thumbnails are addressed by hash, the hash will become the new minimum. You can still request a thumbnail with just that, but we might give you a thumbnail format you don't expect. And you still have the original size issue. If you want to generate thumbnail URLs with certainty that it will work, you're going to want to get the original size alongside the hash. And if you know that you can't process some formats we might throw at you based on our list of possible output formats, you might as well get the list of possible output formats for that file while you're already getting the hash and original dimensions.

Of course the ideal case is a client that supports everything we do, but I think it would be very dangerous to make the default, when the client doesn't tell us what it supports/wants, to send what we think is the best format for a given file among all possible file formats. Our content is heavily skewed towards JPG and PNG: https://commons.wikimedia.org/wiki/Special:MediaStatistics More than 90% of our content is either of those. Meaning that if we serve SVG thumbnails for SVG files when the client doesn't tell us anything about what it wants, we're quite likely to make some clients fail 3% of the time, because people who built them didn't encounter SVGs when testing (because it's easy not to stumble upon them).

Which is why I think the default behavior when nothing special is requested is to stick to JPG & PNG. That creates a really safe environment where the client can ignore the metadata I've just mentioned (which mime types can this original be rendered to) and always get a JPG or PNG back, which should be universally supported. Should the client support more, it can still ignore metadata about the file at hand and advertise through the Accept header that it can treat SVGs. Then we start serving SVGs to it when appropriate. And the client can look at the mime type of the response and act accordingly. To me that's the most likely use case and it makes the extra metadata I mention in the previous paragraph less necessary. That's what browsers already do and it doesn't require prior knowledge about the URL you're requesting.

To sum up, I see two scenarios from our perspective:

- client doesn't tell us what format it supports of wants. We stick to serving only JPG and PNG. We can still make "judgment calls" about which we serve. For example if the client wants a low-quality thumbnail, PNG originals could be served as JPG.
- client gives us a list of formats it supports (can be just one). We do the judgment call based on that list, picking what we think is best for the file at hand, given the other thumbnail parameters (eg. quality), if any. Should we not support any of the request formats for that file, we serve an error (it's the same situation as requesting a thumbnail bigger than the original - the client asked for a conversion we won't do). The solution for a client to avoid that error is to either always include JPG & PNG in the supported list (we can document that) or fetch metadata about the file, ideally at the same time it currently gets the hash/filename, to know which mime types it can be thumbnailed into.