I'm looking for the fastest http server available for:
serving static content -- huge set of large images. Minimal features need, just as fast as possible.
dispatching dynamic content plugins -- think a web server that does on-the-fly watermarking or image transcoding. I'm looking for the fastest, lowest-overhead way of dispatching this.
Environment: linux or OS/X. any language acceptable.
nginx is the current favored choice. LigHTTPd still works fine, but is less actively developed nowadays. LiteSpeed is also a good choice, and may be the best if you want commercial support or a nicer GUI. All of these are very fast, raw speed will not be a meaningful competitive differentiator within this group of webservers.
Hmnn, a custom extension module to nginx is the lowest overhead option. But writing modules in C / C++ is seriously time-consuming. OP says "any language acceptable", well, if that is so then nginx with a C extension, or maybe Apache with a C extension to benefit from Apache's richer set of modules and documentation.
But realistically, who writes C code today for non-mass-market products? I would consider Python code with Tornado, or a similar event-driven webserver in a high-level language to be a better match.
NGINX http://nginx.net for requirement 1) static file serving at least.
SEE Why is Nginx so fast?
Choose between lighttpd and nginx for static content. Choose Apache with modules that do in-house fastcgi process management efficiently (mod_fcgid, mod_wsgi, mod_passenger) for generating the dynamic content.
One solution would be nginx + php-fpm + apc. It'll handle static content and it'll handle dynamic content pretty f'ing quickly, too.
It's also worth noting that nginx has plugins, as well, and you could always write a plugin that did the watermarking. Doing it that way is probably the fastest possible way, although not the simplest to build.
Here's an example module that uses libgd: http://wiki.nginx.org/NginxHttpImageFilterModule
There's no reason you couldn't modify it to spit out watermarked images instead...
My faster Web Server podium is :
1) Nginx
2) Lighttpd
3) Cherokee
Here some benchmark : http://www.cherokee-project.com/benchmarks.html
http://www.rkblog.rk.edu.pl/w/p/lighttpd-and-cherokee-benchmark/
http://www.rkblog.rk.edu.pl/w/p/pylons-benchmark-various-servers/
Cherokee is fast and simple to configure, for this maybe is the better choice.
Perhaps lighttpd, but I'm not sure about the watermark part.
If your talking about the fastest/easiest web server to setup, i'd point to HFS File Server . It doesn't support anything but static content, but its SOOOO easy to use.