I'm working on migrating a site to another platform. We have a Python script to handle the import and we're considering using Varnish for speed. I want to make sure all existing URLs remain functional (in the form of 301 redirects) and I'm considering doing this directly in Varnish. We're talking about 3k+ URLs and we won't be able to catch this into a couple of regular expressions.
Ideally I would keep the list of redirects in a separate file, json, csv or xml but from a bit of research it seems there's no straightforward way to do this except for generating a huge .vcl file with elseifs with the import script. Is there a nicer way to do mass 301 redirects in Varnish?
No
There is no way to link Varnish to any kind of datastore like this. However, Varnish can cache your 301 responses, so you could have a simple backend that returns the 301 and then set a very long TTL on that in your VCL. It should be trivial to write a PHP / python / etc script that reads the data file and sends the 301.
Check this solution: https://www.varnish-cache.org/vmod/database-driven-rewrites and also this one: https://github.com/brandonwamboldt/libvmod-redis
This should do the trick for you.