Where is RewriteMap allowed?

This seems like a very simple question.

I tried putting a RewriteMap directive in an .htaccess file but the logs showed an alert:

RewriteMap not allowed here

Then I tried putting it in the VirtualHost declaration in the apache2/sites-available file, but reloading apache gives me:

RewriteMap not allowed here
...fail!

In both cases with RewriteEngine on.

Is this referring to the location of the actual map on the file system, or the RewriteMap directive?

Just in case it helps:

<VirtualHost *:80>
  ServerName localhost
  etc...
  <Directory /var/www/site>
  AllowOverride all
  Options Indexes FollowSymLinks MultiViews
  Order allow,deny
  Allow from 127.0.0.1
  RewriteEngine On
  RewriteMap map txt:/var/www/site/map.txt
  RewriteRule /path/[^/]+/([^/]+) ${map:$1} 
  etc...