Apache: Add header to one specific url?

I just configured some Content negotiation for my root url based on this Apache Document and it seems to work great.

As my site also set a language cookie, I tested the following in my virtualhost (also mentionned in the doc)

  SetEnvIf Cookie "lang=([^;]+)" prefer-language=$1
  Header append Vary cookie

But then, As I only have one URL with negotiated content, I don't want to send a Vary: Cookie for all requests. It would prevent all caching to be efficient as a sessionid is also stored in a cookie (correct me if i'm wrong).

So my question is how can I only append the Vary header for this specific url.

I already tried to specified the directives in a Location Directory or Files section, but none seemed to work.

<Location "/">
  Header append Vary cookie
</Location>  

The vary header is sent for / but also for /eng/index.html and all requests

<Directory /my/htdocs>
  Header append Vary cookie
</Directory>

The vary header is not sent neither for / nor for any requests

<Files "/index.html">
  Header append Vary cookie
</Files>

The vary header is not sent neither for / nor for any requests