How to limit destination IPs for nginx cache proxy

I'm trying to set up nginx as a cache proxy that can cache/serve pages only from one IP range (example: 31.29.100.0/24).

To set up the proxy I use this conf:

server {
    listen 8080;
    location / {
        resolver 127.0.0.1;
        proxy_pass $scheme://$http_host$uri$is_args$args;
        proxy_set_header X-Remote-Addr $proxy_add_x_forwarded_for;
    }
}

Now, this will serve as an open proxy, any destination client requires... Is there a way to limit the destination IPs?