how to restrict access to directory and subdirs

to restrict access to multiple directories in nginx in one location entry do

...
location ~ /(dir1|dir2|dir3) {
   deny all;
   return 404;
}
...

It's because the "root" directive matches before the "deny" directive can be matched. Reverse the order of your directives and it should work:

...
location /testdir {
  deny all;
  return 404;
}
location ~* ^.+\.(jpg|txt)$ {
  root   /var/www/site;
}
...

To ensure that the testdir match is chosen instead of the jpg/txt match, use the following locations:

location ^~ /testdir {
  deny all;
  return 404;
}
location ~* ^.+\.(jpg|txt)$ {
  root   /var/www/site;
}

In your example, you have two types of locations. location /testdir is a prefix location, as it has no tilde (~) between location and /testdir.

location ~* ^.+\.(jpg|txt)$ is a regex location (a case-insensitive one, due to the * directly after the tilde). From the nginx documentation:

To find location matching a given request, nginx first checks locations defined using the prefix strings (prefix locations). Among them, the location with the longest matching prefix is selected and remembered. Then regular expressions are checked, in the order of their appearance in the configuration file. The search of regular expressions terminates on the first match, and the corresponding configuration is used. If no match with a regular expression is found then the configuration of the prefix location remembered earlier is used.

The problem here, is that your testdir location is being remembered, but then the jpg/txt location is selected during the regex stage, as it matches. The following note from the documentation is what I based my solution (given above) upon:

If the longest matching prefix location has the “^~” modifier then regular expressions are not checked.

location /foo {
    deny all;
    return 404;
}

This will give you a 403 all the time because of the deny all... When you want the server to give you a 404 just only return a 404... like so:

location /foo {
    return 404;
}