user3758232's questions

It is my understanding, from looking at the Varnish pipeline chart, that vcl_pipe can be only called from vcl_recv.

I am setting up a VCL to serve binary resources from a back end contingent on some conditions, including access control.

The way I thought of setting this up so far is thus: go through all my checks, which include moving out of vcl_recv; if the conditions are set for streaming the resource, set a flag; restart the transaction; check the flag first thing in vcl_recv, and if set, call vcl_pipe.

The only, major, issue is that I don't know how to set a flag (e.g. in http.request) that could not be forged by a request header. So a caller who knows my VCL could easily set the pipe flag and bypass all checks.

The other option is to keep all checks in vcl_recv but that may be clunky or even impossible.

Suggestions will be appreciated. Thanks.

I am developing a Varnish pipeline that serves a mix of public and restricted resources.

Since access to public resources makes up the vast majority (>99.9%) of the traffic, I want to create shortcuts to bypass auth token validation and other such things for non-restricted resources—or better yet, only go through the authN/Z path if the resource is in a sort of blacklist.

This blacklist can contain up to about 1M (within the next few years) UUID4 entries. Such a file in plain text occupies about 3.7Gb on disk, so a machine with good RAM capacity should be able to keep it all in memory.

My question is about how to implement this blacklist so lookups are very fast. I thought about a "native" hash set, or Memcached, or similar methods. Memcached would very likely slow down things if it's distributed. Has anybody implemented a similar approach? Which tools does Varnish have at my disposal?

I need to invalidate the cache of a set of URLs related to a resource that was either deleted/unpublished, or updated. E.g if my resource UUID is 1234abcd I want to invalidate all its cached derivatives under /resource/1234abcd/*.

I understand that there is no way in Varnish to do this with purge, but it can be done with bans. However I have a hard time understanding how bans exactly work.

If e.g. I update resource 1234abcd and ban all its derivatives, I assume that the next client request to /resource/1234abcd/derv1 will be a new backend fetch, and that new resource will be cached. Will I have two version of the same derivative, one banned and one not (until the old one expires and eventually its ban expires too)? If my resources have a long expiration date I may accumulate a lot of bans for cache resources that I would much rather have cleared right away.

On a software design level, What is the utility of leaving inaccessible resources around instead of implementing a regex-based purge, which seems more straightforward to manage?

Also, I implemented a ban in my development env, and I see that a ban only takes effect a minute or so after being requested. Does this have to do with the ban lurker or some other timing setting?

Thanks.

I am setting up a virtualenv based on Python 2.7 in Redhat Enterprise 6 (which ships with 2.6.x by default).

I installed the SCL 2.7 version of Python and then created the virtualenv:

$ scl enable python27 bash
$ cd /my/project
$ virtualenv -v --clear --extra-search-dir=/opt/rh/python27/root/usr/bin/ --python=/opt/rh/python27/root/usr/bin/python virtualenv
$ source virtualenv/bin/acticvate

So far so good. The problems started when I tried to run pip behind my institution's proxy:

$ pip install Werkzeug --proxy proxy.example.org:3128                       
Downloading/unpacking Werkzeug                                                 
Cleaning up...                                                                 
Exception:                                                                     
Traceback (most recent call last):                                             
  File "/my/project/virtualenv/lib/python2.7/site-packages/pip/basecommand.py", line 134, in main
    status = self.run(options, args)                                           
  File "/my/project/virtualenv/lib/python2.7/site-packages/pip/commands/install.py", line 236, in run
    requirement_set.prepare_files(finder, force_root_egg_info=self.bundle, bundle=self.bundle)
  File "/my/project/virtualenv/lib/python2.7/site-packages/pip/req.py", line 1085, in prepare_files
    url = finder.find_requirement(req_to_install, upgrade=self.upgrade)        
  File "/my/project/virtualenv/lib/python2.7/site-packages/pip/index.py", line 201, in find_requirement
    page = self._get_page(main_index_url, req)                                 
  File "/my/project/virtualenv/lib/python2.7/site-packages/pip/index.py", line 554, in _get_page
    return HTMLPage.get_page(link, req, cache=self.cache)                      
  File "/my/project/virtualenv/lib/python2.7/site-packages/pip/index.py", line 671, in get_page
    resp = urlopen(url)                                                        
  File "/my/project/virtualenv/lib/python2.7/site-packages/pip/download.py", line 176, in __call__
    response = self.get_opener(scheme=scheme).open(url)                        
  File "/opt/rh/python27/root/usr/lib64/python2.7/urllib2.py", line 404, in open
    response = self._open(req, data)                                           
  File "/opt/rh/python27/root/usr/lib64/python2.7/urllib2.py", line 422, in _open
    '_open', req)                                                              
  File "/opt/rh/python27/root/usr/lib64/python2.7/urllib2.py", line 382, in _call_chain
    result = func(*args)                                                       
  File "/my/project/virtualenv/lib/python2.7/site-packages/pip/download.py", line 155, in https_open
    return self.do_open(self.specialized_conn_class, req)                      
  File "/opt/rh/python27/root/usr/lib64/python2.7/urllib2.py", line 1183, in do_open
    h.request(req.get_method(), req.get_selector(), req.data, headers)         
  File "/opt/rh/python27/root/usr/lib64/python2.7/httplib.py", line 1001, in request
    self._send_request(method, url, body, headers)                             
  File "/opt/rh/python27/root/usr/lib64/python2.7/httplib.py", line 1035, in _send_request
    self.endheaders(body)                                                      
  File "/opt/rh/python27/root/usr/lib64/python2.7/httplib.py", line 997, in endheaders
    self._send_output(message_body)                                            
  File "/opt/rh/python27/root/usr/lib64/python2.7/httplib.py", line 850, in _send_output
    self.send(msg)                                                             
  File "/opt/rh/python27/root/usr/lib64/python2.7/httplib.py", line 812, in send
    self.connect()                                                             
  File "/my/project/virtualenv/lib/python2.7/site-packages/pip/download.py", line 139, in connect
    match_hostname(self.sock.getpeercert(), self.host)                         
  File "/my/project/virtualenv/lib/python2.7/site-packages/pip/backwardcompat/ssl_match_hostname.py", line 61, in match_hostname
    % (hostname, ', '.join(map(repr, dnsnames))))                              
CertificateError: hostname 'proxy.example.org' doesn't match either of 'www.python.org', 'python.org', 'pypi.python.org', 'docs.python.org', 'testpypi.python.org', 'bug>>>s.python.org', 'wiki.python.org', 'hg.python.org', 'mail.python.org', 'packaging.python.org', 'pythonhosted.org', 'www.pythonhosted.org', 'test.pythonhosted.org', 'us.py>>>con.org', 'id.python.org', 'pypi.io'

I have seen this issue in other questions but none of these seem to apply to my case.

Edit:

E.g. I tried setting environment variables as in https://stackoverflow.com/questions/21468550/pip-not-working-behind-firewall . Other cases were addressing Windows issues.

It may be worth mentioning that this virtualenv won't work if I am not in the scl environment, e.g. if I call python enabling the virtualenv but not scl, it will complain that libpython2.7.so.1.0 cannot be found. I don't know if this is bny design, but I expected all the libraries Python 2.7 depends on to be symlinked or copied in the virtualenv. Maybe something I did wrong in the virtualenv installation?