Arnaud's questions

We have this new client for whom we're reviewing our server infrastructure.

I know pretty well the web API because I've helped building it and now I'm on my own maintaining and pushing it forward, so big challenge and very interesting.

It is based on an Amazon m1.large instance, nginx (+ssl), django, amazon RDS (with MySQL) and a self hosted memcached for now.

The thing is we had some inputs from our client saying that they expect something like a max of 2500 users connecting to the API for a range of four hours, twice a day at least.

We have no idea of when exactly those connections will arise and we should not make any assumptions, so the thing I ended up thinking was that our server better has to support the 2500 connections at one point in time.

I've been playing around with apache benchmark sending 2500 concurrent connections while connecting/disconnecting memcache or some nginx settings, just to see the performances changes.

The best I came with was around 100 requests per second but the longest requests take more than 20 seconds (for 2500 concurrent connections, with only 100 the requests take max 1s). From a user point of view, I wouldn't like to wait more than 1 or 2 seconds for getting my result...

I'd like to play more with all the settings I can tune on nginx, django, mysql or memcache but at this point I think I need a methodology and more than a methodology, I need a goal to reach.

Searching on the web I see blog posts talking about services that reach several hundreds of requests per second. I'm far from that.

Seeing all those numbers coming out from apachebench are just giving me the impression that I am launching tests, seeing the results, but that I don't really understand them and don't really know what to do with them to improve our API.

So what would be a good methodology, a good approach to reach the goal of having a web API able to cope with this number of connections as fast as possible?

If you need more details just ask!

I am trying to setup a server where my git repo would be accessible with HTTP(S).

I am using gitolite and nginx (and gitlab for web interface but I doubt it makes any difference).

I have searched the whole afternoon and I think I'm stuck.

I have think I have understood that nginx needs fcgiwrap to work with gitolite, so I tried several configurations, but none of them work.

My repositories are at /home/git/repositories.

Here's the three nginx configurations I have tried.

1:

   location ~ /git(/.*) {
       gzip off;
       root /usr/lib/git-core;

       fastcgi_pass  unix:/var/run/fcgiwrap.socket;
       include /etc/nginx/fcgiwrap.conf;

       fastcgi_param SCRIPT_FILENAME /usr/lib/git-core/git-http-backend;
       fastcgi_param DOCUMENT_ROOT /usr/lib/git-core/;
       fastcgi_param SCRIPT_NAME git-http-backend;

       fastcgi_param GIT_HTTP_EXPORT_ALL "";
       fastcgi_param GIT_PROJECT_ROOT /home/git/repositories;
       fastcgi_param PATH_INFO $1;
       #fastcgi_param PATH_TRANSLATED $document_root$fastcgi_path_info;
    }

Result:

> git clone http://myservername/projectname.git test/
Cloning into test...
fatal: http://myservername/projectname.git/info/refs not found: did you run git update-server-info on the server?

and

> git clone http://myservername/git/projectname.git test/
Cloning into test...
error: The requested URL returned error: 502 while accessing http://myservername/git/projectname.git/info/refs
fatal: HTTP request failed

2:

   location ~ /git(/.*) {
        fastcgi_pass  localhost:9001;
        include       /etc/nginx/fcgiwrap.conf;
        fastcgi_param SCRIPT_FILENAME     /usr/lib/git-core/git-http-backend;
        fastcgi_param GIT_HTTP_EXPORT_ALL "";
        fastcgi_param GIT_PROJECT_ROOT    /home/git/repositories;
        fastcgi_param PATH_INFO           $1;
    }

Result:

> git clone http://myservername/projectname.git test/
Cloning into test...
fatal: http://myservername/projectname.git/info/refs not found: did you run git update-server-info on the server?

and

> git clone http://myservername/git/projectname.git test/
Cloning into test...
error: The requested URL returned error: 502 while accessing http://myservername/git/projectname.git/info/refs
fatal: HTTP request failed

3:

location ~ ^.*\.git/objects/([0-9a-f]+/[0-9a-f]+|pack/pack-[0-9a-f]+.(pack|idx))$ {
    root /home/git/repositories/;
  }

  location ~ ^.*\.git/(HEAD|info/refs|objects/info/.*|git-(upload|receive)-pack)$ {
    root /home/git/repositories;

    fastcgi_pass unix:/var/run/fcgiwrap.socket;
    fastcgi_param SCRIPT_FILENAME   /usr/lib/git-core/git-http-backend;
    fastcgi_param PATH_INFO         $uri;
    fastcgi_param GIT_PROJECT_ROOT  /home/git/repositories;  
    include /etc/nginx/fcgiwrap.conf;
  }

Result:

> git clone http://myservername/projectname.git test/
Cloning into test...
error: The requested URL returned error: 502 while accessing http://myservername/projectname.git/info/refs
fatal: HTTP request failed

and

> git clone http://myservername/git/projectname.git test/
Cloning into test...
error: The requested URL returned error: 502 while accessing http://myservername/git/projectname.git/info/refs
fatal: HTTP request failed

Also note that with any of those configurations, when I try to clone with a project name that actually doesn't exist, I get a 502 error.

Does anyone already succeeded in doing this? What am I doing wrong?

Thanks.

UPDATE:

nginx error log file said:

2012/04/05 17:34:50 [crit] 21335#0: *50 connect() to unix:/var/run/fcgiwrap.socket failed (13: Permission denied) while connecting to upstream, client: 192.168.12.201, server: myservername, request: "GET /git/oct_editor.git/info/refs HTTP/1.1", upstream: "fastcgi://unix:/var/run/fcgiwrap.socket:", host: "myservername"

So I changed permissions for /var/run/fcgiwrap.socket, and now I have :

> git clone http://myservername/git/projectname.git test/
Cloning into test...
error: The requested URL returned error: 403 while accessing     http://myservername/git/projectname.git/info/refs
fatal: HTTP request failed

Here is the error.log file I have now:

2012/04/05 17:36:52 [error] 21335#0: *78 FastCGI sent in stderr: "Cannot chdir to script directory (/usr/lib/git-core/git/projectname.git/info)" while reading response header from upstream, client: 192.168.12.201, server: myservername, request: "GET /git/projectname.git/info/refs HTTP/1.1", upstream: "fastcgi://unix:/var/run/fcgiwrap.socket:", host: "myservername"

I keep on investigating.