IP based load balance in nginx

I have two Backend server running on 192.168.0.101:8080 and 192.168.0.102:8080.

I have nginx running on 192.168.0.111:80, I want to force nginx to reverse traffic in specific backend srever by IP.

geo $upstream {
        192.168.0.150 backend-1;
        192.168.0.250 backend-2;
        default backend-1;
}

upstream backend-1 {
   ip_hash;
   server 192.168.0.101:8080;
}

upstream backend-2 {
   ip_hash;
   server 192.168.0.102:8080;
}


server {
        listen 80;

        location / {
        proxy_pass http://$upstream;
        proxy_redirect off;
            proxy_set_header Host               $host;
            proxy_set_header X-Forwarded-Host   $host;
            proxy_set_header X-Forwarded-Server $host;
            proxy_set_header X-Real-IP          $remote_addr;
            proxy_set_header X-Forwarded-For    $proxy_add_x_forwarded_for;
        }                                                                    
}

I am confused about this configuration. Is it gonna work as it meant ? Will every request but 192.168.0.150 will go to backend-1(192.168.0.102:8080) ? and 192.168.0.250 in backend-2 ?

Question: It is not redirecting based on IP addresses. Why is that ?