ALex_hha

Asked: 2019-02-09 07:12:52 +0800 CST

ldapsearch doesn't work wih ldap.google.com via stunnel

2

Can't get it working. I have to use stunnel to be able integrate Jenkins with Google LDAP service. It works fine without stunnel

$ LDAPTLS_CERT=/etc/stunnel/gldap.crt LDAPTLS_KEY=/etc/stunnel/gldap.key \
ldapsearch -H ldaps://ldap.google.com -b "dc=example,dc=com" uid=alex mail

SASL/EXTERNAL authentication started
SASL username: st=California,c=US,ou=GSuite,cn=LDAP Client,l=Mountain View,o=Google Inc.
SASL SSF: 0
# extended LDIF
#
# LDAPv3
# base <dc=example,dc=com> with scope subtree
# filter: uid=alex
# requesting: mail 
#

# alex, Users, example.com
dn: uid=alex,ou=Users,dc=example,dc=com
mail: [email protected]

# search result
search: 3
result: 0 Success

# numResponses: 2
# numEntries: 1

I have tried both - ubuntu-16.04 and 18.04. Stunnel config is pretty simple and based on official docs

# cat /etc/stunnel/ldap.conf 
debug = 7
output = /tmp/stunnel-gldap.log

[ldap]
client = yes
accept = 127.0.0.1:389
connect = ldap.google.com:636
cert = /etc/stunnel/gldap.crt
key = /etc/stunnel/gldap.key

But doesn't work via stunnel

$ ldapsearch -H ldap://127.0.0.1 -b "dc=example,dc=com" uid=alex
SASL/EXTERNAL authentication started
ldap_sasl_interactive_bind_s: Unknown authentication method (-6)
    additional info: SASL(-4): no mechanism available:

Stunnel logs

2019.02.08 17:00:24 LOG7[main]: Service [ldap] accepted (FD=3) from 127.0.0.1:42296
2019.02.08 17:00:24 LOG7[0]: Service [ldap] started
2019.02.08 17:00:24 LOG5[0]: Service [ldap] accepted connection from 127.0.0.1:42296
2019.02.08 17:00:24 LOG6[0]: s_connect: connecting 216.239.32.58:636
2019.02.08 17:00:24 LOG7[0]: s_connect: s_poll_wait 216.239.32.58:636: waiting 10 seconds
2019.02.08 17:00:24 LOG5[0]: s_connect: connected 216.239.32.58:636
2019.02.08 17:00:24 LOG5[0]: Service [ldap] connected remote server from 192.168.3.13:59504
2019.02.08 17:00:24 LOG7[0]: Remote descriptor (FD=9) initialized
2019.02.08 17:00:24 LOG6[0]: SNI: sending servername: ldap.google.com
2019.02.08 17:00:24 LOG7[0]: SSL state (connect): before/connect initialization
2019.02.08 17:00:24 LOG7[0]: SSL state (connect): SSLv2/v3 write client hello A
2019.02.08 17:00:24 LOG6[0]: Certificate verification disabled
2019.02.08 17:00:24 LOG6[0]: Certificate verification disabled
2019.02.08 17:00:24 LOG7[0]:      1 client connect(s) requested
2019.02.08 17:00:24 LOG7[0]:      1 client connect(s) succeeded
2019.02.08 17:00:24 LOG7[0]:      0 client renegotiation(s) requested
2019.02.08 17:00:24 LOG7[0]:      0 session reuse(s)
2019.02.08 17:00:24 LOG6[0]: SSL connected: new session negotiated
2019.02.08 17:00:24 LOG7[0]: Peer certificate was cached (3201 bytes)
2019.02.08 17:00:24 LOG6[0]: Negotiated TLSv1.2 ciphersuite ECDHE-RSA-AES128-GCM-SHA256 (128-bit encryption)
2019.02.08 17:00:24 LOG7[0]: Compression: null, expansion: null
2019.02.08 17:00:25 LOG6[0]: Read socket closed (readsocket)
2019.02.08 17:00:25 LOG7[0]: Sending close_notify alert
2019.02.08 17:00:25 LOG7[0]: SSL alert (write): warning: close notify
2019.02.08 17:00:25 LOG6[0]: SSL_shutdown successfully sent close_notify alert
2019.02.08 17:00:25 LOG6[0]: SSL socket closed (SSL_read)
2019.02.08 17:00:25 LOG7[0]: Sent socket write shutdown
2019.02.08 17:00:25 LOG5[0]: Connection closed: 71 byte(s) sent to SSL, 71 byte(s) sent to socket
2019.02.08 17:00:25 LOG7[0]: Remote descriptor (FD=9) closed
2019.02.08 17:00:25 LOG7[0]: Local descriptor (FD=3) closed
2019.02.08 17:00:25 LOG7[0]: Service [ldap] finished (0 left)

Also I have tried ldapsearch with debug output

$ ldapsearch -d5 -H ldap://127.0.0.1 -b "dc=example,dc=com" uid=alex
ldap_url_parse_ext(ldap://127.0.0.1)
ldap_create
ldap_url_parse_ext(ldap://127.0.0.1:389/??base)
ldap_pvt_sasl_getmech
ldap_search
put_filter: "(objectclass=*)"
put_filter: simple
put_simple_filter: "objectclass=*"
ldap_build_search_req ATTRS: supportedSASLMechanisms
ldap_send_initial_request
ldap_new_connection 1 1 0
ldap_int_open_connection
ldap_connect_to_host: TCP 127.0.0.1:389
ldap_new_socket: 4
ldap_prepare_socket: 4
ldap_connect_to_host: Trying 127.0.0.1:389
ldap_pvt_connect: fd: 4 tm: -1 async: 0
attempting to connect: 
connect success
ldap_open_defconn: successful
ldap_send_server_request
ber_scanf fmt ({it) ber:
ber_scanf fmt ({) ber:
ber_flush2: 64 bytes to sd 4
ldap_result ld 0x56265b1fd400 msgid 1
wait4msg ld 0x56265b1fd400 msgid 1 (infinite timeout)
wait4msg continue ld 0x56265b1fd400 msgid 1 all 1
** ld 0x56265b1fd400 Connections:
* host: 127.0.0.1  port: 389  (default)
  refcnt: 2  status: Connected
  last used: Fri Feb  8 17:02:39 2019


** ld 0x56265b1fd400 Outstanding Requests:
 * msgid 1,  origid 1, status InProgress
   outstanding referrals 0, parent count 0
  ld 0x56265b1fd400 request count 1 (abandoned 0)
** ld 0x56265b1fd400 Response Queue:
   Empty
  ld 0x56265b1fd400 response count 0
ldap_chkResponseList ld 0x56265b1fd400 msgid 1 all 1
ldap_chkResponseList returns ld 0x56265b1fd400 NULL
ldap_int_select
read1msg: ld 0x56265b1fd400 msgid 1 all 1
ber_get_next
ber_get_next: tag 0x30 len 55 contents:
read1msg: ld 0x56265b1fd400 msgid 1 message type search-entry
wait4msg continue ld 0x56265b1fd400 msgid 1 all 1
** ld 0x56265b1fd400 Connections:
* host: 127.0.0.1  port: 389  (default)
  refcnt: 2  status: Connected
  last used: Fri Feb  8 17:02:39 2019


** ld 0x56265b1fd400 Outstanding Requests:
 * msgid 1,  origid 1, status InProgress
   outstanding referrals 0, parent count 0
  ld 0x56265b1fd400 request count 1 (abandoned 0)
** ld 0x56265b1fd400 Response Queue:
 * msgid 1,  type 100
  ld 0x56265b1fd400 response count 1
ldap_chkResponseList ld 0x56265b1fd400 msgid 1 all 1
ldap_chkResponseList returns ld 0x56265b1fd400 NULL
ldap_int_select
read1msg: ld 0x56265b1fd400 msgid 1 all 1
ber_get_next
ber_get_next: tag 0x30 len 12 contents:
read1msg: ld 0x56265b1fd400 msgid 1 message type search-result
ber_scanf fmt ({eAA) ber:
read1msg: ld 0x56265b1fd400 0 new referrals
read1msg:  mark request completed, ld 0x56265b1fd400 msgid 1
request done: ld 0x56265b1fd400 msgid 1
res_errno: 0, res_error: <>, res_matched: <>
ldap_free_request (origid 1, msgid 1)
adding response ld 0x56265b1fd400 msgid 1 type 101:
ldap_parse_result
ber_scanf fmt ({iAA) ber:
ber_scanf fmt (}) ber:
ldap_get_values
ber_scanf fmt ({x{{a) ber:
ber_scanf fmt ([v]) ber:
ldap_msgfree
ldap_sasl_interactive_bind: server supports: EXTERNAL PLAIN
ldap_int_sasl_bind: EXTERNAL PLAIN
ldap_int_sasl_open: host=ws-alex
SASL/EXTERNAL authentication started
ldap_msgfree
ldap_err2string
ldap_sasl_interactive_bind_s: Unknown authentication method (-6)
    additional info: SASL(-4): no mechanism available: 
ldap_free_connection 1 1
ldap_send_unbind
ber_flush2: 7 bytes to sd 4
ldap_free_connection: actually freed

The same thing with Jenkins when I try to use stunnel.

Login
Authentication: failed for user "uid=alex,ou=Users,dc=example,dc=com"

Lookup
User lookup: user "uid=alex,ou=Users,dc=example,dc=com" does not exist.
Does looking up user details require a Manager Dn and password?
Are the user search base and user search filter settings correct?

LDAP Group lookup: could not verify.
Please try with a user that is a member of at least one LDAP group.

Lockout
The user "uid=alex,ou=Users,dc=example,dc=com" will be unable to login with the supplied password.
If this is your own account this would mean you would be locked out!
Are you sure you want to save this configuration?

Did I miss something? Or maybe there is some other workaround?

Thanks in advance

UPDATE1

I was able to get success with ldapsearch with the following command (had to use bind_dn user)

$ ldapsearch -x -D "BIND_USER" -w 123456789 -H ldap://127.0.0.1 -b "dc=example,dc=com" uid=alex
ldap_bind: Success (0)
    additional info: Valid access code
# extended LDIF
#
# LDAPv3
# base <dc=example,dc=com> with scope subtree
# filter: uid=alex
# requesting: ALL
#

# alex, Users, example.com
dn: uid=alex,ou=Users,dc=example,dc=com

But Jenkins still doesn't work

ALex_hha

Asked: 2017-02-21 12:11:47 +0800 CST

How to grant access to an SQS to a specific IAM user

4

I need to create a really simple IAM policy and grant it to a specific queue. I need to grant access (it should be a full access) to the queue only to specific IAM user.

Because at the moment by default all IAM users with policy AmazonSQSFullAccess/AdministratorAccess can send/read message to/from the queue.

I have tried the following policies but without success

Policy 1

{
  "Version": "2012-10-17",
  "Id": "arn:aws:sqs:us-east-1:930XXXXXX332:task-queue/SQSDefaultPolicy",
  "Statement": [
    {
      "Sid": "Sid1487598389851",
      "Effect": "Deny",
      "Principal": "*",
      "Action": "SQS:*",
      "Resource": "arn:aws:sqs:us-east-1:930XXXXXX332:task-queue",
      "Condition": {
        "ArnNotEquals": {
          "aws:SourceArn": "arn:aws:iam::930XXXXXX332:user/test-sqs"
        }
      }
    },
    {
      "Sid": "Sid1487599825058",
      "Effect": "Allow",
      "Principal": {
        "AWS": "arn:aws:iam::930XXXXXX332:user/test-sqs"
      },
      "Action": "SQS:*",
      "Resource": "arn:aws:sqs:us-east-1:930XXXXXX332:task-queue"
    }
  ]
}

Policy 2 (the same as above but I have tried another condition)

"Condition": {
        "NotPrincipal": { 
             "AWS": "arn:aws:iam::930XXXXXX332:user/test-sqs" 
        }
  }

In other words - I need to get something like the following

Allow: user1, user2
Deny: *

Is it possible at all?

At the moment I have to explicitly specify each user within Deny effect. And this is extremely inconvenient

ALex_hha

Asked: 2016-03-09 03:18:49 +0800 CST

Does FTP service (IIS 7.5) on Windows 2008 R2 support kerberos authentication?

1

I have configured http service on IIS 7.5 for kerberos authentication. And it works fine.

I have checked from linux box

# klist
Ticket cache: FILE:/tmp/krb5cc_0
Default principal: [email protected]

Valid starting     Expires            Service principal
03/08/16 10:19:30  03/08/16 20:19:31  krbtgt/[email protected]
        renew until 03/15/16 10:19:30
03/08/16 10:20:00  03/08/16 20:19:31  HTTP/[email protected]
        renew until 03/15/16 10:19:30

Simple test

# curl --trace-ascii - --negotiate -u : http://www.example.net
Enter host password for user '[email protected]':
== Info: About to connect() to www.example.net port 80 (#0)
== Info:   Trying 192.168.1.100... == Info: connected
== Info: Connected to www.example.net (192.168.1.100) port 80 (#0)
=> Send header, 178 bytes (0xb2)
0000: GET / HTTP/1.1
0010: User-Agent: curl/7.19.7 (x86_64-redhat-linux-gnu) libcurl/7.19.7
0050:  NSS/3.19.1 Basic ECC zlib/1.2.3 libidn/1.18 libssh2/1.4.2
008c: Host: www.example.net
00a3: Accept: */*
00b0:
...
== Info: Connection #0 to host www.example.net left intact
== Info: Issue another request to this URL: 'http://www.example.net/'
== Info: Re-using existing connection! (#0) with host www.example.net
== Info: Connected to www.example.net (192.168.1.100) port 80 (#0)
== Info: Server auth using GSS-Negotiate with user ''
=> Send header, 1969 bytes (0x7b1)
0000: GET / HTTP/1.1
0010: Authorization: Negotiate YIIFJgYJKoZIhvcSAQICAQBuggUVMIIFEaADAgE
0050: FoQMCAQ6iBwMFAAAAAACjggQkYYIEIDCCBBygAwIBBaENGwtFWEFNUExFLk5FVKI
0090: iMCCgAwIBA6EZMBcbBEhUVFAbD2RjMS5leGFtcGxlLm5ldKOCA+AwggPcoAMCARK
...
070f: User-Agent: curl/7.19.7 (x86_64-redhat-linux-gnu) libcurl/7.19.7
074f:  NSS/3.19.1 Basic ECC zlib/1.2.3 libidn/1.18 libssh2/1.4.2
078b: Host: www.example.net
07a2: Accept: */*
07af:
<= Recv header, 17 bytes (0x11)
0000: HTTP/1.1 200 OK
<= Recv header, 25 bytes (0x19)
0000: Content-Type: text/html
<= Recv header, 46 bytes (0x2e)
0000: Last-Modified: Sat, 05 Mar 2016 17:15:16 GMT
<= Recv header, 22 bytes (0x16)
0000: Accept-Ranges: bytes
...

Anonymous authentication has been disabled and Windows authentication has been enabled in the Authentication section for the site and for the whole server. I left only Negotiate:Kerberos in Providers section. Kernel-mode authentication has been disabled.

I have created SPN record for ftp service

> setspn.exe -a ftp/ftp.example.net dc1
Registering ServicePrincipalNames for CN=DC1,OU=Domain Controllers,DC=example,DC=net
        ftp/ftp.example.net
Updated object

And I can see newly created SPN record

> Setspn.exe -L DC1
Registered ServicePrincipalNames for CN=DC1,OU=Domain Controllers,DC=example,DC=net:
        ftp/ftp.example.net
        Dfsr-12F9A27C-BF97-4787-9364-D31B6C55EB04/dc1.example.net
        HOST/DC1/EXAMPLE
        ldap/DC1/EXAMPLE
        ldap/dc1.example.net/ForestDnsZones.example.net
        ldap/dc1.example.net/DomainDnsZones.example.net
        DNS/dc1.example.net
        GC/dc1.example.net/example.net
        RestrictedKrbHost/dc1.example.net
        RestrictedKrbHost/DC1
        HOST/dc1.example.net/EXAMPLE
        HOST/DC1
        HOST/dc1.example.net
        HOST/dc1.example.net/example.net
        ldap/dc1.example.net/EXAMPLE
        ldap/DC1
        ldap/dc1.example.net
        ldap/dc1.example.net/example.net
        E3514235-4B06-11D1-AB04-00C04FC2DCD2/f9c3624c-5f19-4e0a-8afa-d730b0bd4e6d/example.net
        ldap/f9c3624c-5f19-4e0a-8afa-d730b0bd4e6d._msdcs.example.net

But when I trying connect to FTP service I get the following error

# curl --trace-ascii - --krb 5 -u : ftp://ftp.example.net
== Info: About to connect() to ftp.example.net port 21 (#0)
== Info:   Trying 192.168.1.100... == Info: connected
== Info: Connected to ftp.example.net (192.168.1.100) port 21 (#0)
<= Recv header, 27 bytes (0x1b)
0000: 220 Microsoft FTP Service
== Info: Trying GSSAPI...
=> Send header, 13 bytes (0xd)
0000: AUTH GSSAPI
<= Recv header, 41 bytes (0x29)
0000: 504 Security mechanism not implemented.
== Info: GSSAPI is not supported by the server.
== Info: Logging in with password in cleartext!
...

I have read a lot of official documentations but couldn't find out is it possible to configure Kerberos authentication for FTP service.

So my questions are:

Is it possible at all. If it is, what I have missed?
If it is not possible, what FTP server I can use with kerberos authentication?

P.S. In advanced settings of Windows Authentication section there is an option - Kernel-mode authentication with the following description

By default, IIS enables kernel-mode authentication, which may improve authentication performance and prevent authentication problems with application pools configured to use a custom identity. As a best practice, do not disable this setting if Kerberos authentication is used in your environment and the application pool is configured to use a custom identity.

But when I check the box I get the following error

Did I miss something? Because on previous screen IIS has recommended do not disable kernel-mode authentication especially for kerberos authentication. So I'm a little bit confused.

Thanks in advance

ALex_hha

Asked: 2014-02-02 05:40:41 +0800 CST

Subversion on 1 Gb uplink

1

is it possible at all to utilize 1 Gb uplink? I have 2 server with 1 Gb uplink

176.9.xxx.xxx - server

# uname -a
Linux svn.example.net 2.6.32-5-amd64 #1 SMP Mon Sep 23 22:14:43 UTC 2013 x86_64 GNU/Linux

# cat /etc/debian_version
6.0.8

# svnadmin --version
svnadmin, version 1.6.23 (r1485506)
   compiled May 29 2013, 10:00:56

Copyright (C) 2000-2009 CollabNet.
Subversion is open source software, see http://subversion.apache.org/
This product includes software developed by CollabNet (http://www.Collab.Net/).

The following repository back-end (FS) modules are available:

* fs_base : Module for working with a Berkeley DB repository.
* fs_fs : Module for working with a plain file (FSFS) repository.

# ethtool eth0
Settings for eth0:
    Supported ports: [ TP ]
    Supported link modes:   10baseT/Half 10baseT/Full
                            100baseT/Half 100baseT/Full
                            1000baseT/Full
    Supports auto-negotiation: Yes
    Advertised link modes:  10baseT/Half 10baseT/Full
                            100baseT/Half 100baseT/Full
                            1000baseT/Full
    Advertised pause frame use: No
    Advertised auto-negotiation: Yes
    Speed: 1000Mb/s
    Duplex: Full
    Port: Twisted Pair
    PHYAD: 1
    Transceiver: internal
    Auto-negotiation: on
    MDI-X: on
    Supports Wake-on: pumbag
    Wake-on: g
    Current message level: 0x00000001 (1)
    Link detected: yes

144.76.xxx.xxx - client

# uname -a
Linux test.example.net 2.6.32-431.el6.x86_64 #1 SMP Fri Nov 22 03:15:09 UTC 2013 x86_64 x86_64 x86_64 GNU/Linux

# cat /etc/redhat-release
CentOS release 6.5 (Final)

# svnadmin --version
svnadmin, version 1.6.11 (r934486)
compiled Apr 11 2013, 16:13:51

Copyright (C) 2000-2009 CollabNet.
Subversion is open source software, see http://subversion.tigris.org/
This product includes software developed by CollabNet (http://www.Collab.Net/).

The following repository back-end (FS) modules are available:

* fs_base : Module for working with a Berkeley DB repository.
* fs_fs : Module for working with a plain file (FSFS) repository.

# ethtool eth0
Settings for eth0:
    Supported ports: [ TP MII ]
    Supported link modes:   10baseT/Half 10baseT/Full
                            100baseT/Half 100baseT/Full
                            1000baseT/Half 1000baseT/Full
    Supported pause frame use: No
    Supports auto-negotiation: Yes
    Advertised link modes:  10baseT/Half 10baseT/Full
                            100baseT/Half 100baseT/Full
                            1000baseT/Half 1000baseT/Full
    Advertised pause frame use: Symmetric Receive-only
    Advertised auto-negotiation: Yes
    Link partner advertised link modes:  10baseT/Half 10baseT/Full
                                         100baseT/Half 100baseT/Full
                                         1000baseT/Full
    Link partner advertised pause frame use: No
    Link partner advertised auto-negotiation: Yes
    Speed: 1000Mb/s
    Duplex: Full
    Port: MII
    PHYAD: 0
    Transceiver: internal
    Auto-negotiation: on
    Supports Wake-on: pumbg
    Wake-on: g
    Current message level: 0x00000033 (51)
                           drv probe ifdown ifup
    Link detected: yes

Some basic tests

# iperf -c 176.9.xxx.xxx -t 60
------------------------------------------------------------
Client connecting to 176.9.xxx.xxx, TCP port 5001
TCP window size: 19.3 KByte (default)
------------------------------------------------------------
[  3] local 144.76.xxx.xxx port 42619 connected with 176.9.xxx.xxx port 5001
[ ID] Interval       Transfer     Bandwidth
[  3]  0.0-60.0 sec  4.15 GBytes   594 Mbits/sec

# iperf -c 144.76.xxx.xxx -t 60
------------------------------------------------------------
Client connecting to 144.76.xxx.xxx, TCP port 5001
TCP window size: 16.0 KByte (default)
------------------------------------------------------------
[  3] local 176.9.xxx.xxx port 54666 connected with 144.76.xxx.xxx port 5001
[ ID] Interval       Transfer     Bandwidth
[  3]  0.0-60.0 sec  3.17 GBytes    453 Mbits/sec

At the same time when I download some project from svn via http speed is about 100 Mbit/s max. But a plain binary file has been downloaded at max speed

# axel -a -v http://176.9.xxx.xxx/test.img
Initializing download: http://176.9.xxx.xxx/test.img
File size: 1101824020 bytes
Opening output file test.img
Starting download

Connection 3 finished                                                          ]
Connection 0 finished                                                          ]
Connection 1 finished                                                          ]
[100%] [..................................................] [  94.3MB/s] [00:00]

Downloaded 1050.8 megabytes in 11 seconds. (96588.44 KB/s)

# wget http://176.9.xxx.xxx/test.img
--2014-02-01 14:21:13--  http://176.9.xxx.xxx/test.img
Connecting to 176.9.xxx.xxx:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 1101824020 (1.0G) [text/plain]
Saving to: “test.img”

100%[=================================>] 1,101,824,020 51.0M/s   in 21s

2014-02-01 14:21:34 (49.9 MB/s) - “test.img” saved [1101824020/1101824020]

# curl -o test.img http://176.9.xxx.xxx/test.img
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                             Dload  Upload   Total   Spent    Left  Speed
100 1050M  100 1050M    0     0  56.8M      0  0:00:18  0:00:18 --:--:-- 57.2M

Any advice would be highly appreciated.

Update1

As you an see the speed is around 100 Mbit

# time svn co http://svn.example.net/Test/ ./Test/
Authentication realm: <http://svn.example.net:80> Authorization required.
Password for 'user':
A    Test/test.img
Checked out revision 1.

real    1m40.768s
user    0m48.885s
sys     0m3.738s

download speed from svn

Update2 The same file but via svn protocol, the speed around 250 Mbit/s

# time svn co svn://svn.example.net/ ./Test/
A    Test/test.img
Checked out revision 1.

real    0m46.075s
user    0m15.338s
sys     0m4.811s

download speed from svn via svn protocol

System load during checkout

# top
top - 18:26:34 up 60 days, 10:14,  1 user,  load average: 0.25, 0.06, 0.02
Tasks: 214 total,   1 running, 213 sleeping,   0 stopped,   0 zombie
Cpu0  :  0.0%us,  0.0%sy,  0.0%ni,100.0%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st
Cpu1  : 99.0%us,  1.0%sy,  0.0%ni,  0.0%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st
Cpu2  :  0.0%us,  0.9%sy,  0.0%ni, 99.1%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st
Cpu3  :  0.0%us,  0.0%sy,  0.0%ni,100.0%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st
Cpu4  :  0.0%us,  0.0%sy,  0.0%ni,100.0%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st
Cpu5  :  0.0%us,  0.0%sy,  0.0%ni,100.0%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st
Cpu6  :  0.0%us,  0.0%sy,  0.0%ni,100.0%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st
Cpu7  :  0.0%us,  0.0%sy,  0.0%ni,100.0%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st
Mem:  16377584k total, 15345624k used,  1031960k free,   203640k buffers
Swap:  8388600k total,     1956k used,  8386644k free, 13597864k cached

PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
5449 www-data  20   0  409m  68m 3656 S  100  0.4   2:17.41 apache2
6885 root      20   0 10976 1340  944 R    1  0.0   0:00.28 top

ALex_hha

Asked: 2013-05-29 00:36:22 +0800 CST

Is it possible to cache subversion?

3

Is it possible to cache subversion at all? May be any commercial solution?

Any help would by highly appreciated.

ALex_hha

Asked: 2013-04-24 11:53:33 +0800 CST

squid: can't connect to peer with tcp_outgoing_address

1

I encountered the problem with configuration of two squid servers. I have the following scheme -

enter image description here

The main idea is to download all files from rackspace and amazon through parent squid and store all files in his cache.

On the main_squid server was configured PBR (iptables + ip route). All packets go through the same channels through which the requests were received

# ip ru sh
0:      from all lookup local
1000:   from all fwmark 0x3e8 lookup ISP1
2000:   from all fwmark 0x7d0 lookup ISP2
3011:   from all fwmark 0xbc3 lookup ISP3
32762:  from xxx.xxx.xxx.62 lookup ISP1
32763:  from yyy.yyy.yyy.239 lookup ISP2
32764:  from zzz.zzz.zzz.10 lookup ISP3
32766:  from all lookup main
32767:  from all lookup default

main_squid squid.conf http_port 192.168.210.1:3128 transparent

cache_peer 192.168.220.2 sibling 3128 3130
dead_peer_timeout 5 seconds

acl AMAZON dstdom_regex -i (.*)s3\.amazonaws\.com
cache_peer_access 192.168.220.2 allow AMAZON

acl RACKSPACE dstdom_regex -i (.*)rackcdn\.com
cache_peer_access 192.168.220.2 allow RACKSPACE

url_rewrite_program /usr/bin/squidguard
url_rewrite_children 32

cache_dir null /tmp
cache_store_log none
cache deny all

acl local_net src 192.168.0.0/16
http_access allow local_net

parent_squid squid.conf

http_port 192.168.220.2:3128
acl main_squid src 192.168.220.1

http_access allow main_squid
http_access allow manager localhost
http_access allow manager main_squid

icp_access allow main_squid

cache_mem 30 GB
maximum_object_size_in_memory 128 MB
cache_dir aufs /squid 400000 16 256
minimum_object_size 16384 KB
maximum_object_size 1024 MB
cache_swap_low 93
cache_swap_high 98

acl PSD urlpath_regex -i \.psd$
cache allow PSD

acl ZIP urlpath_regex -i \.zip$
cache allow ZIP

acl OTHER url_regex -i ^http://*
cache deny OTHER

refresh_pattern \.psd$ 2592000 100 2592000 override-lastmod
override-expire ignore-reload ignore-no-cache
refresh_pattern \.zip$ 2592000 100 2592000 override-lastmod
override-expire ignore-reload ignore-no-cache

All work fine, until I uncomment on main_squid the following line

tcp_outgoing_address yyy.yyy.yyy.239

When I try to download any zip file from amazon I see the following message in cache.log

2013/04/22 01:00:41| TCP connection to 192.168.220.2/3128 failed

If I run tcpdump on yyy.yyy.yyy.239 I see that main_squid trying to connect to parent via external interface without success.

So my question. How may I configure main_squid that it could connect to the parent even with configured tcp_outgoing_address option?

P.S.

# squid -v
Squid Cache: Version 2.6.STABLE21
configure options:  '--host=x86_64-unknown-linux-gnu'
'--build=x86_64-unknown-linux-gnu' '--program-prefix=' '--prefix=/usr'
'--exec-prefix=/usr' '--bindir=/usr/bin' '--sbindir=/usr/sbin'
'--sysconfdir=/etc' '--includedir=/usr/include' '--libdir=/usr/lib64'
'--libexecdir=/usr/libexec' '--sharedstatedir=/var/lib'
'--mandir=/usr/share/man' '--infodir=/usr/share/info'
'--exec_prefix=/usr' '--bindir=/usr/sbin'
'--libexecdir=/usr/lib64/squid' '--localstatedir=/var'
'--datadir=/usr/share' '--sysconfdir=/etc/squid' '--enable-arp-acl'
'--enable-epoll' '--enable-snmp' '--enable-removal-policies=heap,lru'
'--enable-storeio=aufs,coss,diskd,null,ufs' '--enable-ssl'
'--with-openssl=/usr/kerberos' '--enable-delay-pools'
'--enable-linux-netfilter' '--with-pthreads'
'--enable-ntlm-auth-helpers=SMB,fakeauth'
'--enable-external-acl-helpers=ip_user,ldap_group,unix_group,wbinfo_group'
'--enable-auth=basic,digest,ntlm,negotiate'
'--enable-negotiate-auth-helpers=squid_kerb_auth'
'--enable-digest-auth-helpers=password'
'--with-winbind-auth-challenge' '--enable-useragent-log'
'--enable-referer-log' '--disable-dependency-tracking'
'--enable-cachemgr-hostname=localhost' '--enable-underscores'
'--enable-basic-auth-helpers=LDAP,MSNT,NCSA,PAM,SMB,YP,getpwnam,multi-domain-NTLM,SASL'
'--enable-cache-digests' '--enable-ident-lookups'
'--enable-follow-x-forwarded-for' '--enable-wccpv2'
'--enable-fd-config' '--with-maxfd=16384'
'build_alias=x86_64-unknown-linux-gnu'
'host_alias=x86_64-unknown-linux-gnu' 'CFLAGS=-D_FORTIFY_SOURCE=2
-fPIE -Os -g -pipe -fsigned-char' 'LDFLAGS=-pie'

Any help would be highly appreciated

ALex_hha

Asked: 2013-04-24 06:05:04 +0800 CST

How to use tcp_outgoing_address with cache_peer [duplicate]

0

I encountered the problem with configuration 2 squids. I have the following scheme -

http://i.piccy.info/i7/0ecd5cb8276b78975a791c0e5f55ae60/4-57-1543/57409208/squids_schema.jpg

The main idea is to download all files from rackspace and amazon through parent squid and store all files in his cache.

On the main_squid server was configured PBR (iptables + ip route). All packets go through the same channels through which the requests were received

# ip ru sh
0:      from all lookup local
1000:   from all fwmark 0x3e8 lookup ISP1
2000:   from all fwmark 0x7d0 lookup ISP2
3011:   from all fwmark 0xbc3 lookup ISP3
32762:  from xxx.xxx.xxx.62 lookup ISP1
32763:  from yyy.yyy.yyy.239 lookup ISP2
32764:  from zzz.zzz.zzz.10 lookup ISP3
32766:  from all lookup main
32767:  from all lookup default

main_squid squid.conf http_port 192.168.210.1:3128 transparent

cache_peer 192.168.220.2 sibling 3128 3130
dead_peer_timeout 5 seconds

acl AMAZON dstdom_regex -i (.*)s3\.amazonaws\.com
cache_peer_access 192.168.220.2 allow AMAZON

acl RACKSPACE dstdom_regex -i (.*)rackcdn\.com
cache_peer_access 192.168.220.2 allow RACKSPACE

url_rewrite_program /usr/bin/squidguard
url_rewrite_children 32

cache_dir null /tmp
cache_store_log none
cache deny all

acl local_net src 192.168.0.0/16
http_access allow local_net

parent_squid squid.conf

http_port 192.168.220.2:3128
acl main_squid src 192.168.220.1

http_access allow main_squid
http_access allow manager localhost
http_access allow manager main_squid

icp_access allow main_squid

cache_mem 30 GB
maximum_object_size_in_memory 128 MB
cache_dir aufs /squid 400000 16 256
minimum_object_size 16384 KB
maximum_object_size 1024 MB
cache_swap_low 93
cache_swap_high 98

acl PSD urlpath_regex -i \.psd$
cache allow PSD

acl ZIP urlpath_regex -i \.zip$
cache allow ZIP

acl OTHER url_regex -i ^http://*
cache deny OTHER

refresh_pattern \.psd$ 2592000 100 2592000 override-lastmod
override-expire ignore-reload ignore-no-cache
refresh_pattern \.zip$ 2592000 100 2592000 override-lastmod
override-expire ignore-reload ignore-no-cache

All work fine, until I uncomment on main_squid the following line

tcp_outgoing_address yyy.yyy.yyy.239

When I try to download any zip file from amazon I see the following message in cache.log

2013/04/22 01:00:41| TCP connection to 192.168.220.2/3128 failed

If I run tcpdump on yyy.yyy.yyy.239 I see that main_squid trying to connect to parent via external interface without success.

So my question. How may I configure main_squid that it could connect to the parent even with configured tcp_outgoing_address option?

P.S.

squid -v

Squid Cache: Version 2.6.STABLE21
configure options:  '--host=x86_64-unknown-linux-gnu'
'--build=x86_64-unknown-linux-gnu' '--program-prefix=' '--prefix=/usr'
'--exec-prefix=/usr' '--bindir=/usr/bin' '--sbindir=/usr/sbin'
'--sysconfdir=/etc' '--includedir=/usr/include' '--libdir=/usr/lib64'
'--libexecdir=/usr/libexec' '--sharedstatedir=/var/lib'
'--mandir=/usr/share/man' '--infodir=/usr/share/info'
'--exec_prefix=/usr' '--bindir=/usr/sbin'
'--libexecdir=/usr/lib64/squid' '--localstatedir=/var'
'--datadir=/usr/share' '--sysconfdir=/etc/squid' '--enable-arp-acl'
'--enable-epoll' '--enable-snmp' '--enable-removal-policies=heap,lru'
'--enable-storeio=aufs,coss,diskd,null,ufs' '--enable-ssl'
'--with-openssl=/usr/kerberos' '--enable-delay-pools'
'--enable-linux-netfilter' '--with-pthreads'
'--enable-ntlm-auth-helpers=SMB,fakeauth'
'--enable-external-acl-helpers=ip_user,ldap_group,unix_group,wbinfo_group'
'--enable-auth=basic,digest,ntlm,negotiate'
'--enable-negotiate-auth-helpers=squid_kerb_auth'
'--enable-digest-auth-helpers=password'
'--with-winbind-auth-challenge' '--enable-useragent-log'
'--enable-referer-log' '--disable-dependency-tracking'
'--enable-cachemgr-hostname=localhost' '--enable-underscores'
'--enable-basic-auth-helpers=LDAP,MSNT,NCSA,PAM,SMB,YP,getpwnam,multi-domain-NTLM,SASL'
'--enable-cache-digests' '--enable-ident-lookups'
'--enable-follow-x-forwarded-for' '--enable-wccpv2'
'--enable-fd-config' '--with-maxfd=16384'
'build_alias=x86_64-unknown-linux-gnu'
'host_alias=x86_64-unknown-linux-gnu' 'CFLAGS=-D_FORTIFY_SOURCE=2
-fPIE -Os -g -pipe -fsigned-char' 'LDFLAGS=-pie'

ALex_hha

Asked: 2013-01-31 11:16:50 +0800 CST

Caching large amounts of data

3

We need to cache all psd/zip files from our amazone S3 storage. The amount of the cache between 150-200 Gb (we need to store files at least 2 month). Now we use transparent proxy squid without any caching.

So my question: is it possible to configure "main" squid such way that all queries to the s3.amazonaws.com he will proxy to the parent squid. In the parent squid we need to cache all zip and psd files only from s3.amazonaws.com. If there are no such files in the cache the parent squid will be download and put "new" files in the cache.

How will squid works with files 1-2 gb? Is there any limitations?

The main proof of this setup is that all psd/zip files from amazone storage delivered to the local clients with maximal speeds from local cache. Because the same files at the same time used by different employees and we spent a lot of time to download those files by each employees

Can this issue be solved in different ways?

ldapsearch doesn't work wih ldap.google.com via stunnel

How to grant access to an SQS to a specific IAM user

Does FTP service (IIS 7.5) on Windows 2008 R2 support kerberos authentication?

Subversion on 1 Gb uplink

Is it possible to cache subversion?

squid: can't connect to peer with tcp_outgoing_address

How to use tcp_outgoing_address with cache_peer [duplicate]

squid -v

Caching large amounts of data

Can you pass user/pass for HTTP Basic Authentication in URL parameters?

Ping a Specific Port

Check if port is open or closed on a Linux server?

How to automate SSH login with password?

How do I tell Git for Windows where to find my private RSA key?

What's the default superuser username/password for postgres after a new install?

What port does SFTP use?

Command line to list users in a Windows Active Directory group?

What is a Pem file and how does it differ from other OpenSSL Generated Key File Formats?

How to determine if a bash variable is empty?

ALex_hha's questions

squid -v