Stormshadow's questions

I am reading a large file sequentially from the disk and trying to understand the iostat output while the reading is taking place.

• Size of the file : 10 GB
• Read Buffer : 4 KB
• Read ahead (/sys/block/sda/queue/read_ahead_kb) : 128 KB

The iostat output is as follows

Device:         rrqm/s   wrqm/s     r/s     w/s    rMB/s    wMB/s avgrq-sz avgqu-sz await r_await w_await  svctm  %util
sda               0.00     0.00  833.00   14.00   103.88     0.05   251.30     6.07    5.69    2.33 205.71  1.18 100.00

Computing the average size of an I/O request = (rMB/s divided by r/s) gives ~ 128 KB which is the read ahead value. This seems to indicate that while the read system call has specified a 4KB buffer, the actual disk I/O is happening according to the read ahead value.

When I increased the read ahead value to 256KB, the iostat output was as follows

Device:         rrqm/s   wrqm/s     r/s     w/s    rMB/s    wMB/s avgrq-sz avgqu-sz   await r_await w_await  svctm  %util
sda               0.00    28.00  412.00   12.00   102.50     0.05   495.32    10.78   12.15    4.76  265.83   2.36 100.00

Again the average I/O request size was 256 KB matching the read ahead.

This kept up until I set 512 KB as the read ahead value and did not hold up when I moved up to a read ahead value of 1024 KB - the average size of the I/O request was still 512 KB. Increasing max_sectors_kb (maximum amount of data per I/O request) from the default of 512 KB to 1024 KB also did not help here.

Why is this happening - ideally I would like to minimize my read IOPS as much as possible and read larger amount of data per I/O request (larger than 512 KB per request). Additionally, I am hitting 100% disk utilization in all cases - I would want to throttle myself to read at 50-60% disk utilization with good sequential throughput. In short, what are the optimized application/kernel settings for sequential read I/O.

I have two processes running on my box => A and B.

Whenever there is heavy disk I/O activity caused by writing large files to the disk by process A, I see that file writes in process B are stalling intermittently. I noticed that I had the following virtual memory settings on my linux kernel.

vm.dirty_background_ratio = 5  
vm.dirty_background_bytes = 0  
vm.dirty_ratio = 20  
vm.dirty_bytes = 0  
vm.dirty_writeback_centisecs = 500  
vm.dirty_expire_centisecs = 3000  

Documentation states that 'dirty_background_ratio' is the percentage of the memory occupied by dirty pages before the kernel background flusher threads kick into action to flush the bytes to the disk. Given that I have 64 GB of memory, this background flush should kick off when the dirty page cache size reaches ~ 3.2 GB and I see that it is indeed the case when process A is writing to the disk.

However, as soon as this background flush starts, file writes in process B begin to stall intermittently for periods of about 10-20 seconds. Given that this disk flush is happening in the background, why does a write from process B stall given that all writes first go into the page cache and not directly to disk ?

I am a Windows user who wants programs like grep and sed along with other utilities like ls, cut etc in my command prompt. I came across 2 packages which would accomplish this

• UnxUtils
• GNU Coreutils

I would like to know what is the difference between the two both feature wise and more importantly performance wise before I choose between them. From what I have seen Coreutils is a super set of UnxUtils. Is this correct ?

I have 2 users in my AD installation with the same CN

   CN=jack,CN=Users,DC=xyz,DC=com and 
   CN=jack,OU=abc,DC=xyz,DC=com

When I try to authenticate to the AD Server using the Apache Directory Studio client and give the following credentials

User: jack
Password: <password>

the authentication fails. The following credentials work

User: CN=jack,CN=Users,DC=xyz,DC=com
Passwprd: <password>

When I have only a single user CN=tom,DC=xyz,DC=com with a given CN

I am able to login with

User: tom
Password: <password>

without having to specify the entire DN (CN=tom,DC=xyz,DC=com) .

I need to write a module to authenticate users against an AD installation. I have with me only the usernames and passwords and not the fully qualified BASE DNs. How do I do this as the problem it seems happens when there are more than one user with the same CN ?

EDIT: Can I configure the AD server to bind against mail address of the users instead of CN ? This would solve my problem as mail addresses are unique while CN's are not

I have written some code to fetch user information from an Active Directory Server. Suppose the Active Directory Server has nodes, each of which is another Active Directory Installation in a different geographic location. Eg: one AD server in US and another in Australia with a root AD Server in US with the former two as nodes.

Would the filter queries I write for searching users across geographic locations work if I run them on the root AD server ?. The query I use is

(|(objectClass=user)(objectClass=person)(objectClass=inetOrgPerson))

I cannot actually test this scenario but need to know the what will happen here.

I am having trouble trying to connect to a secure OpenLDAP server which I have set up. On running my LDAP client code

java -Djavax.net.debug=ssl LDAPConnector

I get the following exception trace (java version 1.6.0_17)

trigger seeding of SecureRandom
done seeding SecureRandom
%% No cached client session
*** ClientHello, TLSv1
RandomCookie:  GMT: 1256110124 bytes = { 224, 19, 193, 148, 45, 205, 108, 37, 101, 247, 112, 24, 157, 39, 111, 177, 43, 53, 206, 224, 68, 165, 55, 185, 54, 203, 43, 91 }
Session ID:  {}
Cipher Suites: [SSL_RSA_WITH_RC4_128_MD5, SSL_RSA_WITH_RC4_128_SHA, TLS_RSA_WITH_AES_128_CBC_SHA, TLS_DHE_RSA_WITH_AES_128_CBC_SHA, TLS_DHE_DSS_WITH_AES_128_CBC_SHA, SSL_RSA_W
ITH_3DES_EDE_CBC_SHA, SSL_DHE_RSA_WITH_3DES_EDE_CBC_SHA, SSL_DHE_DSS_WITH_3DES_EDE_CBC_SHA, SSL_RSA_WITH_DES_CBC_SHA, SSL_DHE_RSA_WITH_DES_CBC_SHA, SSL_DHE_DSS_WITH_DES_CBC_SH
A, SSL_RSA_EXPORT_WITH_RC4_40_MD5, SSL_RSA_EXPORT_WITH_DES40_CBC_SHA, SSL_DHE_RSA_EXPORT_WITH_DES40_CBC_SHA, SSL_DHE_DSS_EXPORT_WITH_DES40_CBC_SHA]
Compression Methods:  { 0 }
***
Thread-0, WRITE: TLSv1 Handshake, length = 73
Thread-0, WRITE: SSLv2 client hello message, length = 98
Thread-0, received EOFException: error
Thread-0, handling exception: javax.net.ssl.SSLHandshakeException: Remote host closed connection during handshake
Thread-0, SEND TLSv1 ALERT:  fatal, description = handshake_failure
Thread-0, WRITE: TLSv1 Alert, length = 2
Thread-0, called closeSocket()
main, handling exception: javax.net.ssl.SSLHandshakeException: Remote host closed connection during handshake
javax.naming.CommunicationException: simple bind failed: ldap.natraj.com:636 [Root exception is javax.net.ssl.SSLHandshakeException: Remote host closed connection during hands
hake]
        at com.sun.jndi.ldap.LdapClient.authenticate(Unknown Source)
        at com.sun.jndi.ldap.LdapCtx.connect(Unknown Source)
        at com.sun.jndi.ldap.LdapCtx.<init>(Unknown Source)
        at com.sun.jndi.ldap.LdapCtxFactory.getUsingURL(Unknown Source)
        at com.sun.jndi.ldap.LdapCtxFactory.getUsingURLs(Unknown Source)
        at com.sun.jndi.ldap.LdapCtxFactory.getLdapCtxInstance(Unknown Source)
        at com.sun.jndi.ldap.LdapCtxFactory.getInitialContext(Unknown Source)
        at javax.naming.spi.NamingManager.getInitialContext(Unknown Source)
        at javax.naming.InitialContext.getDefaultInitCtx(Unknown Source)
        at javax.naming.InitialContext.init(Unknown Source)
        at javax.naming.InitialContext.<init>(Unknown Source)
        at javax.naming.directory.InitialDirContext.<init>(Unknown Source)
        at LDAPConnector.CallSecureLDAPServer(LDAPConnector.java:43)
        at LDAPConnector.main(LDAPConnector.java:237)
Caused by: javax.net.ssl.SSLHandshakeException: Remote host closed connection during handshake
        at com.sun.net.ssl.internal.ssl.SSLSocketImpl.readRecord(Unknown Source)
        at com.sun.net.ssl.internal.ssl.SSLSocketImpl.performInitialHandshake(Unknown Source)
        at com.sun.net.ssl.internal.ssl.SSLSocketImpl.readDataRecord(Unknown Source)
        at com.sun.net.ssl.internal.ssl.AppInputStream.read(Unknown Source)
        at java.io.BufferedInputStream.fill(Unknown Source)
        at java.io.BufferedInputStream.read1(Unknown Source)
        at java.io.BufferedInputStream.read(Unknown Source)
        at com.sun.jndi.ldap.Connection.run(Unknown Source)
        at java.lang.Thread.run(Unknown Source)
Caused by: java.io.EOFException: SSL peer shut down incorrectly
        at com.sun.net.ssl.internal.ssl.InputRecord.read(Unknown Source)
        ... 9 more

I am able to connect to the same secure LDAP server however if I use another version of java (1.6.0_14)

I have created and installed the server certificates in the cacerts of both the JRE's as mentioned in this guide --> OpenLDAP with SSL

When I run ldapsearch -x on the server I get

# extended LDIF
#
# LDAPv3
# base <dc=localdomain> (default) with scope subtree
# filter: (objectclass=*)
# requesting: ALL
#

# localdomain
dn: dc=localdomain
objectClass: top
objectClass: dcObject
objectClass: organization
o: localdomain
dc: localdomain

# admin, localdomain
dn: cn=admin,dc=localdomain
objectClass: simpleSecurityObject
objectClass: organizationalRole
cn: admin
description: LDAP administrator

# search result
search: 2
result: 0 Success

# numResponses: 3
# numEntries: 2

On running openssl s_client -connect ldap.natraj.com:636 -showcerts , I obtain the self signed certificate.

My slapd.conf file is as follows

#######################################################################
# Global Directives:

# Features to permit
#allow bind_v2

# Schema and objectClass definitions
include         /etc/ldap/schema/core.schema
include         /etc/ldap/schema/cosine.schema
include         /etc/ldap/schema/nis.schema
include         /etc/ldap/schema/inetorgperson.schema

# Where the pid file is put. The init.d script
# will not stop the server if you change this.
pidfile         /var/run/slapd/slapd.pid

# List of arguments that were passed to the server
argsfile        /var/run/slapd/slapd.args

# Read slapd.conf(5) for possible values
loglevel        none

# Where the dynamically loaded modules are stored
modulepath      /usr/lib/ldap
moduleload      back_hdb

# The maximum number of entries that is returned for a search operation
sizelimit 500

# The tool-threads parameter sets the actual amount of cpu's that is used
# for indexing.
tool-threads 1

#######################################################################
# Specific Backend Directives for hdb:
# Backend specific directives apply to this backend until another
# 'backend' directive occurs
backend         hdb

#######################################################################
# Specific Backend Directives for 'other':
# Backend specific directives apply to this backend until another
# 'backend' directive occurs
#backend                <other>

#######################################################################
# Specific Directives for database #1, of type hdb:
# Database specific directives apply to this databasse until another
# 'database' directive occurs
database        hdb

# The base of your directory in database #1
suffix          "dc=localdomain"

# rootdn directive for specifying a superuser on the database. This is needed
# for syncrepl.
rootdn          "cn=admin,dc=localdomain"

# Where the database file are physically stored for database #1
directory       "/var/lib/ldap"

# The dbconfig settings are used to generate a DB_CONFIG file the first
# time slapd starts.  They do NOT override existing an existing DB_CONFIG
# file.  You should therefore change these settings in DB_CONFIG directly
# or remove DB_CONFIG and restart slapd for changes to take effect.

# For the Debian package we use 2MB as default but be sure to update this
# value if you have plenty of RAM
dbconfig set_cachesize 0 2097152 0

# Sven Hartge reported that he had to set this value incredibly high
# to get slapd running at all. See http://bugs.debian.org/303057 for more
# information.

# Number of objects that can be locked at the same time.
dbconfig set_lk_max_objects 1500
# Number of locks (both requested and granted)
dbconfig set_lk_max_locks 1500
# Number of lockers
dbconfig set_lk_max_lockers 1500

# Indexing options for database #1
index           objectClass eq

# Save the time that the entry gets modified, for database #1
lastmod         on

# Checkpoint the BerkeleyDB database periodically in case of system
# failure and to speed slapd shutdown.
checkpoint      512 30

# Where to store the replica logs for database #1
# replogfile    /var/lib/ldap/replog
# The userPassword by default can be changed
# by the entry owning it if they are authenticated.
# Others should not be able to see it, except the
# admin entry below
# These access lines apply to database #1 only
access to attrs=userPassword,shadowLastChange
        by dn="cn=admin,dc=localdomain" write
        by anonymous auth
        by self write
        by * none

# Ensure read access to the base for things like
# supportedSASLMechanisms.  Without this you may
# have problems with SASL not knowing what
# mechanisms are available and the like.
# Note that this is covered by the 'access to *'
# ACL below too but if you change that as people
# are wont to do you'll still need this if you
# want SASL (and possible other things) to work
# happily.
access to dn.base="" by * read

# The admin dn has full write access, everyone else
# can read everything.
access to *
        by dn="cn=admin,dc=localdomain" write
        by * read

# For Netscape Roaming support, each user gets a roaming
# profile for which they have write access to
#access to dn=".*,ou=Roaming,o=morsnet"
#        by dn="cn=admin,dc=localdomain" write
#        by dnattr=owner write
#######################################################################
# Specific Directives for database #2, of type 'other' (can be hdb too):
# Database specific directives apply to this databasse until another
# 'database' directive occurs
#database        <other>

# The base of your directory for database #2
#suffix         "dc=debian,dc=org"

#######################################################################
# SSL:
# Uncomment the following lines to enable SSL and use the default
# snakeoil certificates.
#TLSCertificateFile /etc/ssl/certs/ssl-cert-snakeoil.pem
#TLSCertificateKeyFile /etc/ssl/private/ssl-cert-snakeoil.key

TLSCipherSuite TLS_RSA_AES_256_CBC_SHA
TLSCACertificateFile /etc/ldap/ssl/server.pem
TLSCertificateFile /etc/ldap/ssl/server.pem
TLSCertificateKeyFile /etc/ldap/ssl/server.pem

My ldap.conf file is

#
# LDAP Defaults
#

# See ldap.conf(5) for details
# This file should be world readable but not world writable.

HOST ldap.natraj.com
PORT 636

BASE    dc=localdomain
URI     ldaps://ldap.natraj.com
TLS_CACERT    /etc/ldap/ssl/server.pem
TLS_REQCERT allow

#SIZELIMIT      12
#TIMELIMIT      15
#DEREF          never

Why is it that I can connect to the same server using one version of JRE while I cannot with another ?