KM.'s questions

In the following setup:

Client -> LB -> Varnish

I would like to configure Varnish acls to take certain action based on the Client's IP. The LB sends the client's IP in a variable called "ClientIP", which Varnish can read via req.httpd.ClientIP. I tried this:

acl admin_net {
  "10.10.1.160"/27;
}

sub vcl_deliever {
  if (req.http.ClientIP ~ admin_net) {
  // do something ... 
  }  
}

but the VCL compilation fails with "Expected CSTR got 'admin_net'" (C String?). I can work around this by req.http.ClientIP ~ "10.10.1.*"), but I have to comment out the ACL lines.

Is there another way to get this working with ACLs? I looked at client.ip as well, which is a read-only variable. In the above setup, it contains the LB's IP and not the Client's IP.

This is a follow up from my earlier question on capturing the X-Forwarded-For IP address in across multiple proxies. Now, I'm looking to capture the Client's IP in the application's (Shibboleth's IdP) logs.

The follow is:

Client > Load Balancer > Apache httpd server > Tomcat server (running Shibboleth's IdP)

The variable with the Client's IP at the Load Balancer is ClientIP which I can capture in the web server's logs using LogFormat "%{ClientIP}i ... " and similarly in Tomcat's access logs using these inserver.xml`:

<Valve className="org.apache.catalina.valves.RemoteIpValve" 
remoteIpHeader="ClientIP" 
protocolHeaderHttpsValue="https" />

<Valve className="org.apache.catalina.valves.AccessLogValve" 
directory="logs" 
prefix="localhost_access_log." 
suffix=".txt"
pattern="%{ClientIP}i %h %l %u %t %r %>s %b %{Referer}i %{User-Agent}i" resolveHosts="false"/>

Now, I'm looking to capture this IP in Shibboleth's IdP audit logs. This discussion talks about using mod_rpaf for Apache but doesn't get into the details. I'm hoping to do without installing additional modules.

Looks like I'm missing a couple of (small?) pieces here. Any ideas? Many thanks!

I have a load balancer in front of an Apache httpd server, which in turn is in front of a server running Tomcat6. We're using Tomcat to running Shibboleth's IdP. The follow looks like this:

Client -> Load Balancer -> Apache httpd server (mod_proxy_ajp) -> Tomcat server

I'm looking to pass the client's IP to the Tomcat server. The LB passes the variable ClientIP to the httpd server, which I can parse in httpd's LogFormat as "%{ClientIP}i", but this obviously does not make it to the Tomcat server, instead Tomcat logs the IP of the LB.

I've tried using Tomcat's RemoteIpValve as (in server.xml insider <Engine>) :

> <Valve className="org.apache.catalina.valves.RemoteIpValve"   
> remoteIpHeader="X-Forwarded-For"    protocolHeader="X-Forwarded-Proto"
> protocolHeaderHttpsValue="https" />

hoping that the use of mod_proxy would pass the IP in X-Forwarded-For without success. I've seen posts on mod_rpaf, but I'm hoping to do this without additional apache httpd mods.

I think I'm a couple of pieces away from tying all this together, but stuck in a rut. Any ideas?

Any ideas on why apache (httpd) creates these files in /tmp? I'm on Redhat 5.5 and Apache 2.2, mpm-prefork.

-rw-------. 1 apache   apache       0 Aug 14 12:46 filec1puD5
-rw-------. 1 apache   apache       0 Aug 14 12:46 fileKJqaih
-rw-------. 1 apache   apache       0 Aug 14 12:46 fileB7j9Ws
-rw-------. 1 apache   apache       0 Aug 14 12:46 file1o7MCE
-rw-------. 1 apache   apache       0 Aug 14 12:46 filefqAvjQ
-rw-------. 1 apache   apache       0 Aug 14 12:46 filexjpv01

Sometimes, I see dozens of these, and I always delete them, but haven't found anything on why or how these files are generated in the first place. Error logs look clean, albeit, they're set to Error.

Update: Application is Drupal 7, running on PHP 5.3.2.

I am looking to test if specific ports are on a host are open. I am using:

nc -z host 22
nc -z host 80
nc -z host 443
nc -z host 8080

which works, but it would be nice to use a one-liner like:

nc -z host 22 80 443 8080

which doesn't work.

I would like to avoid the port range nc -z host 22-8080 as noted in the man page, if possible, as there is a large gap in port #s I am looking to check. Also, I don't want to scan every port and be seen as scanning for open ports.

Short of writing a bash loop, what are my options for testing if the ports are open? I have dozens of hosts each with a handful of ports to check.

I've several varnish cache servers (v2.1), and I'm looking to add the hostname in the headers in the response from varnish -- so I know which server is (or not) serving cached pages. Upon googling, I found this snippet, but it gives errors:

sub vcl_deliver {
  if (obj.hits > 0) {
    set resp.http.X-MH-Cache ="HIT " obj.hits " "*
    server.hostname* " " resp.http.Age;
  } else {
    set resp.http.X-MH-Cache ="MISS " *server.hostname* ""
  }
}

I am not tied to this snippet per se. Ideally, I would like to set hostname in the event of a miss or a hit.

I am fairly new to vcl files, so any guidance and assistance would be much appreciated. Many thanks!

KM

I am trying to get Apache Solr to work on Redhat6 and Tomcat6 (using these instructions), but get this error when browsing to the admin section, http://localhost:8080/solr-example/admin:

HTTP Status 404 - missing core name in path

type Status report

message missing core name in path

description The requested resource (missing core name in path) is not available.

http://localhost:8080/solr-example loads fine, with a link to "Solr Admin."

My setup is as follows:

tomcat6: /etc/tomcat6
Solr: /app/solr/example

I have a solr-example.xml in /etc/tomcat6/Catalina/localhost/, which reads:

<?xml version="1.0" encoding="utf-8"?>
<Context docBase="/app/solr/example/apache-solr-3.4.0.war" debug="0" crossContext="true">
  <Environment name="solr/home" type="java.lang.String" value="/app/solr/example" override="true"/>
</Context>

I don't see anything in the logs (/var/log/tomcat6) ... only entires in catalina.out are regarding the starting and stopping of tomcat6.

My questions are:

1.What else do I need to do to get "Solr Admin" to work under Tomcat?

2.Where are these "cores" supposed to be specified? I see an entry in /app/solr/example/solr/solr.xml ?

<solr persistent="false">
      adminPath: RequestHandler path to manage cores.  
        If 'null' (or absent), cores will not be manageable via request handler
      <cores adminPath="/admin/cores" defaultCoreName="collection1">
        <core name="collection1" instanceDir="." />
      </cores>
    </solr>

3.How do I got about ensuring that logs are working correctly? I can't find logs that contain mention of the 404 above.

Update in response to @quanta's comment:

1. Downloaded former (apache-solr-3.4.0.tgz)
2. dataDir was not set, now set to: <dataDir>${solr.data.dir:../solr/data}</dataDir>
3. JAVA_OPTS: /usr/lib/jvm/java/bin/java -classpath :/usr/share/tomcat6/bin/bootstrap.jar:/usr/share/tomcat6/bin/tomcat-juli.jar:/usr/share/java/commons-daemon.jar -Dcatalina.base=/usr/share/tomcat6 -Dcatalina.home=/usr/share/tomcat6 -Djava.endorsed.dirs= -Djava.io.tmpdir=/var/cache/tomcat6/temp -Djava.util.logging.config.file=/usr/share/tomcat6/conf/logging.properties -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager org.apache.catalina.startup.Bootstrap start
4. catalina.out contains no indication of the above error

How is a varnishhist histogram/graph read? I understand that the x-axis is a log scale.

Specifically:

• I see 9 "|"s or cache hits occur around y=1e-4, what does each "|" refer to? Page? File?

• I see fewer hits to the left of these 9 "|"s and some to the right. What are those? Did they take longer (right) and shorter (left) time ... ?

• In the top, left what are 1:2? and n = 134

• Also, I've noticed that the cache hits are always to the left of the cache misses, i.e., they are on the 1e-x side of things, while cache misses are under + exponent. What is the significance of that? Is that time, if so, time of what ... ?

Many Thanks!

x-axis

1:2, n = 134                                                              hostname








                            #
                            #
                            #
                            #
                            #
                   |        #
                   |        #
                   |        #
                   |        #
                   |       ###
                   |       ###
                  ||       ###                              ##
                  |||      ###         ## #   #             ##
                  ||||     ####      #### ## ##  #          ###
+---------+---------+---------+---------+---------+---------+---------+---------+---------
|1e-6     |1e-5     |1e-4     |1e-3     |1e-2     |1e-1     |1e0      |1e1  |1e2

I am trying to compile PHP with LDAP support using these lines:

./configure --prefix=/app/php --with-ldap=/usr     
sudo make     
sudo make install

Everything appears to install fine without issue, and PHP runs fine, but when I run a phpinfo, I don't see LDAP listed.

Also, I read somewhere that I could use an ldap.so file in php's ext directory and reference it from php.ini ... ? How does on generate the ldap.so file for php ... ?

Any ideas? I am on Mac OS X 10.5.8, php 5.2.17.

Update:

I've used these combinations of flags:

--with-ldap
--with-ldap=/usr
--with-ldap=/usr,shared
--with-ldap=shared

Of these, "--with-ldap=shared" produces an ldap.so, which when placed into php's ext directory gives this error in Apache:

PHP Warning:  PHP Startup: Invalid library (maybe not a PHP library) 'ldap.so'  in Unknown on line 0

I've Googled this error, but haven't found much to resolve this issue. Any help is apprecaited!

Many thanks!

KM

I am getting this error when complinng Apache with mod-mime-magic:

--enable-mime-magic ../build_apache.sh: line 22:

--enable-mime-magic: command not found

When I look at the output on screen, I see the following:

checking whether to enable mod_mime_magic... no

despite the fact that --enable-mime-magic is passed in the build script. Also, a little later, I see this:

checking whether to enable mod_mime... yes (default)

I googled this error, but not much comes up. Any ideas on what might be causing this?

Here is the build script:

PATH=/usr/bin:/bin:/lib

export PATH

set -e
set -x

make distclean || true

./configure --prefix=/app/apache \
        --enable-so \
        --enable-cgi \
        --enable-info \
        --enable-rewrite \
        --enable-speling \
        --enable-usertrack \
        --enable-deflate \
#       --enable-ssl \
#       --with-ssl=/usr/bin \
        --enable-mime-magic
#make
#make install
#/app/apache/bin/apachectl restart

This is on Apache 2.2.17 and Redhat (RHEL) 5.5.

Thanks!

KM

We have a CF8 web application that can be run in two modes: Database or Static file. Under heavy load, the Database mode out performs the Staticfile mode noticeably. And I am trying to figure out why.

First some context:

• We run Solaris Zones & Oracle; ColdFusion 8, 32 bit, w/ JRun 4

• In database mode, all data is obtained from an external database server running Oracle 10g.

• In Static file mode, all the data needed for the app to function are generated ahead of time from the DB, and placed on the web server running CF, and called in via CFINCLUDE.

During a load test where virtual users perform per-defined actions repeatedly, in DB mode we were able to run 31600 sessions (in CDATA table), while in static file, it was 25900 sessions. This is over the course of 60 mins. The average time to complete an iteration (login, perform action, logout), in DB mode is 22 seconds, and in static file mode is 33 seconds. Max java heap usage was ~ 800MB in DB mode, ~ 700 MB in static file mode.

Here is why I would expect static file to be better:

• Data is on the same server; only operation is file I/O. The files have already been generated earlier.
• There is no additional network traffic to the DB servers and back.

Any ideas on why this might be? Where can I look (or ask the respective admins - DB, OS, App) to look to see why this might be?

Thanks!

KM

We have a CF8 app that runs for 20-25 minutes before crashing under heavy load ~ 1200 users. This load is generated by our load testing tool: 1200 users ramped up in 5 mins (approx behavior of our users), running for an hour.

We have this app on Solaris 10, Apache 2, JRun 4 and Oracle 10g. Java version is 1.6.

During the initial load tests, the thread dumps pointed to monitor deadlocks that pointed to sessions.

"jrpp-173":
  waiting to lock monitor 0x019fdc60 (object 0x6b893530, a java.util.Hashtable),
  which is held by "scheduler-1"

 "scheduler-1":
  waiting to lock monitor 0x026c3ce0 (object 0x6abe2f20, a coldfusion.monitor.memory.SessionMemoryMonitor$TopMemoryUsedSessions),
  which is held by "jrpp-167"

 "jrpp-167":
  waiting to lock monitor 0x019fdc60 (object 0x6b893530, a java.util.Hashtable),
  which is held by "scheduler-1"

We increased the number of sessions relative to the number of CPUs (48 simultaneous threads against 32 CPUs), and the deadlock went away. While varying the simultaneous threads helped a little bit in terms of response time, the CF server still tanked in 20-25 minutes during all of these tests.

We ran more thread dumps, and saw a thread locking a monitor, for e.g.:

"jrpp-475" prio=3 tid=0x02230800 nid=0x2c5 runnable [0x4397d000]
   java.lang.Thread.State: RUNNABLE
    at java.util.HashMap.getEntry(HashMap.java:347)
    at java.util.HashMap.containsKey(HashMap.java:335)
    at java.util.HashSet.contains(HashSet.java:184)
    at coldfusion.monitor.memory.MemoryTracker.onAddObject(MemoryTracker.java:124)
    at coldfusion.monitor.memory.MemoryTrackerProxy.onReplaceValue(MemoryTrackerProxy.java:598)
    at coldfusion.monitor.memory.MemoryTrackerProxy.onPut(MemoryTrackerProxy.java:510)
    at coldfusion.util.CaseInsensitiveMap.put(CaseInsensitiveMap.java:250)
    at coldfusion.util.FastHashtable.put(FastHashtable.java:43)
    - locked <0x6f7e1a78> (a coldfusion.runtime.Struct)
    at coldfusion.runtime.CfJspPage._arrayset(CfJspPage.java:1027)
    at coldfusion.runtime.CfJspPage._arraySetAt(CfJspPage.java:2117)
    at cfvalidation2ecfc1052964961$funcSETUSERAUDITDATA.runFunction(/app/docs/apply/cfcs/validation.cfc:377)

As you see in the last line above there were several references CFMs and CFCs, and the lines have "cflock" tags, which were scoped to the "application." We (the dev team) then changed them to be scoped to a "name".

After more load tests, there is no locking going on and there no deadlocks, but now the application tanks in 7-10 minutes.

We've gotten system, network and DB reports from the respective admins, and they are not being taxed; even watched the server stats with server monitor, top, prstat, ran sar reports, etc. So we believe it is an issue with the CF server or maybe the JVM.

I am running out of ideas as to what else we can try. Disclaimer: I am not a CF developer or Admin. I am just running the load test, analyzing the reports, threads etc, and sharing the results with the dev and admin teams, and trying the next change, and so on. So far no dice.

Has anyone run into something similar? How did you go about diagnosing and troubleshooting? All thoughts and pointers welcome.

Thank you for your time!

KM

Is it possible to run CF commands inside an HTML file by updating Apache configs/htaccess file? When I searched online, I didn't come across any answers for CF, saw several posting for PHP (perhaps I am not searching for the right terms ... ?).

Specifically, we have CF 6 (I know, don't say it) on Solaris, and Apache 2, and was thinking of using this line:

AddType application/x-httpd-coldfusion .html

so we can use cfinclude inside the HTML files, but this did not work. Your time, ideas and thoughts are much appreciated!

KM

We have apache servers that proxy content from an application server and also from a legacy web server. We enabled caching using our load balancer, but the caching is not behaving like we expect it to. Here is our setup:

We're setting cache headers specifically for the pages served from our application server using:

<Location ~ /(appDir1|appDir2|appDir3)>
Header set Cache-Control max-age=3600,public
</Location>

This works fine.

On the legacy server, the sys admin has set this in his Apache virtual host that runs his legacy site:

Header set Cache-Control max-age=0,private,no-cache

However, our load balancers are still caching the legacy content that we proxy.

When we visit http://appHost/legacyDir1 - we see cached content. When we visit http://legacyHost/legacyDir1 - we see the non-cached content. At least in theory, we're expecting to see non-cached at http://appHost/legacyDir1.

Any ideas what we might be missing ... ?

KM

I am trying to set two different http headers for cache control for two different portions of our site.

These are the lines in my site.conf for the virtual host:

<Location ~ "^/(sub1|sub2|sub3|sub4)">
Header set Cache-Control max-age=60,public
</Location>

and

<Location ~ "^/(sub5|sub6|sub7|sub8)">
Header set Cache-Control no-cache
</Location>

I can see the headers set correctly for the first set of sub pages, but not for the second set.

I feel like I am missing something very small and obvious.

Any ideas?

Thanks!

KM

We have a central httpd.conf and we include confs for various virtual hosts. Until today, we didn't really have a need for "www.subdomain.site.com" domains, only "subdomain.site.com." Now we do, so I am trying to figure out which of these two approaches is better:

[1]. Use a Rewrite:

    RewriteCond %{HTTP_HOST} ^www.subdomain.site.com
    RewriteRule ^(.*)$ http://subdomain.site.com$1 [R=301]

OR

[2]. Use ServerAlais:

    <VirtualHost 111.22.33.44:80>
    ServerName subdomain.site.com
    ServerAlias www.subdomain.site.com
    Include conf/subdomain.conf
    </VirtualHost>

I imagine there is more processing on the Rewrite, but unsure of how ServerAlias would behave in this mix. Does one get better SEO foo than the other?

Any and all ideas welcome!

Thanks,

KM