Home / server / Questions / 72340
Accepted
John Gietzen
Asked: 2009-10-08 14:51:11 +0800 CST

Splunk: How Do I Extract Fields from W3C Extended Format

  • 772

I'm trying to configure Splunk to properly parse the fields of the W3C log format.

Now, I'm running into configuration confusion: where and how do I specify how to split up the log format?

My Inputs.conf looks like this:

[monitor://C:\WINDOWS\system32\LogFiles\W3SVC98989898]
disabled = false
host = mywebsite.net
sourcetype = iis

I tried adding this to my sourcetypes.conf:

[iis_w3c_default]
DELIMS = " "
FIELDS = "date", "time", "cs-method", "cs-uri-stem", "cs-uri-query", "cs-username", "c-ip", "cs-version", "cs(User-Agent)", "cs(Referer)", "sc-status", "sc-bytes"

But using ths as a sourcetype failed to extract any meaningful fields.

I guess I just don't grasp everything that Splunk is doing...

Once i have this working, I plan on reindexing all of my data with the new fields (which will be an ordeal in-and-of-itself).

I really, really want to like Splunk but configuration is just too wonky...

log-files

2 Answers

  1. Best Answer

    You can filter the lines beginning with a # using a transform.

    In props.conf (you can just add the extra line below your existing setup):

    [iis_w3c_default]
TRANSFORMS-blacklist-hash = iis_blacklist_hash

    In transforms.conf:

    [iis_blacklist_hash]
REGEX = ^#
DEST_KEY = queue
FORMAT = nullQueue
    • 1

  2. Here is what I did:

    etc\system\local\props.conf:

    [iis_w3c_default]
REPORT-foobar=iis_w3c_default_extractions

    etc\system\local\transforms.conf:

    [iis_w3c_default_extractions]
DELIMS = " "
FIELDS = "date", "time", "cs-method", "cs-uri-stem", "cs-uri-query", "cs-username", "c-ip", "cs-version", "cs(User-Agent)", "cs(Referer)", "sc-status", "sc-bytes"

    Will update once I figure out how to remove lines that have a "#" symbol in them.

    • 0