# Missing fields while Indexing Tomcat access logs data in elasticsearch

**URL:** <https://discuss.elastic.co/t/missing-fields-while-indexing-tomcat-access-logs-data-in-elasticsearch/45580>\
**Category:** Logstash\
**Created:** [March 28, 2016, 11:17am UTC](https://discuss.elastic.co/t/missing-fields-while-indexing-tomcat-access-logs-data-in-elasticsearch/45580 "2016-03-28T11:17:55Z")\
**Posts on this page:** 8\
**Page:** 1

<div class="post-metadata">

**Author:** ![aviral\_srivastava](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/aviral_srivastava/32/98018_2.png) [@aviral\_srivastava](https://discuss.elastic.co/u/aviral_srivastava)\
**Post date:** [March 28, 2016, 11:17am UTC](https://discuss.elastic.co/t/missing-fields-while-indexing-tomcat-access-logs-data-in-elasticsearch/45580/1 "2016-03-28T11:17:55Z")

</div>

I have localhost\_access\_log.2016-03-23.txt file which I am trying to index in elasticsearch via logstash.

Above file looks as follows:-  
0:0:0:0:0:0:0:1 - - [23/Mar/2016:12:20:46 +0530] "GET / HTTP/1.1" 200 11444  
0:0:0:0:0:0:0:1 - admin [23/Mar/2016:12:24:42 +0530] "GET /manager/status HTTP/1.1" 200 6595  
10.76.170.72 - - [23/Mar/2016:15:48:54 +0530] "GET / HTTP/1.1" 200 11444

For that I have created the configuration file as follows:-  
input{  
file{  
type=\>"access-log"  
path=\>"D:/apache-tomcat-7.0.37/logs/localhost\_access\_log.2016-03-23.txt"  
codec=\>multiline{  
negate=\>true  
pattern=\>"(^([0-9A-Fa-f]{1,4}:){7}[0-9A-Fa-f]{1,4}|(\d{1,3}.){3}\d{1,3})"  
what=\>"previous"  
}  
}  
}  
filter{  
if [type] == "access-log" {  
grok {  
match =\> ["message", "%{IP:client} %{NOTSPACE:user} %{NOTSPACE:remoteUser} [%{DATA:timestamp}] %{WORD:method} %{NOTSPACE:request}  
%{NUMBER:status} %{NUMBER:bytes}" ]  
}  
}  
}  
output{  
stdout{}  
elasticsearch{  
hosts=\>"localhost"  
index=\>"access-logs\_20160328"  
}  
}

But when I configure index in Kibana I am not able to see fields like **client** , **user** , etc in kibana

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [March 29, 2016, 5:52am UTC](https://discuss.elastic.co/t/missing-fields-while-indexing-tomcat-access-logs-data-in-elasticsearch/45580/2 "2016-03-29T05:52:16Z")

</div>

Reduce the complexity of the system. Comment out the elasticsearch output and replace it with a `stdout { codec => rubydebug }` output. What do the events look like then?

---

<div class="post-metadata">

**Author:** ![aviral\_srivastava](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/aviral_srivastava/32/98018_2.png) [@aviral\_srivastava](https://discuss.elastic.co/u/aviral_srivastava)\
**Post date:** [March 29, 2016, 12:46pm UTC](https://discuss.elastic.co/t/missing-fields-while-indexing-tomcat-access-logs-data-in-elasticsearch/45580/3 "2016-03-29T12:46:03Z")

</div>

As per your reply, I got the following **output[sample]** :-  
{  
"message" =\> "0:0:0:0:0:0:0:1 - admin [23/Mar/2016:13:30:06 +0530] "GET /host-manager/html HTTP/1.1" 403 286928690:0:0:0:0:0:0:1 - admin [23/Ma  
r/2016:13:30:06 +0530] "GET /host-manager/html HTTP/1.1" 403 2869\r",  
"@version" =\> "1",  
"@timestamp" =\> "2016-03-29T12:43:38.861Z",  
"host" =\> "DIN03000865",  
"path" =\> "D:/apache-tomcat-7.0.37/logs/localhost\_access\_log.2016-03-23.txt",  
"type" =\> "access-log",  
"tags" =\> [  
[0] "\_grokparsefailure"  
]  
}

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [March 29, 2016, 12:51pm UTC](https://discuss.elastic.co/t/missing-fields-while-indexing-tomcat-access-logs-data-in-elasticsearch/45580/4 "2016-03-29T12:51:37Z")

</div>

Okay, so this establishes that the grok expression doesn't match the input. Looking more closely at it,

```
 match => ["message", "%{IP:client} %{NOTSPACE:user} %{NOTSPACE:remoteUser} [%{DATA:timestamp}] %{WORD:method} %{NOTSPACE:request} %{NUMBER:status} %{NUMBER:bytes}" ]

```

it's clear that the part after `%{NOTSPACE:request}` isn't correct since you're ignoring the "HTTP/1.1" that follows. I also hope you're escaping the square brackets around the `timestamp` field with backslashes.

---

<div class="post-metadata">

**Author:** ![aviral\_srivastava](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/aviral_srivastava/32/98018_2.png) [@aviral\_srivastava](https://discuss.elastic.co/u/aviral_srivastava)\
**Post date:** [March 29, 2016, 12:56pm UTC](https://discuss.elastic.co/t/missing-fields-while-indexing-tomcat-access-logs-data-in-elasticsearch/45580/5 "2016-03-29T12:56:10Z")

</div>

I changed the configuration as follows:- Now getting success..  
input{  
file{  
type=\>"access-log"  
path=\>"D:/apache-tomcat-7.0.37/logs/localhost\_access\_log.2016-03-23.txt"  
}  
}  
filter{  
if [type] == "access-log" {  
grok {  
match =\> ["message", "%{COMMONAPACHELOG}"]  
}  
}  
}  
output{  
stdout{ codec=\>rubydebug }  
elasticsearch{  
hosts=\>"localhost"  
index=\>"access-logs\_20160329"  
}  
}

But it would be nice if you would assist me with the problem in my **previous config**

---

<div class="post-metadata">

**Author:** ![aviral\_srivastava](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/aviral_srivastava/32/98018_2.png) [@aviral\_srivastava](https://discuss.elastic.co/u/aviral_srivastava)\
**Post date:** [March 29, 2016, 1:01pm UTC](https://discuss.elastic.co/t/missing-fields-while-indexing-tomcat-access-logs-data-in-elasticsearch/45580/6 "2016-03-29T13:01:58Z")

</div>

I used **grok debugger** for generating the grok pattern **%{COMMONAPACHELOG}**. But I am failing to understand how we create these patterns by self. Is there any reference u can point to for **creating grok patterns** and **patterns** we define under **multiline codec** for beginners.

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [March 29, 2016, 1:54pm UTC](https://discuss.elastic.co/t/missing-fields-while-indexing-tomcat-access-logs-data-in-elasticsearch/45580/7 "2016-03-29T13:54:37Z")

</div>

Have you looked at the grok filter's documentation? It seems to describe things on a rather basic level.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [July 6, 2017, 5:04am UTC](https://discuss.elastic.co/t/missing-fields-while-indexing-tomcat-access-logs-data-in-elasticsearch/45580/8 "2017-07-06T05:04:56Z")

</div>