# Logstash Index creation and parsing custom app log

**URL:** <https://discuss.elastic.co/t/logstash-index-creation-and-parsing-custom-app-log/57711>\
**Category:** Logstash\
**Created:** [August 10, 2016, 2:17pm UTC](https://discuss.elastic.co/t/logstash-index-creation-and-parsing-custom-app-log/57711 "2016-08-10T14:17:44Z")\
**Posts on this page:** 16\
**Page:** 1

<div class="post-metadata">

**Author:** ![jukkendar](https://avatars.discourse-cdn.com/v4/letter/j/ec9cab/32.png) [@jukkendar](https://discuss.elastic.co/u/jukkendar)\
**Post date:** [August 10, 2016, 2:17pm UTC](https://discuss.elastic.co/t/logstash-index-creation-and-parsing-custom-app-log/57711/1 "2016-08-10T14:17:44Z")

</div>

I have installed Logstash 2.3.1, elastic search 2.3.1 and Kibana 4.5.0 in windows 2008 server as services.  
I can launch Kibana but unable to create Index in Kibana , Refer the log below.  
I have verified the below config using command "logstash -f logstash.conf --configtest" and getting Configuration ok. Please suggest why index is not getting created and shown in Kibana.

input {  
file {

```
  path => "E:/logstash/New/Logsetup/log/system.day20160518.log"
  start_position => "beginning"
  type => "logs"
}

```

}

filter {  
grok {  
match =\> { "message" =\> "%{DATESTAMP:StartTime} %{WORD:code1}-%{WORD:code2} %{WORD:code3} %{WORD:code4} : %{GREEDYDATA:log\_message}"}  
add\_field =\> ["StartTime", "%{@timestamp}"]  
}

}

output {  
elasticsearch {  
type =\> "logs"  
hosts =\> ["localhost:9200"]  
index =\> "logstash-%{+YYYY.MM.dd}"  
}  
}

---

<div class="post-metadata">

**Author:** ![theuntergeek](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/theuntergeek/32/44961_2.png) [@theuntergeek](https://discuss.elastic.co/u/theuntergeek)\
**Post date:** [August 10, 2016, 2:41pm UTC](https://discuss.elastic.co/t/logstash-index-creation-and-parsing-custom-app-log/57711/2 "2016-08-10T14:41:24Z")

</div>

This line should not be in your elasticsearch output block configuration, and should actually result in an error (at the very least, a deprecation message):

> [@jukkendar](#):
>
> ```auto
> type => "logs"
> 
> ```

And these lines are unnecessary as they are the default values:

> [@jukkendar](#):
>
> ```auto
> hosts => ["localhost:9200"]    
> index => "logstash-%{+YYYY.MM.dd}"
> 
> ```

Also, if you ran with this statement:

> [@jukkendar](#):
>
> path =\> "E:/logstash/New/Logsetup/log/system.day20160518.log"  
> start\_position =\> "beginning"

more than one time, it will not re-read the file, even with `start_position => "beginning"`. This is because there is a `sincedb` file created that wants to tail and resume where it left off. For re-reading old log files that will not receive any more data, you can get them to reread by adding `sincedb_path => NUL` (`NUL` is like `/dev/null` for Windows machines).

---

<div class="post-metadata">

**Author:** ![jukkendar](https://avatars.discourse-cdn.com/v4/letter/j/ec9cab/32.png) [@jukkendar](https://discuss.elastic.co/u/jukkendar)\
**Post date:** [August 10, 2016, 3:40pm UTC](https://discuss.elastic.co/t/logstash-index-creation-and-parsing-custom-app-log/57711/3 "2016-08-10T15:40:33Z")

</div>

> [@theuntergeek](#):
>
> sincedb\_path =\>

Thanks for your reply. I have altered the config file as below but no go.

input {  
file {

```
  path => "E:/logstash/New/Logsetup/log/system.day20160518.log"
  sincedb_path => "/dev/null"
  start_position => "beginning"
  type => "logs"
}

```

}

filter {  
grok {  
match =\> { "message" =\> "%{DATESTAMP:StartTime} %{WORD:code1}-%{WORD:code2} %{WORD:code3} %{WORD:code4} : %{GREEDYDATA:log\_message}"}  
add\_field =\> ["StartTime", "%{@timestamp}"]  
}

}

output {  
elasticsearch {

```
}

```

}

---

<div class="post-metadata">

**Author:** ![theuntergeek](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/theuntergeek/32/44961_2.png) [@theuntergeek](https://discuss.elastic.co/u/theuntergeek)\
**Post date:** [August 10, 2016, 3:52pm UTC](https://discuss.elastic.co/t/logstash-index-creation-and-parsing-custom-app-log/57711/4 "2016-08-10T15:52:49Z")

</div>

> [@jukkendar](#):
>
> sincedb\_path =\> "/dev/null"

As mentioned, `/dev/null` is for UNIX systems. You have a Windows path, so you should be using `sincedb_path => NUL`.

You should also test the output by changing the output block to be:

```auto
output {
  stdout { codec => rubydebug } 
# elasticsearch { }
}

```

With the `elasticsearch` output disabled, by being commented. You will be able to see output at the command-line this way. If you do not get any output, that will also indicate why the index is not being created.

---

<div class="post-metadata">

**Author:** ![jukkendar](https://avatars.discourse-cdn.com/v4/letter/j/ec9cab/32.png) [@jukkendar](https://discuss.elastic.co/u/jukkendar)\
**Post date:** [August 10, 2016, 4:05pm UTC](https://discuss.elastic.co/t/logstash-index-creation-and-parsing-custom-app-log/57711/5 "2016-08-10T16:05:54Z")

</div>

> [@theuntergeek](#):
>
> stdout { codec =\> rubydebug }

When I try debugging in command line , getting the below message.

E:\logstash\New\logstash-2.3.1\bin\>logstash -f E:\logstash\New\logstash-2.3.1\bi  
n logstash.json --debug

T\_HANDLER:nagios\_type}: %{DATA:nagios\_hostname};%{DATA:nagios\_service};%{DATA:na  
gios\_state};%{DATA:nagios\_statelevel};%{DATA:nagios\_event\_handler\_name}", :level  
=\>:info, :file=\>"/logstash/New/logstash-2.3.1/vendor/bundle/jruby/1.9/gems/jls-g  
rok-0.11.2/lib/grok-pure.rb", :line=\>"62", :method=\>"add\_pattern"}←[0m  
←[32mAdding pattern {"NAGIOS\_HOST\_EVENT\_HANDLER"=\>"%{NAGIOS\_TYPE\_HOST\_EVENT\_HAND  
LER:nagios\_type}: %{DATA:nagios\_hostname};%{DATA:nagios\_state};%{DATA:nagios\_sta  
televel};%{DATA:nagios\_event\_handler\_name}", :level=\>:info, :file=\>"/logstash/Ne  
w/logstash-2.3.1/vendor/bundle/jruby/1.9/gems/jls-grok-0.11.2/lib/grok-pure.rb",  
:line=\>"62", :method=\>"add\_pattern"}←[0m  
←[32mAdding pattern {"NAGIOS\_TIMEPERIOD\_TRANSITION"=\>"%{NAGIOS\_TYPE\_TIMEPERIOD\_T  
RANSITION:nagios\_type}: %{DATA:nagios\_service};%{DATA:nagios\_unknown1};%{DATA:na  
gios\_unknown2}", :level=\>:info, :file=\>"/logstash/New/logstash-2.3.1/vendor/bund  
le/jruby/1.9/gems/jls-grok-0.11.2/lib/grok-pure.rb", :line=\>"62", :method=\>"add\_  
pattern"}←[0m  
←[32mAdding pattern {"NAGIOS\_EC\_LINE\_DISABLE\_SVC\_CHECK"=\>"%{NAGIOS\_TYPE\_EXTERNAL  
\_COMMAND:nagios\_type}: %{NAGIOS\_EC\_DISABLE\_SVC\_CHECK:nagios\_command};%{DATA:nagi  
os\_hostname};%{DATA:nagios\_service}", :level=\>:info, :file=\>"/logstash/New/logst  
ash-2.3.1/vendor/bundle/jruby/1.9/gems/jls-grok-0.11.2/lib/grok-pure.rb", :line=

> "62", :method=\>"add\_pattern"}←[0m  
> ←[32mAdding pattern {"NAGIOS\_EC\_LINE\_DISABLE\_HOST\_CHECK"=\>"%{NAGIOS\_TYPE\_EXTERNA  
> L\_COMMAND:nagios\_type}: %{NAGIOS\_EC\_DISABLE\_HOST\_CHECK:nagios\_command};%{DATA:na  
> gios\_hostname}", :level=\>:info, :file=\>"/logstash/New/logstash-2.3.1/vendor/bund  
> le/jruby/1.9/gems/jls-grok-0.11.2/lib/grok-pure.rb", :line=\>"62", :method=\>"add\_  
> pattern"}←[0m  
> ←[32mAdding pattern {"NAGIOS\_EC\_LINE\_ENABLE\_SVC\_CHECK"=\>"%{NAGIOS\_TYPE\_EXTERNAL\_  
> COMMAND:nagios\_type}: %{NAGIOS\_EC\_ENABLE\_SVC\_CHECK:nagios\_command};%{DATA:nagios  
> \_hostname};%{DATA:nagios\_service}", :level=\>:info, :file=\>"/logstash/New/logstas  
> h-2.3.1/vendor/bundle/jruby/1.9/gems/jls-grok-0.11.2/lib/grok-pure.rb", :line=\>"  
> 62", :method=\>"add\_pattern"}←[0m  
> ←[32mAdding pattern {"NAGIOS\_EC\_LINE\_ENABLE\_HOST\_CHECK"=\>"%{NAGIOS\_TYPE\_EXTERNAL  
> \_COMMAND:nagios\_type}: %{NAGIOS\_EC\_ENABLE\_HOST\_CHECK:nagios\_command};%{DATA:nagi  
> os\_hostname}", :level=\>:info, :file=\>"/logstash/New/logstash-2.3.1/vendor/bundle  
> /jruby/1.9/gems/jls-grok-0.11.2/lib/grok-pure.rb", :line=\>"62", :method=\>"add\_pa  
> ttern"}←[0m

---

<div class="post-metadata">

**Author:** ![theuntergeek](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/theuntergeek/32/44961_2.png) [@theuntergeek](https://discuss.elastic.co/u/theuntergeek)\
**Post date:** [August 10, 2016, 4:22pm UTC](https://discuss.elastic.co/t/logstash-index-creation-and-parsing-custom-app-log/57711/6 "2016-08-10T16:22:57Z")

</div>

This seems incomplete. The `Adding pattern` lines aren't helpful and do not indicate any errors. Can you filter those out?

---

<div class="post-metadata">

**Author:** ![theuntergeek](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/theuntergeek/32/44961_2.png) [@theuntergeek](https://discuss.elastic.co/u/theuntergeek)\
**Post date:** [August 10, 2016, 4:24pm UTC](https://discuss.elastic.co/t/logstash-index-creation-and-parsing-custom-app-log/57711/7 "2016-08-10T16:24:49Z")

</div>

> [@jukkendar](#):
>
> match =\> { "message" =\> "%{DATESTAMP:StartTime}

and in the same grok block:

> [@jukkendar](#):
>
> add\_field =\> ["StartTime", "%{@timestamp}"]

Is StartTime getting its data from grok, or from `@timestamp`?

---

<div class="post-metadata">

**Author:** ![jukkendar](https://avatars.discourse-cdn.com/v4/letter/j/ec9cab/32.png) [@jukkendar](https://discuss.elastic.co/u/jukkendar)\
**Post date:** [August 10, 2016, 4:59pm UTC](https://discuss.elastic.co/t/logstash-index-creation-and-parsing-custom-app-log/57711/8 "2016-08-10T16:59:56Z")

</div>

> [@theuntergeek](#):
>
> match =\> { "message" =\> "%{DATESTAMP:StartTime}

App log file will look like , so used grok timefield based on log.

2016/08/09 00:43:38.169 S-300000 text text : ssytem error.

Updated conf without grok and getting the below error

input {  
file {

```
  path => "E:/logstash/New/Logsetup/log/system.day20160518.log"
  sincedb_path => "Nul"
  start_position => "beginning"
  type => "logs"
}

```

}

output {  
stdout { codec =\> rubydebug }  
#elasticsearch { }  
}

←[36mconfig LogStash::Inputs::File/@delimiter = "\n" {:level=\>:debug, :file=\>"/l  
ogstash/New/logstash-2.3.1/vendor/bundle/jruby/1.9/gems/logstash-core-2.3.1-java  
/lib/logstash/config/mixin.rb", :line=\>"153", :method=\>"config\_init"}←[0m  
←[36mconfig LogStash::Inputs::File/@ignore\_older = 86400 {:level=\>:debug, :file=

> "/logstash/New/logstash-2.3.1/vendor/bundle/jruby/1.9/gems/logstash-core-2.3.1-  
> java/lib/logstash/config/mixin.rb", :line=\>"153", :method=\>"config\_init"}←[0m  
> ←[36mconfig LogStash::Inputs::File/@close\_older = 3600 {:level=\>:debug, :file=\>"  
> /logstash/New/logstash-2.3.1/vendor/bundle/jruby/1.9/gems/logstash-core-2.3.1-ja  
> va/lib/logstash/config/mixin.rb", :line=\>"153", :method=\>"config\_init"}←[0m  
> ←[36mPlugin not defined in namespace, checking for plugin file {:type=\>"output",  
> :name=\>"stdout", :path=\>"logstash/outputs/stdout", :level=\>:debug, :file=\>"/log  
> stash/New/logstash-2.3.1/vendor/bundle/jruby/1.9/gems/logstash-core-2.3.1-java/l  
> ib/logstash/plugin.rb", :line=\>"76", :method=\>"lookup"}←[0m  
> ←[32mstarting agent {:level=\>:info, :file=\>"/logstash/New/logstash-2.3.1/vendor/  
> bundle/jruby/1.9/gems/logstash-core-2.3.1-java/lib/logstash/agent.rb", :line=\>"2  
> 07", :method=\>"execute"}←[0m

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [August 10, 2016, 5:33pm UTC](https://discuss.elastic.co/t/logstash-index-creation-and-parsing-custom-app-log/57711/9 "2016-08-10T17:33:42Z")

</div>

If the modification time of system.day20160518.log is older than 24 hours you need to adjust the file input's `ignore_older` option.

---

<div class="post-metadata">

**Author:** ![jukkendar](https://avatars.discourse-cdn.com/v4/letter/j/ec9cab/32.png) [@jukkendar](https://discuss.elastic.co/u/jukkendar)\
**Post date:** [August 10, 2016, 5:55pm UTC](https://discuss.elastic.co/t/logstash-index-creation-and-parsing-custom-app-log/57711/10 "2016-08-10T17:55:59Z")

</div>

I tried after changing the DATESTAMP with current date as a file name and also the DATESTAMP  
inside the log file also.

Even though i am getting a same error as i posted below:

E:\logstash\New\logstash-2.3.1\bin\>logstash -f logstash.json --debug  
io/console not supported; tty will not be manipulated  
←[36mReading config file {:config\_file=\>"E:/logstash/New/logstash-2.3.1/bin/logs  
tash.json", :level=\>:debug, :file=\>"/logstash/New/logstash-2.3.1/vendor/bundle/j  
ruby/1.9/gems/logstash-core-2.3.1-java/lib/logstash/config/loader.rb", :line=\>"6  
9", :method=\>"local\_config"}←[0m  
←[36mPlugin not defined in namespace, checking for plugin file {:type=\>"input",  
:name=\>"file", :path=\>"logstash/inputs/file", :level=\>:debug, :file=\>"/logstash/  
New/logstash-2.3.1/vendor/bundle/jruby/1.9/gems/logstash-core-2.3.1-java/lib/log  
stash/plugin.rb", :line=\>"76", :method=\>"lookup"}←[0m  
←[36mPlugin not defined in namespace, checking for plugin file {:type=\>"codec",  
:name=\>"plain", :path=\>"logstash/codecs/plain", :level=\>:debug, :file=\>"/logstas  
h/New/logstash-2.3.1/vendor/bundle/jruby/1.9/gems/logstash-core-2.3.1-java/lib/l  
ogstash/plugin.rb", :line=\>"76", :method=\>"lookup"}←[0m  
←[36mconfig LogStash::Codecs::Plain/@charset = "UTF-8" {:level=\>:debug, :file=\>"  
/logstash/New/logstash-2.3.1/vendor/bundle/jruby/1.9/gems/logstash-core-2.3.1-ja  
va/lib/logstash/config/mixin.rb", :line=\>"153", :method=\>"config\_init"}←[0m  
←[36mconfig LogStash::Inputs::File/@path = ["E:/logstash/New/Logsetup/log/system  
.day20160810.log"] {:level=\>:debug, :file=\>"/logstash/New/logstash-2.3.1/vendor/  
bundle/jruby/1.9/gems/logstash-core-2.3.1-java/lib/logstash/config/mixin.rb", :l  
ine=\>"153", :method=\>"config\_init"}←[0m  
←[36mconfig LogStash::Inputs::File/@sincedb\_path = "Nul" {:level=\>:debug, :file=

> "/logstash/New/logstash-2.3.1/vendor/bundle/jruby/1.9/gems/logstash-core-2.3.1-  
> java/lib/logstash/config/mixin.rb", :line=\>"153", :method=\>"config\_init"}←[0m  
> ←[36mconfig LogStash::Inputs::File/@start\_position = "beginning" {:level=\>:debug  
> , :file=\>"/logstash/New/logstash-2.3.1/vendor/bundle/jruby/1.9/gems/logstash-cor  
> e-2.3.1-java/lib/logstash/config/mixin.rb", :line=\>"153", :method=\>"config\_init"  
> }←[0m

---

<div class="post-metadata">

**Author:** ![theuntergeek](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/theuntergeek/32/44961_2.png) [@theuntergeek](https://discuss.elastic.co/u/theuntergeek)\
**Post date:** [August 10, 2016, 8:24pm UTC](https://discuss.elastic.co/t/logstash-index-creation-and-parsing-custom-app-log/57711/11 "2016-08-10T20:24:53Z")

</div>

I wouldn't even bother with grok before ensuring that data flows. You can comment out the entire filter block with `#` on each line. If you don't see data flowing, chasing grok is not going to help.

Try the `ignore_older` suggestion from @magnusbaeck. I'm also not sure whether `Nul` and `NUL` are the same to Windows, so I suggest making `Nul` into `NUL`.

---

<div class="post-metadata">

**Author:** ![jukkendar](https://avatars.discourse-cdn.com/v4/letter/j/ec9cab/32.png) [@jukkendar](https://discuss.elastic.co/u/jukkendar)\
**Post date:** [August 15, 2016, 9:25am UTC](https://discuss.elastic.co/t/logstash-index-creation-and-parsing-custom-app-log/57711/12 "2016-08-15T09:25:26Z")

</div>

I have my log file name like 00331\_ln2\_systemday20160814 , so when creating a index I want to add 2 new fields like store no from file name "00331" and Laneno as "ln2". Refer my grok file below.  
Help me to add add\_fields.

filter {  
grok {  
match =\> { "message" =\> "%{DATESTAMP:timestamp} %{WORD:code1}-%{WORD:code2} %{WORD:code3} %{WORD:code4} : %{GREEDYDATA:log\_message}" }  
break\_on\_match =\> false  
}  
}

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [August 15, 2016, 10:39am UTC](https://discuss.elastic.co/t/logstash-index-creation-and-parsing-custom-app-log/57711/13 "2016-08-15T10:39:56Z")

</div>

You'll find the path to the file in the `path` field, so just add a grok filter that extracts the desired fields from that field.

---

<div class="post-metadata">

**Author:** ![jukkendar](https://avatars.discourse-cdn.com/v4/letter/j/ec9cab/32.png) [@jukkendar](https://discuss.elastic.co/u/jukkendar)\
**Post date:** [August 15, 2016, 11:04am UTC](https://discuss.elastic.co/t/logstash-index-creation-and-parsing-custom-app-log/57711/14 "2016-08-15T11:04:42Z")

</div>

Thanks magnus for quick reply. Do you mean the below line is correct? . Path has full file path, so is there any param for filename?

Please share me some example.

add\_field =\> {"Store\_num","%{GREEDYDATA}/%{GREEDYDATA:path}"}

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [August 15, 2016, 7:41pm UTC](https://discuss.elastic.co/t/logstash-index-creation-and-parsing-custom-app-log/57711/15 "2016-08-15T19:41:27Z")

</div>

> Do you mean the below line is correct?

No. For starters it's not a grok filter.

> Path has full file path, so is there any param for filename?

```plaintext
grok {
  match => {
    "path" => "/(?<filename>[^/]+)$"
  }
}

```

I gave this exact example and explained how it worked in another thread just a few days ago.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [July 6, 2017, 4:43am UTC](https://discuss.elastic.co/t/logstash-index-creation-and-parsing-custom-app-log/57711/16 "2017-07-06T04:43:28Z")

</div>