# Index a text file into elasticsearch using logstash

**URL:** <https://discuss.elastic.co/t/index-a-text-file-into-elasticsearch-using-logstash/51584>\
**Category:** Logstash\
**Created:** [June 1, 2016, 3:34pm UTC](https://discuss.elastic.co/t/index-a-text-file-into-elasticsearch-using-logstash/51584 "2016-06-01T15:34:42Z")\
**Posts on this page:** 11\
**Page:** 1

<div class="post-metadata">

**Author:** ![sanju1323](https://avatars.discourse-cdn.com/v4/letter/s/5daacb/32.png) [@sanju1323](https://discuss.elastic.co/u/sanju1323)\
**Post date:** [June 1, 2016, 3:34pm UTC](https://discuss.elastic.co/t/index-a-text-file-into-elasticsearch-using-logstash/51584/1 "2016-06-01T15:34:42Z")

</div>

Is there a way to index a text file which has the data in the following format.

```
Date Time : 23/05/2016-12.01.38.AM
File : log
Reason : parser is not working
Stack : at java.util.package(Exception e)
            at java.util.package(Exception1 e)
^^-------------------------------------------------------------------^^
Date Time : 23/05/2016-12.01.38.AM
File : log
Reason : parser is working
Stack : at java.lang.package(Exception e)
            at java.lang.package(Exception1 e)
^^-------------------------------------------------------------------^^

```

Here "Date TIme", "File", "Reason", "Stack" are the fields.  
I have tried the following code. But it is not working at all.

```
input {
  file {
     path => "/Libraries/Documents/InFile.txt"
  }
}
filter{
  grok {
    match => 
	 ["message", "(?:(Date Time:)%{DATE:date})\n(?:(File:)%{WORD:file})\n(?:(Reason:)%{GREEDYDATA:reason})\n(?:(Stack:)%{GREEDYDATA:stack})"]
    }
 }
 output {
      elasticsearch_http {
	host => "localhost"
	index => "fast"
	index_type => "fast"
      }

      stdout {
          codec => "json"
      }
 }

```

I want this file to be indexed in Elasticsearch with the fields specified above.  
Please help me to index this file.

Regards,  
Sanjay Reddy.

---

<div class="post-metadata">

**Author:** ![purbon](https://avatars.discourse-cdn.com/v4/letter/p/edb3f5/32.png) [@purbon](https://discuss.elastic.co/u/purbon)\
**Post date:** [June 1, 2016, 4:38pm UTC](https://discuss.elastic.co/t/index-a-text-file-into-elasticsearch-using-logstash/51584/2 "2016-06-01T16:38:30Z")

</div>

Hi,  
you basically have here a situation of a multiline message, so you should be investigating into the multiline codec (see [https://www.elastic.co/guide/en/logstash/current/plugins-codecs-multiline.html](https://www.elastic.co/guide/en/logstash/current/plugins-codecs-multiline.html) ) for indeep details on how to use it.

Then once the multiline had all message in one line, you can use the kv filter ([https://www.elastic.co/guide/en/logstash/current/plugins-filters-kv.html](https://www.elastic.co/guide/en/logstash/current/plugins-filters-kv.html)) to actually do the split of the line into different fields. This should make all message be processed.

Also I recommend at the beginning to use the stdout with codec rubydebug, and keep the elasticsearch one until you are good with all the output.

hope it helps!

BTW: another way of doing mutliline would be using filebeat multiline support, see [https://www.elastic.co/guide/en/beats/filebeat/current/filebeat-getting-started.html](https://www.elastic.co/guide/en/beats/filebeat/current/filebeat-getting-started.html) for more details.

---

<div class="post-metadata">

**Author:** ![sanju1323](https://avatars.discourse-cdn.com/v4/letter/s/5daacb/32.png) [@sanju1323](https://discuss.elastic.co/u/sanju1323)\
**Post date:** [June 2, 2016, 10:22am UTC](https://discuss.elastic.co/t/index-a-text-file-into-elasticsearch-using-logstash/51584/3 "2016-06-02T10:22:41Z")

</div>

@purbon  
As per your suggestion, I have tried the following code. But still nothing is happening.

```
input {
   file {
       path => "/Libraries/Documents/InFile.txt"
       codec => multiline {
	pattern => "\\^"
	negate => true
	what => previous
  }
}
filter{
  grok {
    match => 
          ["message", "(?:(Date Time:)%{DATE:date})\n(?:(File:)%{WORD:file})\n(?:(Reason:)%{GREEDYDATA:reason})\n(?:(Stack:)%{GREEDYDATA:stack})"]
    add_field => ["LogTime", "%{date}"]

    }

 }
 output {
          stdout {
                 codec => rubydebug
          }
 }

```

when I run this file, it is stopping at this point. Its been almost 1 hour, nothing is moving forward. Should I change anything in the config file. Please suggest the changes.

![](https://us1.discourse-cdn.com/elastic/original/2X/a/a30a974c9cd17880497529032866a2740234d7c3.png)

---

<div class="post-metadata">

**Author:** ![sanju1323](https://avatars.discourse-cdn.com/v4/letter/s/5daacb/32.png) [@sanju1323](https://discuss.elastic.co/u/sanju1323)\
**Post date:** [June 7, 2016, 7:39am UTC](https://discuss.elastic.co/t/index-a-text-file-into-elasticsearch-using-logstash/51584/4 "2016-06-07T07:39:46Z")

</div>

Can anyone please help me to solve this issue ☹

---

<div class="post-metadata">

**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\
**Post date:** [June 7, 2016, 7:43am UTC](https://discuss.elastic.co/t/index-a-text-file-into-elasticsearch-using-logstash/51584/5 "2016-06-07T07:43:53Z")

</div>

Why are you running Logstash 1.3.3, which is over 2 years old?

---

<div class="post-metadata">

**Author:** ![sanju1323](https://avatars.discourse-cdn.com/v4/letter/s/5daacb/32.png) [@sanju1323](https://discuss.elastic.co/u/sanju1323)\
**Post date:** [June 7, 2016, 7:46am UTC](https://discuss.elastic.co/t/index-a-text-file-into-elasticsearch-using-logstash/51584/6 "2016-06-07T07:46:43Z")

</div>

@Christian_Dahlqvist

I have tried with new logstash 2.3.2 file also. But nothing has changed.

This is what happening when I run using the new logstash 2.3.2  
 ![](https://us1.discourse-cdn.com/elastic/original/2X/8/8ed0018b90b5dfb297d8b6df1f130e94a901bc38.png)

Not sure whats happening ☹

---

<div class="post-metadata">

**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\
**Post date:** [June 7, 2016, 8:41am UTC](https://discuss.elastic.co/t/index-a-text-file-into-elasticsearch-using-logstash/51584/7 "2016-06-07T08:41:34Z")

</div>

Start Logstash in [debug mode](https://www.elastic.co/guide/en/logstash/current/command-line-flags.html) so you can see if the file gets picked up. You may also want to set the [sincedb\_path](https://www.elastic.co/guide/en/logstash/current/plugins-inputs-file.html#plugins-inputs-file-sincedb_path) to `/dev/null` while you are debugging so the file is reprocessed every time you restart. You probably also want to set `start_position` to `beginning`.

---

<div class="post-metadata">

**Author:** ![sanju1323](https://avatars.discourse-cdn.com/v4/letter/s/5daacb/32.png) [@sanju1323](https://discuss.elastic.co/u/sanju1323)\
**Post date:** [June 7, 2016, 10:47am UTC](https://discuss.elastic.co/t/index-a-text-file-into-elasticsearch-using-logstash/51584/8 "2016-06-07T10:47:53Z")

</div>

@Christian_Dahlqvist

Have added sincedb\_path and start\_position, but still it is not running. Have tried running in debug mode, the following error is coming up

```
←[32mAdding pattern {"RUBY_LOGGER"=>"[DFEWI], \\[%{TIMESTAMP_ISO8601:timestamp}
#%{POSINT:pid}\\] *%{RUBY_LOGLEVEL:loglevel} -- +%{DATA:progname}: %{GREEDYDATA:
message}", :level=>:info, :file=>"/softwares/ElasticSearch/logstash-2.3.2/vendor
/bundle/jruby/1.9/gems/jls-grok-0.11.2/lib/grok-pure.rb", :line=>"62", :method=>
"add_pattern"}←[0m
←[36mGrok compiled OK {:pattern=>"dd/mm/yyyy-HH:mm:ss", :expanded_pattern=>"dd/m
m/yyyy-HH:mm:ss", :level=>:debug, :file=>"/softwares/ElasticSearch/logstash-2.3.
2/vendor/bundle/jruby/1.9/gems/jls-grok-0.11.2/lib/grok-pure.rb", :line=>"128",
:method=>"compile"}←[0m
←[32mStarting pipeline {:id=>"main", :pipeline_workers=>4, :batch_size=>125, :ba
tch_delay=>5, :max_inflight=>500, :level=>:info, :file=>"/softwares/ElasticSearc
h/logstash-2.3.2/vendor/bundle/jruby/1.9/gems/logstash-core-2.3.2-java/lib/logst
ash/pipeline.rb", :line=>"188", :method=>"start_workers"}←[0m
Pipeline main started {:file=>"/softwares/ElasticSearch/logstash-2.3.2/vendor/bu
 ndle/jruby/1.9/gems/logstash-core-2.3.2-java/lib/logstash/agent.rb", :line=>"465
", :method=>"start_pipeline"}
←[36mPushing flush onto pipeline {:level=>:debug, :file=>"/softwares/ElasticSear
ch/logstash-2.3.2/vendor/bundle/jruby/1.9/gems/logstash-core-2.3.2-java/lib/logs
tash/pipeline.rb", :line=>"458", :method=>"flush"}←[0m
←[36mPushing flush onto pipeline {:level=>:debug, :file=>"/softwares/ElasticSear
ch/logstash-2.3.2/vendor/bundle/jruby/1.9/gems/logstash-core-2.3.2-java/lib/logs
tash/pipeline.rb", :line=>"458", :method=>"flush"}←[0m
←[36m_globbed_files: Q:/sanjay/PhysicianLog.txt: glob is: [] {:level=>:debug, :f
ile=>"/softwares/ElasticSearch/logstash-2.3.2/vendor/bundle/jruby/1.9/gems/filew
atch-0.8.1/lib/filewatch/watch.rb", :line=>"346", :method=>"_globbed_files"}←[0m

←[36mPushing flush onto pipeline {:level=>:debug, :file=>"/softwares/ElasticSear
ch/logstash-2.3.2/vendor/bundle/jruby/1.9/gems/logstash-core-2.3.2-java/lib/logs
tash/pipeline.rb", :line=>"458", :method=>"flush"}←[0m
←[36m_globbed_files: Q:/sanjay/PhysicianLog.txt: glob is: [] {:level=>:debug, :f
ile=>"/softwares/ElasticSearch/logstash-2.3.2/vendor/bundle/jruby/1.9/gems/filew
atch-0.8.1/lib/filewatch/watch.rb", :line=>"346", :method=>"_globbed_files"}←[0m

←[36mPushing flush onto pipeline {:level=>:debug, :file=>"/softwares/ElasticSear
ch/logstash-2.3.2/vendor/bundle/jruby/1.9/gems/logstash-core-2.3.2-java/lib/logs
tash/pipeline.rb", :line=>"458", :method=>"flush"}←[0m
←[36m_globbed_files: Q:/sanjay/PhysicianLog.txt: glob is: [] {:level=>:debug, :f
ile=>"/softwares/ElasticSearch/logstash-2.3.2/vendor/bundle/jruby/1.9/gems/filew
atch-0.8.1/lib/filewatch/watch.rb", :line=>"346", :method=>"_globbed_files"}←[0m

←[36mPushing flush onto pipeline {:level=>:debug, :file=>"/softwares/ElasticSear
ch/logstash-2.3.2/vendor/bundle/jruby/1.9/gems/logstash-core-2.3.2-java/lib/logs
tash/pipeline.rb", :line=>"458", :method=>"flush"}←[0m 

```

Not able to understand anything from this error ☹

---

<div class="post-metadata">

**Author:** ![sanju1323](https://avatars.discourse-cdn.com/v4/letter/s/5daacb/32.png) [@sanju1323](https://discuss.elastic.co/u/sanju1323)\
**Post date:** [June 14, 2016, 11:33am UTC](https://discuss.elastic.co/t/index-a-text-file-into-elasticsearch-using-logstash/51584/9 "2016-06-14T11:33:39Z")

</div>

Can anyone please help me in this ☹

---

<div class="post-metadata">

**Author:** ![fbaligand](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/fbaligand/32/5657_2.png) [@fbaligand](https://discuss.elastic.co/u/fbaligand)\
**Post date:** [July 10, 2016, 6:47pm UTC](https://discuss.elastic.co/t/index-a-text-file-into-elasticsearch-using-logstash/51584/10 "2016-07-10T18:47:32Z")

</div>

I managed to have what you expect using this configuration :

```
input {
    file {
        path => ["/path/to/file.txt"]
        start_position => "beginning"
        codec => multiline {
            pattern => "[\^\-]+"
            negate => true
            what => previous
        }
    }
}

filter {
    if "multiline" not in [tags] {
        drop {}
    }
    
    grok {
        match => { "message" => "Date Time : %{NOTSPACE:date}\r\nFile : %{NOTSPACE:file}\r\nReason : %{DATA:reason}\r\nStack : %{GREEDYDATA:stack}" }
    }
}

```

If you don't run under Windows OS, remove `\r` occurrences.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [July 6, 2017, 4:48am UTC](https://discuss.elastic.co/t/index-a-text-file-into-elasticsearch-using-logstash/51584/11 "2017-07-06T04:48:44Z")

</div>