Index a text file into elasticsearch using logstash

Elastic Stack Logstash
1

Is there a way to index a text file which has the data in the following format.

Date Time : 23/05/2016-12.01.38.AM
File    : log
Reason    : parser is not working
Stack :    at java.util.package(Exception e)
            at java.util.package(Exception1 e)
^^-------------------------------------------------------------------^^
Date Time : 23/05/2016-12.01.38.AM
File    : log
Reason    : parser is working
Stack :    at java.lang.package(Exception e)
            at java.lang.package(Exception1 e)
^^-------------------------------------------------------------------^^

Here "Date TIme", "File", "Reason", "Stack" are the fields.
I have tried the following code. But it is not working at all.

input {
  file {
     path => "/Libraries/Documents/InFile.txt"
  }
}
filter{
  grok {
    match => 
	 [ "message", "(?:(Date Time:)%{DATE:date})\n(?:(File:)%{WORD:file})\n(?:(Reason:)%{GREEDYDATA:reason})\n(?:(Stack:)%{GREEDYDATA:stack})"  ]
    }
 }
 output {
      elasticsearch_http {
	host => "localhost"
	index => "fast"
	index_type => "fast"
      }

      stdout   {
          codec => "json"
      }
 }

I want this file to be indexed in Elasticsearch with the fields specified above.
Please help me to index this file.

Regards,
Sanjay Reddy.

2

Hi,
you basically have here a situation of a multiline message, so you should be investigating into the multiline codec (see https://www.elastic.co/guide/en/logstash/current/plugins-codecs-multiline.html ) for indeep details on how to use it.

Then once the multiline had all message in one line, you can use the kv filter (https://www.elastic.co/guide/en/logstash/current/plugins-filters-kv.html) to actually do the split of the line into different fields. This should make all message be processed.

Also I recommend at the beginning to use the stdout with codec rubydebug, and keep the elasticsearch one until you are good with all the output.

hope it helps!

BTW: another way of doing mutliline would be using filebeat multiline support, see https://www.elastic.co/guide/en/beats/filebeat/current/filebeat-getting-started.html for more details.

3

@purbon
As per your suggestion, I have tried the following code. But still nothing is happening.

input {
   file {
       path => "/Libraries/Documents/InFile.txt"
       codec => multiline {
	pattern => "\\^"
	negate => true
	what => previous
  }
}
filter{
  grok {
    match => 
          [ "message", "(?:(Date Time:)%{DATE:date})\n(?:(File:)%{WORD:file})\n(?:(Reason:)%{GREEDYDATA:reason})\n(?:(Stack:)%{GREEDYDATA:stack})"  ]
    add_field => [ "LogTime", "%{date}" ]

    }

 }
 output {
          stdout   {
                 codec => rubydebug
          }
 }

when I run this file, it is stopping at this point. Its been almost 1 hour, nothing is moving forward. Should I change anything in the config file. Please suggest the changes.

4

Can anyone please help me to solve this issue :frowning:

5

Why are you running Logstash 1.3.3, which is over 2 years old?

6

@Christian_Dahlqvist

I have tried with new logstash 2.3.2 file also. But nothing has changed.

This is what happening when I run using the new logstash 2.3.2

Not sure whats happening :frowning:

7

Start Logstash in debug mode so you can see if the file gets picked up. You may also want to set the sincedb_path to /dev/null while you are debugging so the file is reprocessed every time you restart. You probably also want to set start_position to beginning.

8

@Christian_Dahlqvist

Have added sincedb_path and start_position, but still it is not running. Have tried running in debug mode, the following error is coming up

←[32mAdding pattern {"RUBY_LOGGER"=>"[DFEWI], \\[%{TIMESTAMP_ISO8601:timestamp}
#%{POSINT:pid}\\] *%{RUBY_LOGLEVEL:loglevel} -- +%{DATA:progname}: %{GREEDYDATA:
message}", :level=>:info, :file=>"/softwares/ElasticSearch/logstash-2.3.2/vendor
/bundle/jruby/1.9/gems/jls-grok-0.11.2/lib/grok-pure.rb", :line=>"62", :method=>
"add_pattern"}←[0m
←[36mGrok compiled OK {:pattern=>"dd/mm/yyyy-HH:mm:ss", :expanded_pattern=>"dd/m
m/yyyy-HH:mm:ss", :level=>:debug, :file=>"/softwares/ElasticSearch/logstash-2.3.
2/vendor/bundle/jruby/1.9/gems/jls-grok-0.11.2/lib/grok-pure.rb", :line=>"128",
:method=>"compile"}←[0m
←[32mStarting pipeline {:id=>"main", :pipeline_workers=>4, :batch_size=>125, :ba
tch_delay=>5, :max_inflight=>500, :level=>:info, :file=>"/softwares/ElasticSearc
h/logstash-2.3.2/vendor/bundle/jruby/1.9/gems/logstash-core-2.3.2-java/lib/logst
ash/pipeline.rb", :line=>"188", :method=>"start_workers"}←[0m
Pipeline main started {:file=>"/softwares/ElasticSearch/logstash-2.3.2/vendor/bu
 ndle/jruby/1.9/gems/logstash-core-2.3.2-java/lib/logstash/agent.rb", :line=>"465
", :method=>"start_pipeline"}
←[36mPushing flush onto pipeline {:level=>:debug, :file=>"/softwares/ElasticSear
ch/logstash-2.3.2/vendor/bundle/jruby/1.9/gems/logstash-core-2.3.2-java/lib/logs
tash/pipeline.rb", :line=>"458", :method=>"flush"}←[0m
←[36mPushing flush onto pipeline {:level=>:debug, :file=>"/softwares/ElasticSear
ch/logstash-2.3.2/vendor/bundle/jruby/1.9/gems/logstash-core-2.3.2-java/lib/logs
tash/pipeline.rb", :line=>"458", :method=>"flush"}←[0m
←[36m_globbed_files: Q:/sanjay/PhysicianLog.txt: glob is: [] {:level=>:debug, :f
ile=>"/softwares/ElasticSearch/logstash-2.3.2/vendor/bundle/jruby/1.9/gems/filew
atch-0.8.1/lib/filewatch/watch.rb", :line=>"346", :method=>"_globbed_files"}←[0m

←[36mPushing flush onto pipeline {:level=>:debug, :file=>"/softwares/ElasticSear
ch/logstash-2.3.2/vendor/bundle/jruby/1.9/gems/logstash-core-2.3.2-java/lib/logs
tash/pipeline.rb", :line=>"458", :method=>"flush"}←[0m
←[36m_globbed_files: Q:/sanjay/PhysicianLog.txt: glob is: [] {:level=>:debug, :f
ile=>"/softwares/ElasticSearch/logstash-2.3.2/vendor/bundle/jruby/1.9/gems/filew
atch-0.8.1/lib/filewatch/watch.rb", :line=>"346", :method=>"_globbed_files"}←[0m

←[36mPushing flush onto pipeline {:level=>:debug, :file=>"/softwares/ElasticSear
ch/logstash-2.3.2/vendor/bundle/jruby/1.9/gems/logstash-core-2.3.2-java/lib/logs
tash/pipeline.rb", :line=>"458", :method=>"flush"}←[0m
←[36m_globbed_files: Q:/sanjay/PhysicianLog.txt: glob is: [] {:level=>:debug, :f
ile=>"/softwares/ElasticSearch/logstash-2.3.2/vendor/bundle/jruby/1.9/gems/filew
atch-0.8.1/lib/filewatch/watch.rb", :line=>"346", :method=>"_globbed_files"}←[0m

←[36mPushing flush onto pipeline {:level=>:debug, :file=>"/softwares/ElasticSear
ch/logstash-2.3.2/vendor/bundle/jruby/1.9/gems/logstash-core-2.3.2-java/lib/logs
tash/pipeline.rb", :line=>"458", :method=>"flush"}←[0m

Not able to understand anything from this error :frowning:

9

Can anyone please help me in this :frowning:

10

I managed to have what you expect using this configuration :

input {
    file {
        path => [ "/path/to/file.txt" ]
        start_position => "beginning"
        codec => multiline {
            pattern => "[\^\-]+"
            negate => true
            what => previous
        }
    }
}

filter {
    if "multiline" not in [tags] {
        drop {}
    }
    
    grok {
        match => { "message" => "Date Time : %{NOTSPACE:date}\r\nFile    : %{NOTSPACE:file}\r\nReason    : %{DATA:reason}\r\nStack :    %{GREEDYDATA:stack}" }
    }
}

If you don't run under Windows OS, remove \r occurrences.