I am newbie to ELK and need to integrate my logs as below to elasticsearch from logstash.
Can some help me by providing log patterrns for below log format
#Filelog
2017-06-05 00:03:03 INFO HeartBeatDetailsTimerTask:94 - Live Integration List is empty
2017-06-05 00:03:03 INFO HeartBeatDetailsTimerTask:96 - HeartBeatDetails end
2017-06-05 00:07:03 INFO HeartBeatDetailsTimerTask:94 - Live Integration List is empty
2017-06-05 00:07:03 INFO HeartBeatDetailsTimerTask:96 - HeartBeatDetails end
#webservicelog
2017-06-02 12:36:47 INFO PortalUpload- Upload Medical Record webservice invoked from partner Id medall
2017-06-02 12:36:48 DEBUG PortalUpload- ROR response Json:{"ref_id":351883,"original_file_path":"files/13701_351883_1496387026446.jpg","image_path":"preview/13701_351883_1496387026446.jpg","thumbnail_file_path":"thumbnails/13701_351883_1496387026446_thumbnail.jpg","healthhubId":"LJWA-9923","status":"success"}
2017-06-02 12:36:48 INFO DatabaseConnection- Get Mongo DB Database connection...
#Emaillog
2017-06-03 00:03:03 INFO HeartBeatDetailsTimerTask:94 - Live Integration List is empty
2017-06-03 00:03:03 INFO HeartBeatDetailsTimerTask:96 - HeartBeatDetails end
2017-06-03 00:07:03 INFO HeartBeatDetailsTimerTask:94 - Live Integration List is empty
Thanks 
You can use the grok constructor site as a guide to creating grok expressions that match a particular line of input.
2017-06-05 00:03:03 INFO HeartBeatDetailsTimerTask:94 - Live Integration List is empty
Is this correct for above input
grok pattern= %{TIMESTAMP_ISO8601:timestamp} %{LOGLEVEL:logLevel}%{SPACE}%{GREEDYDATA:message}
That looks okay, although I'd extract HeartBeatDetailsTimerTask:94 to separate fields too. Don't forget to set the overwrite option so that you can overwrite the existing message value.
mean
grok {
match => [ %{TIMESTAMP_ISO8601:timestamp} %{LOGLEVEL:logLevel}%{SPACE}%{GREEDYDATA:message} ]
overwrite => [ "message" ]
}
I didn't get "extract HeartBeatDetailsTimerTask:94 to separate fields too" Can you please elaborate in details or with examples
Thanks,
mean
Yes.
I didn't get "extract HeartBeatDetailsTimerTask:94 to separate fields too" Can you please elaborate in details or with examples
In addition to the timestamp, loglevel, and message fields I'd want to have one field with "HeartBeatDetailsTimerTask" and one field with "94".
For
2017-06-02 12:36:47 INFO PortalUpload - Upload Medical Record webservice invoked from partner Id medall
> \A%{TIMESTAMP_ISO8601:timestamp}\s+%{LOGLEVEL:loglevel}\s+(?<logger>(?:-[a-zA-Z0-9-]+\.)*[A-Za-z0-9$]+)\s
Is working fine.
But Original Log is
2017-06-02 12:36:47 INFO PortalUpload- Upload Medical Record webservice invoked from partner Id medall
note: PortalUpload- no space between eiphens and after space message starts, in that case how to correct my above grok filters
Use ? to indicate that the preceding token is optional, i.e. foo ?bar matches both "foo bar" and "foobar".
Can you please correct my grok filter for expected original logs format
2017-06-02 12:36:47 INFO PortalUpload- Upload Medical Record webservice invoked from partner Id medall
Is this below correct
filter{
grok {
match => [ "message", %{TIMESTAMP_ISO8601:timestamp} %{LOGLEVEL:loglevel} %{GREEDYDATA:message} (?<logger>(?:[a-zA-Z0-9]+\.)*[-A-Za-z0-9$]+)
overwrite => [ "message" ]
}
}
Does it work and give the expected results? Then it's probably correct.
Does the logger really come after the message?
grok {
match => [ "message", %{TIMESTAMP_ISO8601:timestamp} %{LOGLEVEL:loglevel} (?<logger>(?:[a-zA-Z0-9]+\.)*[-A-Za-z0-9$]+) %{GREEDYDATA:message}
overwrite => [ "message" ]
No Message comes last, reordered,
I am using 5.4 version ELK, where throwing error as below
bin/logstash -f /etc/logstash/conf.d/filelog.conf
WARNING: Could not find logstash.yml which is typically located in $LS_HOME/config or /etc/logstash. You can specify the path using --path.settings. Continuing using the defaults
Could not find log4j2 configuration at path //usr/share/logstash/config/log4j2.properties. Using default config which logs to console
19:04:25.883 [LogStash::Runner] FATAL logstash.runner - Logstash could not be started because there is already another instance using the configured data directory. If you wish to run multiple instances, you must change the "path.data" setting.
I have set values in elasticsearch.yml
path.data: /var/lib/logstash
path.config: /etc/logstash/conf.d
Still throwing error also tried
bin/logstash --path.config /etc/logstash/conf.d/filelog.conf
WARNING: Could not find logstash.yml which is typically located in $LS_HOME/config or /etc/logstash. You can specify the path using --path.settings. Continuing using the defaults
Could not find log4j2 configuration at path //usr/share/logstash/config/log4j2.properties. Using default config which logs to console
19:08:53.683 [LogStash::Runner] FATAL logstash.runner - Logstash could not be started because there is already
another instance using the configured data directory. If you wish to run multiple instances, you must change the "path.data" setting.
Let me know how to know proceed further
Thanks,
Pandiyan
I have set values in elasticsearch.yml
elasticsearch.yml?
You can override the data path setting with the --path.data command line option. If you run Logstash as a service but also want to run it interactively like in this case this is a reasonable workaround.
It was logstash.yml, typo error 
bin/logstash --path.config /etc/logstash/conf.d/filelog.conf
got same error
Got Error as below
bin/logstash --path.config /etc/logstash/conf.d/filelog.conf
WARNING: Could not find logstash.yml which is typically located in $LS_HOME/config or /etc/logstash. You can specify the path using --path.settings. Continuing using the defaults
Could not find log4j2 configuration at path //usr/share/logstash/config/log4j2.properties. Using default config which logs to console
12:36:59.314 [LogStash::Runner] ERROR logstash.agent - Cannot create pipeline {:reason=>"Expected one of #, \", ', -, [, { at line 11, column 29 (byte 167) after filter{\n \n grok { \n match => [ \"message\", "}
Below is the file.conf file
input {
file {
path => "/var/log/file.log"
codec => json
start_position => "beginning"
}
}
filter{
grok {
match => [ "message", %{TIMESTAMP_ISO8601:timestamp} %{LOGLEVEL:loglevel} (?<logger>(?:[a-zA-Z0-9]+\.)*[-A-Za-z0-9$]+) %{GREEDYDATA:message}
overwrite => [ "message" ]
}
}
output {
elasticsearch {
codec => rubydebug
host => ['localhost:9200']
index => 'filelogs-%{+YYYY.MM.dd}'
}
}
As I think I've said in another thread already you need to surround your grok expression with quotes.