Nothing is getting indexed

Elastic Stack Logstash
#1

Hi,

I am new to ELK. I had some log files indexed. Then I deleted the indexes using XDELETE and _all.

Now when i run logstash, and try to see if the files get indexed again. it does not show any index.

Please help.

The Conf. file looks like this

input
{
file {
path => ['/var/log/logstash/ETLLogs/*.log']
sincedb_path => "dev/null"
codec => multiline
{
pattern => "^%{TIMESTAMP_ISO8601}"
what => "previous"
negate => "true"
}
}
}
filter
{
grok {
match => {"message"=> "%{WORD:Exception}%{GREEDYDATA:shortmessage}"}
tag_on_failure => []
}
date {
match => ["event_timestamp", "yyyy/MM/dd HH:mm:ss"]
}
}

output
{
elasticsearch {
index => "%{logstash-}%{+yyyy-mm-dd}"
}
stdout {
codec => rubydebug
}
}

#2

If that isn't a typo it'd be why, you need to delete this file or fix the typo.

#3

Thankyou.

I added the following lines to my config file:

start_position => "beginning"

it worked

#4

Right now I have another issue:

My log file is of the pattern:

Job STARTED at 2014/02/19 19:03:48 (jobId=20140216_190624_s2aZn, jobExecutionId=20140219190348_rmzXR)

for this my filter looks like this:

filter
{
grok {

    	match => {"message" => "[\n]### Job STARTED at %{GREEDYDATA} (jobId=%{GREEDYDATA:jobid}[\n]"}
	}

date {
match => ["event_timestamp", "yyyy/MM/dd HH:mm:ss"]
}

}

When i run the logstash, i get the below error:
'end pattern with unmatched parentheses'

Need some help around it real quick,

#5

I have another question.

In my log file i have the following pattern:
2014-02-19 19:13:12|oAmwic|20140219190348_rmzXR|20140219190348_rmzXR|ADR_TST|Dim_State_Trade|Default|6|Java Exception|tMSSqlOutput_1|java.sql.BatchUpdateException:

2014-02-19 19:13:12|oAmwic|20140219190348_rmzXR|20140219190348_rmzXR|916|ADR_TST|Dim_State_Trade|__a8TIEepEeG9yIseFOxTIA|0.1|Default||end|failure|80029
java.lang.RuntimeException: Child job running failed

I have defined the filter as below:

filter
{
grok {
break_on_match => false
match => ["message", ["java[.]%{GREEDYDATA}[.]%{GREEDYDATA:error_desc}Exception[:]"]]
add_tag => ["%{error_desc}"]
}
}

Problem:

Only the first pattern is captured.

The second error, though of the same pattern is not captured.

#6