Unable to change index off default logstash-*

Elastic Stack Logstash
1

Hello,

I am trying to get logstash to not use the default index but it isn't working for me. I have a JSON template that is successful updating in ES but logstash keeps trying to use the default index and parsing errors occur. I have set the index on the ES output but I am not sure why it is skipping it.

Error:

{:timestamp=>"2016-04-20T22:46:04.554000-0400", :message=>"Failed action. ", :status=>400, :action=>["index", {:_id=>nil, :_index=>"logstash-2016.04.21", :_type=>"logs", :_routing=>nil}, #<LogStash::Event:0x180d5390 @metadata_accessors=#<LogStash::Util::Accessors:0x406ba261 @store={"path"=>"/events-test.csv"}, @lut={"[path]"=>[{"path"=>"/events-test.csv"}, "path"]}>, @cancelled=false, @data={"message"=>"

Input and output:

input {
file {
path => "/events-test.csv"
start_position => "beginning"
type => "firepower"
}
}

output {
elasticsearch {
hosts => ["localhost:9200"]
template => "/firepower.json"
index => "firepower"
template_name => "firepower"
manage_template => true
template_overwrite => true
workers => 1
}

I deleted all my templates to double check that the template was being created. It is indeed being done.

curl -XGET 'http://localhost:9200/_template/*?pretty'

{
"firepower" : {
"order" : 0,
"template" : "firepower*",
"settings" : { },
"mappings" : {
"firepower" : {
"properties" : {
"security_zone_egress_uuid" : {
"index" : "not_analyzed",
"type" : "string"
},
<! more in here !>

I also created the index:

curl 'localhost:9200/_cat/indices?v'
health status index pri rep docs.count docs.deleted store.size pri.store.size
yellow open .kibana 1 1 1 0 3.1kb 3.1kb
yellow open firepower 5 1 0 0 795b 795b

Thoughts??

2

Wild guess: You have another outdated file in /etc/logstash/conf.d that lacks the index => "firepower" option for its output. If that's not it, start Logstash with --debug to see exactly what configuration it loads.

3

There are no other configuration files and looking at the logs it is very quiet.

So I booted it up in debug and went through the mountain of text and found this...

:response=>{"create"=>{"_index"=>"firepower", "_type"=>"firepower", "_id"=>"AVQ6WKjSHNuH89_3Iwfv", "status"=>400, "error"=>{"type"=>"mapper_parsing_exception", "reason"=>"failed to parse [@timestamp]", "caused_by"=>{"type"=>"illegal_argument_exception", "reason"=>"Invalid format: "2016-04-21T19:42:48.652Z" is malformed at "-04-21T19:42:48.652Z""}}}}, :level=>:warn, :file=>"logstash/outputs/elasticsearch/common.rb", :line=>"119", :method=>"submit"}

I do have a date filter setup, it must be wrong.

    date {
    match => [ "date", "ISO8601" ]
    target => "date"
}

}

from the debug:

Date/@match = ["date", "ISO8601"]", :level=>:debug,

I just read a post you had two months back and I am going to be deleting the timestamp:

date {
locale => "en"
match => [ "timestamp", "dd/MMM/yyyy:HH:mm:ss Z" ]
remove_field => ["timestamp"]
}

4

No, I don't think that's it. It's the @timestamp field it's complaining about so the presence of timestamp shouldn't matter (although it's still a good idea too remove it—why keep multiple timestamp fields?). I think the problem is related to the fact that since the index is named firepower Logstash's index template that maps @timestamp as a ISO8601 timestamp doesn't apply.

5

I removed the date timestamp (as above). I then checked the debug to make sure that there where no errors this time. No errors. In fact, everything looked promising. I then checked the indexes in ES and I got a full throated chuckle (or a crazy persons laugh - haven't decided yet -).

kyle@ELK:~$ curl 'localhost:9200/_cat/indices?v'
health status index pri rep docs.count docs.deleted store.size pri.store.size
yellow open .kibana 1 1 1 0 3.1kb 3.1kb
yellow open firepower 5 1 0 0 795b 795b
yellow open logstash-2016.04.21 5 1 1 0 14.3kb 14.3kb

Nowhere do I have logstash-%{+YYYY.MM.dd}.... It almost seems that by removing the timestamp it no longer selects that index.

UPDATE: restarted the service and tailed the logs instead of 'sudo /opt/logstash/bin/logstash agent --debug --log test.log --config /etc/logstash/conf.d/10-estreamer.conf'

and I get this:

{"create"=>{"_index"=>"firepower", "_type"=>"firepower", "_id"=>"AVQ7x_NbHNuH89_3Iwf1", "status"=>400, "error"=>{"type"=>"mapper_parsing_exception", "reason"=>"failed to parse [@timestamp]", "caused_by"=>{"type"=>"illegal_argument_exception", "reason"=>"Invalid format: "2016-04-22T02:23:59.664Z" is malformed at "-04-22T02:23:59.664Z""}}}}, :level=>:warn}

Now I am even more confused.

6

Fixed. Changed the format in the JSON template.