Data is sent to created index and the default index

Elastic Stack Logstash
1

I have an interesting issue where data is being sent to the new index and also the old default index. I am using Elasticsearch 2.4 and here is my output file:

output {

    if [type] == "Corporate" {

            elasticsearch {

                    hosts => ["localhost:9200"]
                    index => "corp_windows_events-%{+YYYY-MM-dd}"

            }

    }
    else {

            elasticsearch {

                    hosts => ["localhost:9200"]
            }
    }

}

I checked both indices and there is an identical entry for each record and both records have the same record number.

It is very strange because I did this exact same thing in the lab and everything worked fine. I could even change the index name and update the output to reflect the new index name and it worked like a charm. Maybe there is a slight difference somewhere, but I cannot seem to find it.

Any help would be appreciated.

2

I moved this over to logstash, as I assume this is more of a logstash like question

3

Thank you

4

Update;

I checked out the Elasticsearch logs and there is an error in there with the new index I created. It says "@timestamp" doesn't exist, but @timestamp shows in all the entries when I look at it with Kibana. If I comment out the new index and send everything to the default one, this error no longer ocurrs.

I tried this exact same method of adding an index in my lab environment and I have no issues with this Elasticsearch error and all of my data goes to the defined index and is not duplicated to the default one. I think there may be some mapping issue with the production Elasticstack, but I am not sure on that.

5

Does the configuration file reside in /etc/logstash/conf.d? Are there any other files in that directory?

6

Yes the files reside in /etc/logstash/logstash.conf and there are other files in that directory.

We have it set up where each filter has it's own file. The same for output and input.

example: /etc/logstash/logstash.conf/01-input.conf
then all the filter conf's are numbered 10 through whatever and the output file is 99-output.conf

This helps with adding new things to the environment and makes things a little easier to find

What I found out from other articles is that there was an issue in the Kibana index within /elasticsearch/indices. I got rid of that Kibana index file and everything started working as expected. I lost my visualizations in the process, but that was no big deal to me.

7

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.