Create multiple indexes while pushing data from filebeat to elasticsearch

[souravtyson]

Hi team,

I am trying to push the nginx logs using filebeat to elasticsearch.
I wanted to created multiple indexes based on data logged in my file.
For example : i have a field of response time that gets logged into the log file. But for some cases the value comes as "-".
So what i want to do is push the data to two different indexes of elasticsearch based on the field value.

I have gone through the documentation and question blog and had found something useful and tried to do the same. Here is a link https://discuss.elastic.co/t/set-indices-based-on-a-field-value/69579 and https://www.elastic.co/guide/en/beats/filebeat/5.3/elasticsearch-output.html#_indices.

My configuration is shown here :

 output.elasticsearch:
 # Array of hosts to connect to.
   hosts: ["localhost:9200"]
   index: "nginx-access-%{[beat.version]}-%{+yyyy.MM.dd}"
   indices:
     - index: "abc-%{[beat.version]}-%{+yyyy.MM.dd}"
       when.equals:
        nginx.access.upstr_resp_time: "-"

 setup.template.settings:
   index.number_of_shards: 3
   #index.codec: best_compression
   #_source.enabled: false
 setup.template.name: "nginx-access"
 setup.template.pattern: "nginx-access-*"

 setup.template.name: "abc"
 setup.template.pattern: "abc-*"

 setup.template.enable: true
 setup.template.overwrite: false

As you can see i wanted to push the data to index nginx-access-* by default and if the response time have "-" will send to abc-*. In my log file their are 14 data. Now i want to see all those data in my kibana. So when i click on discover i can see only one index being displayed. Is their something wrong i am doing. Please help me with the issue

Note
The index shown on kibana show 13 data and the another 1 data that is left over have response time value as "_" which is not shown on kibana.

[Christian_Dahlqvist]

Filebeat modules use ingest pipelines to parse the data, and these are run in Elasticsearch. The fields you are looking to filter on in the Elasticsearch output does therefore not exist at that point, which is why it fails to do what you are anticipating. To do what you want to do I suspect you may need to modify the ingest pipeline used to change the index name based the contents of the fields you mentioned.

[souravtyson]

can you give one example of how i can do it

[Christian_Dahlqvist]

I do unfortunately not have any suitable example handy. Why not simply delete the response time field if it is set to - and then send it all to the same index?

[souravtyson]

actually i need to push the data so that i can see what had happened to the those kind of request.

So i tried one more option

index: "nginx-access-%{[beat.version]}-%{+yyyy.MM.dd}"
indices:
    - index: "abc-%{[beat.version]}-%{+yyyy.MM.dd}"
      when.contains:
           message: "list"    

this created me two indexes. so why does message works? Is it that the message exist at that point in time