Logstash Index Name as "%{[@metadata][beat]}-%{[@metadata][version]}"

[robscott27]

How can I get the index name for netflow to be set correctly upon creation?

Some background:
Prior to using Netflow, I've been using Logstash to send beats data into Elasticsearch - Filebeat, Auditbeat, Metricbeat, etc - with no problem.

When I went to use Netflow, since it's a module, I had it configured in the logstash.yml file. And that worked just fine once I got Logstash to load the netflow template and dashboards. The problem there was that the data from the beats stopping coming in, which I discovered was due to it skipping over the config files in /etc/logstash/conf.d/ and only listening on the Netflow UDP port, which is one number higher than the beats logstash port. Which got me thinking about multiple pipelines.

So then I started down the multiple pipelines road. But I couldn't find any documentation about getting modules and pipelines to work together. I never did find any official documentation, but I did find this post which describes getting the netflow module to work with the rest of the conf files in the /etc/logstash/conf.d/ area.

The only difference I made from that above post was to comment out the "output" section from the 01-netflow-input.conf file and just send it through the the output the rest of the beats were/are going through which is 99-output-elasticsearch.conf:

output {
elasticsearch {
hosts => "localhost:9200"
manage_template => false
index => "%{[@metadata][beat]}-%{[@metadata][version]}-%{+YYYY.MM.dd}"
}
}

This sort of works as it does allow for the beats data to go to their respective indexes (on a TCP port), but the netflow data (coming in on a UDP port) ends up in an index named "%{[@metadata][beat]}-%{[@metadata][version]}-2019.06.04" - it's picking up the date format at the end, but not the name and version.

I suspect this is due to Netflow not being a "beat" but rather a Logstash module.

Then I thought about adding the output data back to the end of the netflow.conf file

output {
elasticsearch {
hosts => "localhost:9200"
manage_template => false
index => "netflow-%{+YYYY.MM.dd}" }
stdout { codec => rubydebug }
}

but when I do THAT I get the following error:

[2019-06-04T10:44:25,414][WARN ][logstash.outputs.elasticsearch] Could not index event to Elasticsearch. {:status=>400, :action=>["index", {:_id=>nil, :_index=>"netflow-2019.06.04", :_type=>"_doc", :routing=>nil}, #LogStash::Event:0x2f6e36e3], :response=>{"index"=>{"_index"=>"netflow-2019.06.04", "_type"=>"_doc", "_id"=>"_OvyImsBHEmPc_lPyJnn", "status"=>400, "error"=>{"type"=>"mapper_parsing_exception", "reason"=>"failed to parse field [host] of type [keyword] in document with id '_OvyImsBHEmPc_lPyJnn'", "caused_by"=>{"type"=>"illegal_state_exception", "reason"=>"Can't get text on a START_OBJECT at 1:1372"}}}}}

Some of the data still makes it into the index though.

So:

1. Netflow module config in logstash.yml = Netflow data, but no beats data
2. Netflow config in 01-netflow-input.conf = Netflow data and beats data but netflow index isn't named correctly
3. Netflow config in 01-netflow-input.conf WITH output section = only some of the data and the rest generates a field parsing error

I feel like I'm close to getting this to work, but I'm missing something silly. It's seems it's just the index naming issue from Logstash that's holding me up.

It could also be I've been staring at the config files too long.

Anyone run into something like this before?

Many thanks in advance!

[robscott27]

I got everything working. Apparently, I just had to step away from it for a bit.

After all of the above, I decided to try a combined if/else conditional output statement. What I ended up with (after some trial and error) was the following:

output {
if [type] == "netflow" {
elasticsearch {
hosts => ["localhost:9200"]
manage_template => false
index => "netflow-%{+YYYY.MM.dd}"
}
} else {
elasticsearch {
hosts => ["localhost:9200"]
manage_template => false
index => "%{[@metadata][beat]}-%{[@metadata][version]}-%{+YYYY.MM.dd}"
}
}
}

Now, I know we're not supposed to to use the [type] field any longer. However, it was the easiest thing to check for that gave me a working if/else statement. It was also part of the default template that Logstash 7.1.1 loaded during the setup process of templates and dashboards.

For those of you playing along at home, here's the rough process I went through to get this working.

1. Follow the post I linked to above and have Logstash load the Netflow template, dashboards, and visualizations (this itself took a few a tries). Also copy over the netflow.conf.erb file into your logstash/conf.d directory as a .conf file.
2. Alter the input settings in that conf file you just copied and change to your desired port, worker number, etc.
3. Update the variable listings in that same conf file to point to direct file paths. Again, this is listed in the post I linked to.
4. For me, I also removed the entire output area at the end of that netflow conf file.
5. In whatever file you used in the /etc/logstash/conf.d/ directory to dictate output to elasticsearch, add in the lf/else statement as shown above or modify yours to include the if [type] == "netflow" statement.
6. Restart Logstash.

When Logstash starts up, it'll combine all the conf files into one big config. As a result, it'll spawn two listening two listening ports: one for the TCP beats data and one for the UDP netflow data. Things should run at this point on two different ports. Go into Kibana and check to see if the netflow index is/was created. Then check to make sure the data is arriving for whatever beats you're using.

Is there an easier/better way to do this? Maybe? I don't know, to be honest. I dug through the ES forums and a few others and cobbled together a means that worked for me via several semi-unrelated answers. Maybe this will help someone else out.