Some logstash feature requests

Elastic Stack Logstash
1

Dear Logstash Team,

I've grown to love Elastic Stack and we're now adopting and seeing the power of it. I'd love to see the following features:

  • For Record Last Run & Meta Data Path. This is a great way of tracking your last process point to pick up again on your next run/schedule.

Use Case:
In our automated build pipeline, I have Spinnaker deploying a fedora AMI with logstash installed and configured as a systemd process. It does its thing logstash starts, health endpoints resolve and my conf processes (all great). But, what if i have since processed 1 million messages and the instance terminates. By design of logstash it will check for the file, if not exist create and set default value to either 0 or 1-1-1970 (depending on the datatype of that field)

The ask: It would be nice to have an ability to direct this to either DB (RDS / NoSQL) or S3 or alternative mechanism that isn't local to the instance

  • Everything is log file and console driven.

I have a logstash AMI running separately in AWS which is pushing events to Hosted ES. These are all text based configuration files.

The Ask:
It would be nice e.g. Kibana to have the ability to see your logstash configurations, last runs and metrics that can be used for debugging purposes. With the improvements adding in 5.x i see a world where your APIs ship those metrics to provide a nice monitoring UI.

Long term, i'd love to see the ability to handle this all UI manner with functionality such as managed ES logstash instances. This would be another revenue source for you and reduce the need client side to spin up the infrastructure etc.

thanks and welcome to opinions on my asks.

Wayne

2

not sure if I understand your first ask, Logstash Can send data to other destinations with the logstash-output plugins
https://www.elastic.co/guide/en/logstash/current/output-plugins.html

As for your statistics and config file ask.

That would be interesting to have a kibana interface/tab to look at all logstash instances, but in the mean time there are ways to measure its performance in 5.3

https://www.elastic.co/guide/en/logstash/current/monitoring.html

Also many people including my self use the Metrics filter to understand how many events each logstash is processing.

3

eperry, for ask one..

Specifically i am talking about updating the last run meta file to S3 or a DB table. That way if the instance dies your last known value is "known" and your new instance can pick up from that point.

Let me know if not clear... Happy to discuss

4

Oh you want to have the .sincedb stored in a separate location

https://www.elastic.co/guide/en/logstash/current/plugins-inputs-file.html#plugins-inputs-file-sincedb_path

be interesting to write it to some more then just a file, but since S3 can be mounted as a filesystem, you could have a nfs mount or fuse filesystem to write it anywere you want.

this is a pretty unique case, but in the world of a dynamic servers and whatnot and interesting one.

5

or even just create a small separate EBS , that just holds this sincedb file, and when your AMI ends and restarts just makes sure it mounts that EBS as part of it. Then you never loose your position