I've grown to love Elastic Stack and we're now adopting and seeing the power of it. I'd love to see the following features:
For Record Last Run & Meta Data Path. This is a great way of tracking your last process point to pick up again on your next run/schedule.
Use Case:
In our automated build pipeline, I have Spinnaker deploying a fedora AMI with logstash installed and configured as a systemd process. It does its thing logstash starts, health endpoints resolve and my conf processes (all great). But, what if i have since processed 1 million messages and the instance terminates. By design of logstash it will check for the file, if not exist create and set default value to either 0 or 1-1-1970 (depending on the datatype of that field)
The ask: It would be nice to have an ability to direct this to either DB (RDS / NoSQL) or S3 or alternative mechanism that isn't local to the instance
Everything is log file and console driven.
I have a logstash AMI running separately in AWS which is pushing events to Hosted ES. These are all text based configuration files.
The Ask:
It would be nice e.g. Kibana to have the ability to see your logstash configurations, last runs and metrics that can be used for debugging purposes. With the improvements adding in 5.x i see a world where your APIs ship those metrics to provide a nice monitoring UI.
Long term, i'd love to see the ability to handle this all UI manner with functionality such as managed ES logstash instances. This would be another revenue source for you and reduce the need client side to spin up the infrastructure etc.
That would be interesting to have a kibana interface/tab to look at all logstash instances, but in the mean time there are ways to measure its performance in 5.3
Specifically i am talking about updating the last run meta file to S3 or a DB table. That way if the instance dies your last known value is "known" and your new instance can pick up from that point.
be interesting to write it to some more then just a file, but since S3 can be mounted as a filesystem, you could have a nfs mount or fuse filesystem to write it anywere you want.
this is a pretty unique case, but in the world of a dynamic servers and whatnot and interesting one.
or even just create a small separate EBS , that just holds this sincedb file, and when your AMI ends and restarts just makes sure it mounts that EBS as part of it. Then you never loose your position
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant
logo are trademarks of the
Apache Software Foundation
in the United States and/or other countries.