We have a mail gateway based on Postfix, and we want to get these logs into Elastic.
For our product, our MSP that helps us with it has a logstash based integration that is also merges the mutliline log of postfix to one document.
Now at the moment we have no logstash and are not planning to install it, so we are trying to get this working without Logstash. This gets us to some different questsions, and I hope I can post them all in one topic:
Is there any way in Elastic with simple ingest pipelines and without Logstashto merge these multiline logs into one document at ingest time? I think I read about ways to do this after ingest by reindexing to a new index with merged documents, but it looks like this is not currently possible at ingest time.
Are there plans in the future tu be able to merge such multiline logs with an agent-based ingest pipline and without Logstash?
If both questions are answered with "no", we might have to start using Logstash. But can somebody tell me about the "future" of this product? As far as I read and heard Logstash gets more and more replaced by agent functionality - does this mean that Logstash will not be needed anymore in a near or far future? Or will logstash stay a helpfull ingest method for the foreseeable future?
This should be done before sending the data to Elasticsearch, if you do not want to use Logstash you can use Filebeat or Elastic Agent as both can also aggregate multiline based on a pattern.
It is not possible to do at ingestion time.
As mentioned this can be done with Filebeat or Elastic Agent.
Even though Logstash may feels in maintenance mode sometimes as it does not received much improvement or new features in the past years, I think that the official position at the moment is that there are no plans to discontinue Logstash.
I'm a longtime Logstash user and rely on it for a lot of data ingestion flows, but if I'm not wrong, Logstash was always seen as an advanced use case.
There are things that are only possible with Logstash and a lot of things are way more easier to implement with Logstash than with Ingest Pipelines.
In recent versions if you have an Enterprise License you can even run Elasticsearch Ingest Pipelines inside Logstash, for example, I do that to offload some log processing from Elasticsearch nodes to Logstash nodes.
immortal words of the software industry. No plans, only until there are plans. Remember Solaris, CentOS, Flash, Internet Explorer, ...
Yeah, but they were obviously stagnating products with no future and better replacements, I hear you say.
Is Logstash any different?
Define foreseeable? It will still be available and used for years. But maybe , I simply dont know, it will slowly accrue technical debt over that time, and eventually end up supported by some guy in Nebraska
About the multiline problem: I did not know that Elastic Agent can do this. I looked into it, but if I understand correctly this is not usable for Postfix: Postfix logs are not multiple lines next to each other, it is more like:
Entry for QueueID A
Entry for QueueID B
New Data about QueueID A
Entry for QueueID C
New Data about Event C
However integrations for Postfix are implemented, I would guess the biggest problem is to make sure that no new log line is created for an event before aggregating it, so you would be forced to delay the log reader for several seconds. Might it be possible that Elastic Agent can work with a log like this, aggregating by a "Unique Key"? Otherwise this would mean that without Logstash we would have to parse the log ourselfs, write it into a new file and ingest this file into Elastic.
And a clarification about my Logstash question: I heard from some people that Logstash currently has some features and capabilities that are not possible with other ingest methods like Agent or Beats. But that on the one hand Logstash is not developed as intensivly as the rest, and that more and more features that were exclusive to Logstash are getting ported to the Elastic Agent. So this just looked like Logstash would sooner or later get obsolete and Elastic Agent will be the new Way to Go for any ingest need. But this is purely hearsay from other people working with Elastic longer than me.
Now, thats a black box to me, but someone somewhere has created that integration. How it works, what it really does, what the end result looks like is not given. But the implication is that it's "good enough", right ?
If I were you, knowing what I know now, I'd not try to re-invent the wheel and use what I understand is a supported (by your MSP) working setup. I don't see great advantage of say developing/testing/supporting something with an Agent or other tool to do same job as the one you already have. If the only fear is "Logstash might be retired in X months/years", then for me that risk does not outweigh the advantages.
btw there was a thread long ago on postfix logs and correlating on queueID, and this blog entry covers much the same territory.
Ah, so this is not just a multiline log, but you need to aggregate events, then no, you cannot use neither Filebeat or Elastic Agent (which uses filebeat under the hood), nor does this in the Ingest Pipeline, only Logstash is able to aggregate those lines using the aggregate filter.
Yes, if you do not want to use Logstash you will need to build something that would aggregate the files before sending it to Elasticsearch.
I've asked this myself in multiple interactions that I've had with people from Elastic, my perception about Logstash is that it feels that it is in maintenance mode, it is supported, but it hasn't got a lot of improvements in the past years and there are some feature request that could improve it open for almost a decade now.
But as always the official answer is that there are no plans to phase out Logstash, we will only know if/when they announce it, and since a lot of people and companies rely on Logstash, if this happens I think it will first be deprecated for some time before being descontinued.
In my case it would be easier to move to a third-party ETL tool like Vector from Datadog, than to move to Elastic Agent or Filebeat, I have some ingestion flows where I replaced some ingest pipelines with Vector.
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant
logo are trademarks of the
Apache Software Foundation
in the United States and/or other countries.
