Flushing of pipelines

Elastic Stack Logstash
1

Hi there
Would really appreciate the help, trying to understand what "pushing flush into pipeline" means. Running some sflow with elastiflow on a now newly installed OS and ELK stack these inputs into logstash in the debug logs logstash starts up perfectly I have suricata logs flying in. I can see the data (from ElastiFlow) correctly formatted and after the output is printed on the debug log, it says :
2018-09-27T23:14:29,082][DEBUG][logstash.pipeline
] Pushing flush onto pipeline
{:pipeline_id=>"elastiflow",
:thread=>"#<Thread:0x4df2635a@
/usr/share/logs
tash/logstash-
core/lib/logstash/pipeline_action/create.rb:46
sleep>

Now I can see the correctly formatted pipeline in the monitoring section in kibana for logstash > pipelines

I have imported the index pattern as well.

What I am seeing :
Logs of
Logstash

  1. Data in the logs
  2. Creation of pipeline in the logs
  3. Flush of that pipeline in the logs (see above)
  4. No errors in the logs

Kibana:

  1. Created pipeline
  2. If I do DELETE _ingest/pipeline/elasticflow/ it deletes the pipeline. Restart logstash recreates the pipeline
  3. Monitor pipeline it's as flatline as someone that just passed away.....
  4. I don't see an index but on elasticsearch logs I don't see any errors

How I try rationalize it
Logstash starts up with no errors
I see pipelines created no errors

There is input> I see the transformation into an output with data>

Flushing the pipeline (whatever that means! After I see the output in the log. )

In kibana
I see other sources populating from suricata etc so there is coms and these are working
I see a pipeline for Elastiflow
I see a flatline on that pipeline others I see them doing their work
I don't see a index
I see my imported index pattern

In Elasticsearch logs
I see no errors starting up
I see elastiflow creating a template for elastiflow3.3.0-date and the log stops there.
In elastic search I see the template if I search for it I don't see any index

These things I AM seeing is running three separate ssh sessions to the server tailing the logs and checking journalctl

(xenon 128gig wash part of the stack Elasticsearch 24g heap, Kibana 10gig , logstash 24gig nothing else runs on it OS Ubuntu 18.10)

Would really appreciate the help

2

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.