No Structured Logs Found in Stack Monitoring when using Logstash

[zx8086]

Same issue

Running filebeat -> Elasticsearch = Works, can see Server, GC, Kibana etc logs...
filebeat -> Logstash = No Logs displayed

No Structured Logs Found in Stack Monitoring when using Logstashs

Hello All, Hope someone can help as I spent the past week trying to resolve this issue but couldn't. So some backstory. If I setup Elasticsearch with filebeat pointing to Elasticsearch, the Logs section in Stack Monitoring reports the server logs and recent logs no problem. When I go modify the filebeat to point to logstash, the logs do make it to Elasticsearch but I always have the "No Structured Logs Found" with a recommendation of pointing to the .json logs which are already setup and goin…

[stephenb (Stephen Brown)]

Perhaps this...

No Structured Logs Found in Stack Monitoring when using Logstashs

You may be missing the pipeline setting input { beats { port => 5044 } } output { if [@metadata][pipeline] { elasticsearch { hosts => "https://061ab24010a2482e9d64729fdb0fd93a.us-east-1.aws.found.io:9243" manage_template => false index => "%{[@metadata][beat]}-%{[@metadata][version]}-%{+YYYY.MM.dd}" pipeline => "%{[@metadata][pipeline]}" user => "elastic" password => "secret" } } else { elasticsearch { hosts => "https://061ab…

[zx8086]

@stephenb So can i use a data stream, which is what i was doing and missing the manage_template => false.

Everything else was identical

[stephenb (Stephen Brown)]

I would think so... I have not played with data streams much yet ..

[zx8086]

@stephenb

Just confirmed... you cannot use these two settings, with data_stream => "true"

        manage_template => false
        index => "%{[@metadata][beat]}-%{[@metadata][version]}-%{+YYYY.MM.dd}"

So there must be a way, while using data streams, to keep the Structured logs in the Stack Monitoring UI, as we are migrating to recommended future architecture with Data Streams.

[stephenb (Stephen Brown)]

Did you enable the Elasticsearch module and then take out those 2 lines from logstash? If so what data stream does the data end up in?

[zx8086]

@stephenb

The data goes into a data stream, the default one, as i don't specify any other data stream

.ds-logs-generic-default