How do you see data sent from syslog > logstash > elasticsearch (no filebeats)?

[meatwad]

Hi,

I'm just getting started so please forgive me if this is a stupid question.

I've got Elkstack installed onto a CentOS 7 system that is ingesting syslog feeds from remote systems. Rsyslog is configured to listen on port 514 then forward those messages to the local instance of Logstash that's listening on port 5044, runs those messages through a filter, then sends the output to the local instance of Elasticsearch.

For the life of me, I can't figure out how to see this data in Kibana. Since no logs are being written to disk, Filebeat isn't involved, correct? The only way I can see how to search for data is using a Filebeat filter.

Sorry if I'm just being an idiot (which is likely the case)

Thanks!

[Joe_Fleming]

Since no logs are being written to disk, Filebeat isn't involved, correct?

That's right, Filebeat is just a way to send the contents of a file to Elasticsearch, which is useful for tailing log files and such. But you're sending data to Logstash and ingesting that way, so no Filebeat.

Are you sure you're ingesting data? You can check with curl or even your browser to make sure. Just curl -XGET http://localhost:9200/<your index name or pattern>/_count?pretty and check the count value. Alternatively, just open http://localhost:9200/<your index name or pattern>/_count?pretty in your browser too. Adjust the hostname and port as needed.

{
  "count" : 20, <-- THIS IS WHAT YOU WANT TO LOOK FOR
  "_shards" : {
    "total" : 1,
    "successful" : 1,
    "skipped" : 0,
    "failed" : 0
  }
}

If the value is more than zero, then you have documents, and you're successfully writing things to Elasticsearch.

Now in Kibana, you need to configure the index pattern, which is one of the first things Kibana will ask you to do. The value you enter for the pattern can be the index itself, or some wildcard value (like logs-* as an example). Once entered, it should tell you whether or not it found the index, and when you add it in Kibana, you should see a list of the fields.

Once you've done that, you should be able to select the index pattern in Discover and see your documents. And now you can use that index pattern to create visualizations as well.