Logs are not visible in Kibana

Elastic Stack Elasticsearch
1

Hello,

do test some logs Logstash configurations I want to use this docker-compose.yml and I'm running Debian 12 .

  • Docker version 25.0.3, build 4debf41
  • docker-compose version 1.29.2
  • my user is in the docker group, to run docker

My usecase:

In my usecase I want to place some logs mainlog into the logstash directory. See tree below:

~/local-ELK/logstash$ tree -d
.
├── config
├── mainlog.log
└── pipeline

~/local-ELK/logstash/mainlog.log$ ls
mainlog

# filter.conf is my custom logstash filter
~/local-ELK/logstash/pipeline$ ls
filter.conf  logstash.conf

Now I start my composition with docker compose up. I can login to Kibana on localhost, but there are no field names from my filter.conf and no data from my mainlog.
I also tried after login into Kibana to restart the logstash container, but the result was the same.

image
image1355×667 69.7 KB

I checked with docker exec -it local-elk-logstash /bin/bash if the filter.conf and the mainlog exists and if its possible to read it and it was.

Do you have any suggestions?

2

You need to share both your logstash configurations and your docker compose, without it is not possible to know what may be the issue.

3

My docker-compose.yml is linked see below.
My filter.conf looks like:

input { 
        file {
                id => "main"
                path => "/usr/share/logstash/mainlog.log"
                sincedb_path => "/dev/null"
                start_position => "beginning"
        }
        stdin { } 
}



filter {
        # here starts everything which contains in the mainlog
        if [log][file][path] =~ "mainlog" {
                mutate {
                        add_field => {
                                "[@metadata][sourcetype]" => "exim-mainlog"
                                "state" => "mainlog"
                                "exim_msg_state" => "not-processed"
                                "action_id" => "not-processed"
                        }
                }
       # alot of groks more


   # here starts everything which contains in the rejectlog
        if [log][file][path] =~ "rejectlog" {
          mutate {
            add_field => {
              "[@metadata][sourcetype]" => "exim-reject"
              "state" => "rejectlog"
            }
          }

          mutate {
            add_field => { "exim_msg_state" => "not-processed" }
            add_field => { "action_id" => "not-processed" }
          }

         # more groks

output {
        elasticsearch { 
                hosts => ["elasticsearch:9200"]
        }
         
       stdout { codec => rubydebug }
}

The filter.conf is running in production, just without the rubydebug part. But I don't want to do testing on a productive system.

4

I found a solution. I was assuming everything in this path will be mappend and I can see it.
After changing it to:

path => "/usr/share/logstash/mainlog.log/mainlog.log"

its working now.

5

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.