Why forwarding logs by Logstash to Elasticsearch is not working

Elastic Stack Logstash
1

Hi all,

I am playing with Elastic stack through Docker containers (Filebeat -> Logstash Reciever -> Elasticsearch <- Kibana). Everything works pretty fine, but I have one problem.

Assuming the following logstash.conf file:

input {
    beats {
      port => 5044
    }
  }
 
  filter {
     if [docker][container][name] =~ /ucp/ {
        mutate {
          add_tag => ["system"]
        }
     }
     else if [docker][container][name] =~ /dtr/  {
        mutate {
          add_tag => ["system"]
        }
     } else {
        mutate {
          add_tag => ["application"]
        }
     }
  }

  output {
   if "system" in [tags] {
    elasticsearch {
      hosts => ["elasticsearch:9200"]
      index => "infrastructure-%{+YYYY.MM.dd}"
      } 
    
    } else if "application" in [tags] {
       elasticsearch {
        hosts => ["elasticsearch:9200"]
        index => "application-%{+YYYY.MM.dd}" 
      }
    } else {
      elasticsearch {
        hosts => ["elasticsearch:9200"]
        index => "trash-%{+YYYY.MM.dd}" 
      }
    }
  }

I can not see containers logs in the "application-%{+YYYY.MM.dd}" Elasticsearch index.
The "infrastructure-%{+YYYY.MM.dd}" index recieves data normally !

Can anyone help me to understand what I am missing?

Thanks in advance!

2

The configuration looks fine to me.

Do you have evidence that logs are being sent that match neither of the patterns given, and therefore should fall into the else bucket to be tagged as application?

3

So, using the [docker][container][name] as a string and =~ operator for pattern matching against /dtr/ or /ucp/ I am sure that logs to infrastructure index are sent. I can see them all with the codec => rubydebug on stdout of logstash container. But I continue to not see application logs. It seems to me like a passing high filter :slight_smile: only infrastructure logs are passing. Weird!

4

in the codec => rubydebug output, can you tell what the value is for [docker][container][name] for the events that don't make it through?

5

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.