Logstash on kubernetes: not seeing any output

Elastic Stack Logstash
1

Hi,

I'm trying to bring up Logstash on our kubernetes cluster. I'm using the 5.1.2-alpine Docker image and bringing up the Logstash pod.

I'm seeing that the input/output configuration is being parsed properly, but I don't see output on either stdout or elasticsearch.

Input:

input {
  file {
    path => "/var/log/containers/*.log"
  }
}

Output:

output {
  elasticsearch { hosts => ["172.19.215.224:9200"] }
  stdout { codec => rubydebug }
}

I've gone through lot of discussion forums and have tried lot of things, but still no luck.

I see logs from logstash on stdout like following:

23:30:51.087 [pool-2-thread-3] DEBUG logstash.instrument.collector - Collector: Sending snapshot to observers {:created_at=>2017-01-26 23:30:51 +0000}
23:30:52.088 [pool-2-thread-5] DEBUG logstash.instrument.collector - Collector: Sending snapshot to observers {:created_at=>2017-01-26 23:30:52 +0000}
23:30:52.407 [Ruby-0-Thread-15: /usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:489] DEBUG logstash.pipeline - Pushing flush onto pipeline
23:30:53.089 [pool-2-thread-3] DEBUG logstash.instrument.collector - Collector: Sending snapshot to observers {:created_at=>2017-01-26 23:30:53 +0000}
23:30:54.090 [pool-2-thread-6] DEBUG logstash.instrument.collector - Collector: Sending snapshot to observers {:created_at=>2017-01-26 23:30:54 +0000}
23:30:55.092 [pool-2-thread-2] DEBUG logstash.instrument.collector - Collector: Sending snapshot to observers {:created_at=>2017-01-26 23:30:55 +0000}
23:30:56.093 [pool-2-thread-8] DEBUG logstash.instrument.collector - Collector: Sending snapshot to observers {:created_at=>2017-01-26 23:30:56 +0000}
23:30:57.094 [pool-2-thread-6] DEBUG logstash.instrument.collector - Collector: Sending snapshot to observers {:created_at=>2017-01-26 23:30:57 +0000}
23:30:57.407 [Ruby-0-Thread-15: /usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:489] DEBUG logstash.pipeline - Pushing flush onto pipeline
23:30:58.096 [pool-2-thread-8] DEBUG logstash.instrument.collector - Collector: Sending snapshot to observers {:created_at=>2017-01-26 23:30:58 +0000}
23:30:59.097 [pool-2-thread-1] DEBUG logstash.instrument.collector - Collector: Sending snapshot to observers {:created_at=>2017-01-26 23:30:59 +0000}

2

Any ideas here?

Do I need to post some more data?

TIA.

3

Nothing in stdout may indicate nothing going into the input?

4

As you have not specified start_position for the file input, log files will be tailed. Are the being written to once Logstash has started?

5

Thanks for replying @warkolm and @Christian_Dahlqvist. There's nothing indicating in stdout that there's any error with either input or output. I even see messages on stdout saying connection to elasticsearch is fine. Also, the containers are constantly writing messages to the log files. So this is not an issue about start_position.

Strange part is that I ran logstash outside kubernetes pod on the same machine with same configuration and it works fine. I see logs on stdout as well as elasticsearch.

What should I investigate here?

6

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.