Help on: Error creating runner from config: Can only start an input when all related states are finished

Elastic Stack Beats
1

Hello,

I have an issue with my filebeat configuration. I have an api running on k8s, logs are written to stdout and requests-logs to stderr in order to get them both in my ES Cluster.
I m using filebeat 6.7.1 OSS with autodicovery (to work with AWS ElasticSearch). In my config I have defined one kubernetes provider with 2 docker type templates:

filebeat.autodiscover:
      providers:
        - type: kubernetes
          templates:
            - condition:
                equals:
                  kubernetes.labels.team: pipeline
              config:
                - type: docker
                  containers.ids:
                    - "${data.kubernetes.container.id}"
                  containers.stream: stdout
                  pipeline: java_application
                  tags: [ "application" ]
                  # any line not starting with know log levels match multiline pattern
                  multiline.pattern: "^[FEWIDT]+"
                  multiline.negate: true
                  multiline.match: after
                  exclude_lines: "^[^FEWIDT].*$"
                  processors:
                  - dissect:
                      tokenizer: "%{level} [%{timestamp}] %{classpath}: %{message}"
                      field: "message"
                      target_prefix: "ingest"
            - condition:
                equals:
                  kubernetes.labels.team: pipeline
              config:
                - type: docker
                  containers.ids:
                    - "${data.kubernetes.container.id}"
                  containers.stream: stderr
                  pipeline: http_requests
                  tags: [ "requests" ]
                  processors:
                  - dissect:
                      tokenizer: '[%{timestamp}] %{client_ip} "%{method} %{path} %{?http_version}" %{status} %{response_size} %{response_time}'
                      field: "message"
                      target_prefix: "http"

processors:
      - add_cloud_metadata: ~

output.elasticsearch:
      hosts: ['${ELASTICSEARCH_HOST}:${ELASTICSEARCH_PORT}']
      indices:
        - index: "logs-filebeat-%{[beat.version]}-%{+yyyy.MM.dd}"
          when:
            contains:
              tags: application
        - index: "requests-filebeat-%{[beat.version]}-%{+yyyy.MM.dd}"
          when:
            contains:
              tags: requests

This configuration seems to work but after a short time filebeat stop reporting logs to ES and show this error:
Error creating runner from config: Can only start an input when all related states are finished
I found an explanation to this error, filebeat has the same file open twice but I need to do this to read stdout and stderr to process either application and requests logs with a different dissect processor, multiline configuration and writing to different indexes in ES. Does anyone knows how to do this?

2

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.