Filebeat to index with different structures

Hello, newby here,
My task is to take multiple (8) logs with different formats and using filebeat send them directly to elastic. They should all end inside the same index and each log should be searchable using "level" field:

"log" : {
  "level": "access log",
  "message": "access message 1"
},
"log": {
  "level": "error log",
  "message": "error message 1"
}

The problem I am facing is, I am getting no fields inside kibana once filebeat harvests the data:

    "hits" : [
      {
        "_index" : "tomcat_logs",
        "_type" : "_doc",
        "_id" : "XO8HEngBdkAWhdismvyE",
        "_score" : 1.0,
        "_source" : {
          "@timestamp" : "2021-03-08T13:29:50.444Z",
          "agent" : {
            "version" : "7.10.2"
          },
          "fields" : {
            "type" : "tomcat_logs"
          }
        }
      }

My filebeat:

filebeat.inputs:
- type: log
  enabled: true
  paths:
    - /opt/apache-tomcat-9.0.41/logs/catalina.*
  fields:
    codec: plain
    type: tomcat_logs
  processors:
    - drop_event:
        when:
          regexp:
            message: 'INFO|DEBUG'
    - dissect:
        tokenizer: '%{date} %{time} %{logLevel} [%{method}] %{class} %{message}'
        field: "message"
        target_prefix: 'log'

- type: log
  enabled: true
  paths:
    - /opt/apache-tomcat-9.0.41/logs/localhost.*
  fields:
    codec: plain
    type: tomcat_logs
  processors:
    - drop_event:
        when:
          regexp:
            message: 'INFO|DEBUG'
    - dissect:
        tokenizer: '%{date} %{time} %{logLevel} [%{method}] %{class} %{listenerType}: %{listenerLog}'
        field: "message"
        target_prefix: 'log'
- type: log
  enabled: true
  paths:
    - /opt/apache-tomcat-9.0.41/logs/localhost_access_log.*.txt
  fields:
    codec: plain
    type: tomcat_logs
  processors:
    - drop_event:
        when:
          regexp:
            message: 'INFO|DEBUG'
    - dissect:
        tokenizer: '%{hostName} %{userName} %{authenticatedUser} [%{dateTime}] "%{request}" %{response} %{bytes}'
        field: "message"
        target_prefix: 'log'
- type: log
  enabled: true
  paths:
    - /var/log/httpd/error_log-*
  fields:
    codec: plain
    type: tomcat_logs
  processors:
    - drop_event:
        when:
          not:
            regexp:
              message: 'auth_openidc:error'
    - dissect:
        tokenizer: '[%{dateTime}] [%{logLevel}] [pid %{pid|integer}] [client %{clientIP}] %{message}'
        field: "message"
        target_prefix: 'log'
- type: log
  enabled: true
  paths:
    - /var/log/httpd/error_log-*
  fields:
    codec: plain
    type: tomcat_logs
  processors:
    - drop_event:
        when:
          not:
            regexp:
              message: 'proxy:error'
    - dissect:
        tokenizer: '[%{dateTime}] [%{logLevel}] [pid %{pid|integer}] %{message}'
        field: "message"
        target_prefix: 'log'

- type: log
  enabled: true
  paths:
    - /var/log/httpd/error_log-*
  fields:
    codec: plain
    type: tomcat_logs
  processors:
    - drop_event:
        when:
          not:
            regexp:
              message: 'proxy_http:error'
    - dissect:
        tokenizer: '[%{dateTime}] [%{logLevel}] [pid %{pid|integer}] [client %{clientIP}] %{message}'
        field: "message"
        target_prefix: 'log'
- type: log
  enabled: true
  paths:
    - /var/log/httpd/ssl_error_log-*
  fields:
    codec: plain
    type: tomcat_logs
  processors:
    - dissect:
        tokenizer: '[%{dateTime}] [%{logLevel}] [pid %{pid|integer}] %{message}'
        field: "message"
        target_prefix: 'log'
- type: log
  enabled: true
  paths:
    - /var/log/httpd/access_log-*
  fields:
    codec: plain
    type: tomcat_logs
  processors:
    - dissect:
        tokenizer: '%{hostName} %{logicalUsername} %{authenticatedUser} [%{dateTime}] "%{request}" %{statusCode} %{bytes} "-" "%{header}"'
        field: "message"
        target_prefix: 'log'
- type: log
  enabled: true
  paths:
    - /var/log/httpd/ssl_access_log-*
  fields:
    codec: plain
    type: tomcat_logs
  processors:
    - dissect:
        tokenizer: '%{hostName} %{logicalUsername} %{authenticatedUser} [%{dateTime}] "%{request}" %{statusCode} %{bytes}'
        field: "message"
        target_prefix: 'log'
- type: log
  enabled: true
  paths:
    - /var/log/httpd/ssl_request_log-*
  fields:
    codec: plain
    type: tomcat_logs
  processors:
    - dissect:
        tokenizer: '[%{dateTime}] %{logLevel} %{sslProtocol} %{sslCipher} "%{request}" %{bytes}'
        field: "message"
        target_prefix: 'log'
# ======================= Elasticsearch template setting =======================

setup.template:
  name: "tomcat_logs"
  pattern: "tomcat_logs-*"
  overwrite: true
  enabled: false

setup.ilm.enabled: false


setup.template.settings:
  index.number_of_shards: 1
  #index.codec: best_compression
  #_source.enabled: false
# ---------------------------- Elasticsearch Output ----------------------------
output.elasticsearch:
  # Array of hosts to connect to.
  hosts: ["xxxx:9200"]
  index: "tomcat_logs"

Before, each log had its own index and it worked well, now I am completely stuck and would appreciate any help.

hi @Edi1 , can you check the Filebeat logs, they might offer some info? Can you also set logging level to debug first?

on starting filebeat -e -d "publisher"

2021-03-09T09:57:38.983+0100    INFO    instance/beat.go:645    Home path: [/usr/share/filebeat] Config path: [/etc/filebeat] Data path: [/var/lib/filebeat] Logs path: [/var/log/filebeat]
2021-03-09T09:57:38.984+0100    INFO    instance/beat.go:653    Beat ID: 0da2671d-2708-46c8-a523-07463e36a28a
2021-03-09T09:57:38.984+0100    INFO    [seccomp]       seccomp/seccomp.go:124  Syscall filter successfully installed
2021-03-09T09:57:38.984+0100    INFO    [beat]  instance/beat.go:981    Beat info       {"system_info": {"beat": {"path": {"config": "/etc/filebeat", "data": "/var/lib/filebeat", "home": "/usr/share/filebeat", "logs": "/var/log/filebeat"}, "type": "filebeat", "uuid": "0da2671d-2708-46c8-a523-07463e36a28a"}}}
2021-03-09T09:57:38.985+0100    INFO    [beat]  instance/beat.go:990    Build info      {"system_info": {"build": {"commit": "aacf9ecd9c494aa0908f61fbca82c906b16562a8", "libbeat": "7.10.2", "time": "2021-01-12T23:11:24.000Z", "version": "7.10.2"}}}
2021-03-09T09:57:38.985+0100    INFO    [beat]  instance/beat.go:993    Go runtime info {"system_info": {"go": {"os":"linux","arch":"amd64","max_procs":2,"version":"go1.14.12"}}}
2021-03-09T09:57:38.986+0100    INFO    [beat]  instance/beat.go:997    Host info       {"system_info": {"host": {"architecture":"x86_64","boot_time":"2021-01-04T11:17:51+01:00","containerized":false,"name":"poc1","ip":["xxx/8","::1/128","xxx/24","fe80::20c:29ff:fed8:5ea4/64","10.254.1.21/24","fe80::20c:29ff:fed8:5eae/64"],"kernel_version":"4.14.35-2025.401.4.el7uek.x86_64","mac":["00:0c:29:d8:5e:a4","00:0c:29:d8:5e:ae"],"os":{"family":"","platform":"ol","name":"Oracle Linux Server","version":"7.9","major":7,"minor":9,"patch":0},"timezone":"CET","timezone_offset_sec":3600,"id":"81c6c5ad48f94d2287c55291113d5ac8"}}}
2021-03-09T09:57:38.986+0100    INFO    [beat]  instance/beat.go:1026   Process info    {"system_info": {"process": {"capabilities": {"inheritable":null,"permitted":["chown","dac_override","dac_read_search","fowner","fsetid","kill","setgid","setuid","setpcap","linux_immutable","net_bind_service","net_broadcast","net_admin","net_raw","ipc_lock","ipc_owner","sys_module","sys_rawio","sys_chroot","sys_ptrace","sys_pacct","sys_admin","sys_boot","sys_nice","sys_resource","sys_time","sys_tty_config","mknod","lease","audit_write","audit_control","setfcap","mac_override","mac_admin","syslog","wake_alarm","block_suspend","audit_read"],"effective":["chown","dac_override","dac_read_search","fowner","fsetid","kill","setgid","setuid","setpcap","linux_immutable","net_bind_service","net_broadcast","net_admin","net_raw","ipc_lock","ipc_owner","sys_module","sys_rawio","sys_chroot","sys_ptrace","sys_pacct","sys_admin","sys_boot","sys_nice","sys_resource","sys_time","sys_tty_config","mknod","lease","audit_write","audit_control","setfcap","mac_override","mac_admin","syslog","wake_alarm","block_suspend","audit_read"],"bounding":["chown","dac_override","dac_read_search","fowner","fsetid","kill","setgid","setuid","setpcap","linux_immutable","net_bind_service","net_broadcast","net_admin","net_raw","ipc_lock","ipc_owner","sys_module","sys_rawio","sys_chroot","sys_ptrace","sys_pacct","sys_admin","sys_boot","sys_nice","sys_resource","sys_time","sys_tty_config","mknod","lease","audit_write","audit_control","setfcap","mac_override","mac_admin","syslog","wake_alarm","block_suspend","audit_read"],"ambient":null}, "cwd": "/var/log/filebeat", "exe": "/usr/share/filebeat/bin/filebeat", "name": "filebeat", "pid": 5409, "ppid": 4595, "seccomp": {"mode":"filter","no_new_privs":true}, "start_time": "2021-03-09T09:57:38.160+0100"}}}
2021-03-09T09:57:38.986+0100    INFO    instance/beat.go:299    Setup Beat: filebeat; Version: 7.10.2
2021-03-09T09:57:38.987+0100    INFO    eslegclient/connection.go:99    elasticsearch url: http://xxx:9200
2021-03-09T09:57:38.987+0100    DEBUG   [publisher]     pipeline/consumer.go:148        start pipeline event consumer
2021-03-09T09:57:38.987+0100    INFO    [publisher]     pipeline/module.go:113  Beat name: poc1
2021-03-09T09:57:38.988+0100    INFO    instance/beat.go:455    filebeat start running.
2021-03-09T09:57:38.988+0100    INFO    [monitoring]    log/log.go:118  Starting metrics logging every 30s
2021-03-09T09:57:38.990+0100    INFO    memlog/store.go:119     Loading data file of '/var/lib/filebeat/registry/filebeat' succeeded. Active transaction id=1060052
2021-03-09T09:57:39.260+0100    INFO    memlog/store.go:124     Finished loading transaction log file for '/var/lib/filebeat/registry/filebeat'. Active transaction id=1068894
2021-03-09T09:57:39.261+0100    INFO    [registrar]     registrar/registrar.go:109      States Loaded from registrar: 56
2021-03-09T09:57:39.261+0100    INFO    [crawler]       beater/crawler.go:71    Loading Inputs: 11
2021-03-09T09:57:39.262+0100    DEBUG   [publisher]     pipeline/client.go:231  Pipeline client receives callback 'onFilteredOut' for event: {Timestamp:0001-01-01 00:00:00 +0000 UTC Meta:null Fields:null Private:{Id:native::50414589-64512 PrevId: Finished:true Fileinfo:<nil> Source:/opt/apache-tomcat-9.0.41/logs/catalina.2021-02-15.log Offset:8243 Timestamp:2021-03-09 09:57:17.23585363 +0100 CET TTL:-1ns Type:log Meta:map[] FileStateOS:50414589-64512 IdentifierName:native} TimeSeries:false}
2021-03-09T09:57:39.262+0100    DEBUG   [publisher]     pipeline/client.go:231  Pipeline client receives callback 'onFilteredOut' for event: {Timestamp:0001-01-01 00:00:00 +0000 UTC Meta:null Fields:null Private:{Id:native::54852818-64512 PrevId: Finished:true Fileinfo:<nil> Source:/opt/apache-tomcat-9.0.41/logs/catalina.2021-02-05.log Offset:34243 Timestamp:2021-03-09 09:57:17.228389144 +0100 CET TTL:-1ns Type:log Meta:map[] FileStateOS:54852818-64512 IdentifierName:native} TimeSeries:false}

On adding new log

2021-03-09T09:57:39.413+0100    INFO    log/input.go:157        Configured paths: [/var/log/httpd/ssl_request_log-*]
2021-03-09T09:57:39.413+0100    INFO    [crawler]       beater/crawler.go:141   Starting input (ID: 13903261551536249677)
2021-03-09T09:57:39.413+0100    INFO    [crawler]       beater/crawler.go:108   Loading and starting Inputs completed. Enabled inputs: 10
2021-03-09T09:57:39.413+0100    INFO    cfgfile/reload.go:164   Config reloader started
2021-03-09T09:57:39.414+0100    INFO    cfgfile/reload.go:224   Loading of config files completed.
2021-03-09T09:57:59.398+0100    DEBUG   [publisher]     pipeline/client.go:231  Pipeline client receives callback 'onFilteredOut' for event: {Timestamp:0001-01-01 00:00:00 +0000 UTC Meta:null Fields:null Private:{Id:native::50414591-64512 PrevId: Finished:false Fileinfo:0xc000c0f2b0 Source:/var/log/httpd/ssl_error_log-20210307 Offset:4028 Timestamp:2021-03-09 09:57:59.39810365 +0100 CET m=+20.528680756 TTL:-1ns Type:log Meta:map[] FileStateOS:50414591-64512 IdentifierName:native} TimeSeries:false}
2021-03-09T09:57:59.398+0100    INFO    log/harvester.go:302    Harvester started for file: /var/log/httpd/ssl_error_log-20210307
2021-03-09T09:58:00.399+0100    INFO    [publisher_pipeline_output]     pipeline/output.go:143  Connecting to backoff(elasticsearch(http://xxx:9200))
2021-03-09T09:58:00.400+0100    INFO    [publisher]     pipeline/retry.go:219   retryer: send unwait signal to consumer
2021-03-09T09:58:00.400+0100    INFO    [publisher]     pipeline/retry.go:223     done
2021-03-09T09:58:00.401+0100    INFO    [esclientleg]   eslegclient/connection.go:314   Attempting to connect to Elasticsearch version 7.10.2
2021-03-09T09:58:00.452+0100    INFO    [license]       licenser/es_callback.go:51      Elasticsearch license: Basic
2021-03-09T09:58:00.452+0100    INFO    [publisher_pipeline_output]     pipeline/output.go:151  Connection to backoff(elasticsearch(http://xxx:9200)) established
2021-03-09T09:58:00.456+0100    DEBUG   [publisher]     memqueue/ackloop.go:160 ackloop: receive ack [0: 0, 1]
2021-03-09T09:58:00.456+0100    DEBUG   [publisher]     memqueue/eventloop.go:535       broker ACK events: count=1, start-seq=1, end-seq=1

2021-03-09T09:58:00.456+0100    DEBUG   [publisher]     memqueue/ackloop.go:128 ackloop: return ack to broker loop:1
2021-03-09T09:58:00.456+0100    DEBUG   [publisher]     memqueue/ackloop.go:131 ackloop:  done send ack

Thank you for reply, I have added the code from filebeat output

I think I havent also explained well what my goal is.

I want to get multiple different logs in same index as follows:

Index:

"log": {
"a": "something a",
"b": "something b",
"c": "something c"
}

Filebeat log 1

  • this will go inside index "log"
{
"a": "something a",
"b": ------> this filebeat log doesnt have field b and inside proseccor dissect it does not exist,
"c": ----> same for "c"
}

Filebeat log b

{
"a": ----> doesnt exist in this log,
"b": "something b",
"c": "something c"
}

Final index:

"log": {
"a":  "something a",
"b": "-",
"c": "-"
},
"log": {
"a":  "-",
"b": "something b",
"c": "something c"
}

Solved. Had some minor mistakes in filebeat

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.