# Filebeat to index with different structures

**URL:** <https://discuss.elastic.co/t/filebeat-to-index-with-different-structures/266565>\
**Category:** Beats\
**Tags:** filebeat\
**Created:** [March 8, 2021, 1:43pm UTC](https://discuss.elastic.co/t/filebeat-to-index-with-different-structures/266565 "2021-03-08T13:43:45Z")\
**Posts on this page:** 8\
**Page:** 1

<div class="post-metadata">

**Author:** ![Edi1](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/edi1/32/84053_2.png) [@Edi1](https://discuss.elastic.co/u/Edi1)\
**Post date:** [March 8, 2021, 1:43pm UTC](https://discuss.elastic.co/t/filebeat-to-index-with-different-structures/266565/1 "2021-03-08T13:43:45Z")

</div>

Hello, newby here,  
My task is to take multiple (8) logs with different formats and using filebeat send them directly to elastic. They should all end inside the same index and each log should be searchable using "level" field:

```auto
"log" : {
  "level": "access log",
  "message": "access message 1"
},
"log": {
  "level": "error log",
  "message": "error message 1"
}

```

The problem I am facing is, I am getting no fields inside kibana once filebeat harvests the data:

```auto
    "hits" : [
      {
        "_index" : "tomcat_logs",
        "_type" : "_doc",
        "_id" : "XO8HEngBdkAWhdismvyE",
        "_score" : 1.0,
        "_source" : {
          "@timestamp" : "2021-03-08T13:29:50.444Z",
          "agent" : {
            "version" : "7.10.2"
          },
          "fields" : {
            "type" : "tomcat_logs"
          }
        }
      }

```

My filebeat:

```auto
filebeat.inputs:
- type: log
  enabled: true
  paths:
    - /opt/apache-tomcat-9.0.41/logs/catalina.*
  fields:
    codec: plain
    type: tomcat_logs
  processors:
    - drop_event:
        when:
          regexp:
            message: 'INFO|DEBUG'
    - dissect:
        tokenizer: '%{date} %{time} %{logLevel} [%{method}] %{class} %{message}'
        field: "message"
        target_prefix: 'log'

- type: log
  enabled: true
  paths:
    - /opt/apache-tomcat-9.0.41/logs/localhost.*
  fields:
    codec: plain
    type: tomcat_logs
  processors:
    - drop_event:
        when:
          regexp:
            message: 'INFO|DEBUG'
    - dissect:
        tokenizer: '%{date} %{time} %{logLevel} [%{method}] %{class} %{listenerType}: %{listenerLog}'
        field: "message"
        target_prefix: 'log'
- type: log
  enabled: true
  paths:
    - /opt/apache-tomcat-9.0.41/logs/localhost_access_log.*.txt
  fields:
    codec: plain
    type: tomcat_logs
  processors:
    - drop_event:
        when:
          regexp:
            message: 'INFO|DEBUG'
    - dissect:
        tokenizer: '%{hostName} %{userName} %{authenticatedUser} [%{dateTime}] "%{request}" %{response} %{bytes}'
        field: "message"
        target_prefix: 'log'
- type: log
  enabled: true
  paths:
    - /var/log/httpd/error_log-*
  fields:
    codec: plain
    type: tomcat_logs
  processors:
    - drop_event:
        when:
          not:
            regexp:
              message: 'auth_openidc:error'
    - dissect:
        tokenizer: '[%{dateTime}] [%{logLevel}] [pid %{pid|integer}] [client %{clientIP}] %{message}'
        field: "message"
        target_prefix: 'log'
- type: log
  enabled: true
  paths:
    - /var/log/httpd/error_log-*
  fields:
    codec: plain
    type: tomcat_logs
  processors:
    - drop_event:
        when:
          not:
            regexp:
              message: 'proxy:error'
    - dissect:
        tokenizer: '[%{dateTime}] [%{logLevel}] [pid %{pid|integer}] %{message}'
        field: "message"
        target_prefix: 'log'

- type: log
  enabled: true
  paths:
    - /var/log/httpd/error_log-*
  fields:
    codec: plain
    type: tomcat_logs
  processors:
    - drop_event:
        when:
          not:
            regexp:
              message: 'proxy_http:error'
    - dissect:
        tokenizer: '[%{dateTime}] [%{logLevel}] [pid %{pid|integer}] [client %{clientIP}] %{message}'
        field: "message"
        target_prefix: 'log'
- type: log
  enabled: true
  paths:
    - /var/log/httpd/ssl_error_log-*
  fields:
    codec: plain
    type: tomcat_logs
  processors:
    - dissect:
        tokenizer: '[%{dateTime}] [%{logLevel}] [pid %{pid|integer}] %{message}'
        field: "message"
        target_prefix: 'log'
- type: log
  enabled: true
  paths:
    - /var/log/httpd/access_log-*
  fields:
    codec: plain
    type: tomcat_logs
  processors:
    - dissect:
        tokenizer: '%{hostName} %{logicalUsername} %{authenticatedUser} [%{dateTime}] "%{request}" %{statusCode} %{bytes} "-" "%{header}"'
        field: "message"
        target_prefix: 'log'
- type: log
  enabled: true
  paths:
    - /var/log/httpd/ssl_access_log-*
  fields:
    codec: plain
    type: tomcat_logs
  processors:
    - dissect:
        tokenizer: '%{hostName} %{logicalUsername} %{authenticatedUser} [%{dateTime}] "%{request}" %{statusCode} %{bytes}'
        field: "message"
        target_prefix: 'log'
- type: log
  enabled: true
  paths:
    - /var/log/httpd/ssl_request_log-*
  fields:
    codec: plain
    type: tomcat_logs
  processors:
    - dissect:
        tokenizer: '[%{dateTime}] %{logLevel} %{sslProtocol} %{sslCipher} "%{request}" %{bytes}'
        field: "message"
        target_prefix: 'log'
# ======================= Elasticsearch template setting =======================

setup.template:
  name: "tomcat_logs"
  pattern: "tomcat_logs-*"
  overwrite: true
  enabled: false

setup.ilm.enabled: false

setup.template.settings:
  index.number_of_shards: 1
  #index.codec: best_compression
  #_source.enabled: false
# ---------------------------- Elasticsearch Output ----------------------------
output.elasticsearch:
  # Array of hosts to connect to.
  hosts: ["xxxx:9200"]
  index: "tomcat_logs"

```

Before, each log had its own index and it worked well, now I am completely stuck and would appreciate any help.

---

<div class="post-metadata">

**Author:** ![MarianaD](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/marianad/32/42687_2.png) [@MarianaD](https://discuss.elastic.co/u/MarianaD)\
**Post date:** [March 8, 2021, 4:50pm UTC](https://discuss.elastic.co/t/filebeat-to-index-with-different-structures/266565/2 "2021-03-08T16:50:58Z")

</div>

hi @Edi1 , can you check the Filebeat logs, they might offer some info? Can you also set logging level to debug first?

---

<div class="post-metadata">

**Author:** ![Edi1](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/edi1/32/84053_2.png) [@Edi1](https://discuss.elastic.co/u/Edi1)\
**Post date:** [March 9, 2021, 9:00am UTC](https://discuss.elastic.co/t/filebeat-to-index-with-different-structures/266565/3 "2021-03-09T09:00:37Z")

</div>

on starting filebeat -e -d "publisher"

```auto
2021-03-09T09:57:38.983+0100 INFO instance/beat.go:645 Home path: [/usr/share/filebeat] Config path: [/etc/filebeat] Data path: [/var/lib/filebeat] Logs path: [/var/log/filebeat]
2021-03-09T09:57:38.984+0100 INFO instance/beat.go:653 Beat ID: 0da2671d-2708-46c8-a523-07463e36a28a
2021-03-09T09:57:38.984+0100 INFO [seccomp] seccomp/seccomp.go:124 Syscall filter successfully installed
2021-03-09T09:57:38.984+0100 INFO [beat] instance/beat.go:981 Beat info {"system_info": {"beat": {"path": {"config": "/etc/filebeat", "data": "/var/lib/filebeat", "home": "/usr/share/filebeat", "logs": "/var/log/filebeat"}, "type": "filebeat", "uuid": "0da2671d-2708-46c8-a523-07463e36a28a"}}}
2021-03-09T09:57:38.985+0100 INFO [beat] instance/beat.go:990 Build info {"system_info": {"build": {"commit": "aacf9ecd9c494aa0908f61fbca82c906b16562a8", "libbeat": "7.10.2", "time": "2021-01-12T23:11:24.000Z", "version": "7.10.2"}}}
2021-03-09T09:57:38.985+0100 INFO [beat] instance/beat.go:993 Go runtime info {"system_info": {"go": {"os":"linux","arch":"amd64","max_procs":2,"version":"go1.14.12"}}}
2021-03-09T09:57:38.986+0100 INFO [beat] instance/beat.go:997 Host info {"system_info": {"host": {"architecture":"x86_64","boot_time":"2021-01-04T11:17:51+01:00","containerized":false,"name":"poc1","ip":["xxx/8","::1/128","xxx/24","fe80::20c:29ff:fed8:5ea4/64","10.254.1.21/24","fe80::20c:29ff:fed8:5eae/64"],"kernel_version":"4.14.35-2025.401.4.el7uek.x86_64","mac":["00:0c:29:d8:5e:a4","00:0c:29:d8:5e:ae"],"os":{"family":"","platform":"ol","name":"Oracle Linux Server","version":"7.9","major":7,"minor":9,"patch":0},"timezone":"CET","timezone_offset_sec":3600,"id":"81c6c5ad48f94d2287c55291113d5ac8"}}}
2021-03-09T09:57:38.986+0100 INFO [beat] instance/beat.go:1026 Process info {"system_info": {"process": {"capabilities": {"inheritable":null,"permitted":["chown","dac_override","dac_read_search","fowner","fsetid","kill","setgid","setuid","setpcap","linux_immutable","net_bind_service","net_broadcast","net_admin","net_raw","ipc_lock","ipc_owner","sys_module","sys_rawio","sys_chroot","sys_ptrace","sys_pacct","sys_admin","sys_boot","sys_nice","sys_resource","sys_time","sys_tty_config","mknod","lease","audit_write","audit_control","setfcap","mac_override","mac_admin","syslog","wake_alarm","block_suspend","audit_read"],"effective":["chown","dac_override","dac_read_search","fowner","fsetid","kill","setgid","setuid","setpcap","linux_immutable","net_bind_service","net_broadcast","net_admin","net_raw","ipc_lock","ipc_owner","sys_module","sys_rawio","sys_chroot","sys_ptrace","sys_pacct","sys_admin","sys_boot","sys_nice","sys_resource","sys_time","sys_tty_config","mknod","lease","audit_write","audit_control","setfcap","mac_override","mac_admin","syslog","wake_alarm","block_suspend","audit_read"],"bounding":["chown","dac_override","dac_read_search","fowner","fsetid","kill","setgid","setuid","setpcap","linux_immutable","net_bind_service","net_broadcast","net_admin","net_raw","ipc_lock","ipc_owner","sys_module","sys_rawio","sys_chroot","sys_ptrace","sys_pacct","sys_admin","sys_boot","sys_nice","sys_resource","sys_time","sys_tty_config","mknod","lease","audit_write","audit_control","setfcap","mac_override","mac_admin","syslog","wake_alarm","block_suspend","audit_read"],"ambient":null}, "cwd": "/var/log/filebeat", "exe": "/usr/share/filebeat/bin/filebeat", "name": "filebeat", "pid": 5409, "ppid": 4595, "seccomp": {"mode":"filter","no_new_privs":true}, "start_time": "2021-03-09T09:57:38.160+0100"}}}
2021-03-09T09:57:38.986+0100 INFO instance/beat.go:299 Setup Beat: filebeat; Version: 7.10.2
2021-03-09T09:57:38.987+0100 INFO eslegclient/connection.go:99 elasticsearch url: http://xxx:9200
2021-03-09T09:57:38.987+0100 DEBUG [publisher] pipeline/consumer.go:148 start pipeline event consumer
2021-03-09T09:57:38.987+0100 INFO [publisher] pipeline/module.go:113 Beat name: poc1
2021-03-09T09:57:38.988+0100 INFO instance/beat.go:455 filebeat start running.
2021-03-09T09:57:38.988+0100 INFO [monitoring] log/log.go:118 Starting metrics logging every 30s
2021-03-09T09:57:38.990+0100 INFO memlog/store.go:119 Loading data file of '/var/lib/filebeat/registry/filebeat' succeeded. Active transaction id=1060052
2021-03-09T09:57:39.260+0100 INFO memlog/store.go:124 Finished loading transaction log file for '/var/lib/filebeat/registry/filebeat'. Active transaction id=1068894
2021-03-09T09:57:39.261+0100 INFO [registrar] registrar/registrar.go:109 States Loaded from registrar: 56
2021-03-09T09:57:39.261+0100 INFO [crawler] beater/crawler.go:71 Loading Inputs: 11
2021-03-09T09:57:39.262+0100 DEBUG [publisher] pipeline/client.go:231 Pipeline client receives callback 'onFilteredOut' for event: {Timestamp:0001-01-01 00:00:00 +0000 UTC Meta:null Fields:null Private:{Id:native::50414589-64512 PrevId: Finished:true Fileinfo:<nil> Source:/opt/apache-tomcat-9.0.41/logs/catalina.2021-02-15.log Offset:8243 Timestamp:2021-03-09 09:57:17.23585363 +0100 CET TTL:-1ns Type:log Meta:map[] FileStateOS:50414589-64512 IdentifierName:native} TimeSeries:false}
2021-03-09T09:57:39.262+0100 DEBUG [publisher] pipeline/client.go:231 Pipeline client receives callback 'onFilteredOut' for event: {Timestamp:0001-01-01 00:00:00 +0000 UTC Meta:null Fields:null Private:{Id:native::54852818-64512 PrevId: Finished:true Fileinfo:<nil> Source:/opt/apache-tomcat-9.0.41/logs/catalina.2021-02-05.log Offset:34243 Timestamp:2021-03-09 09:57:17.228389144 +0100 CET TTL:-1ns Type:log Meta:map[] FileStateOS:54852818-64512 IdentifierName:native} TimeSeries:false}

```

---

<div class="post-metadata">

**Author:** ![Edi1](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/edi1/32/84053_2.png) [@Edi1](https://discuss.elastic.co/u/Edi1)\
**Post date:** [March 9, 2021, 9:02am UTC](https://discuss.elastic.co/t/filebeat-to-index-with-different-structures/266565/4 "2021-03-09T09:02:31Z")

</div>

On adding new log

```auto
2021-03-09T09:57:39.413+0100 INFO log/input.go:157 Configured paths: [/var/log/httpd/ssl_request_log-*]
2021-03-09T09:57:39.413+0100 INFO [crawler] beater/crawler.go:141 Starting input (ID: 13903261551536249677)
2021-03-09T09:57:39.413+0100 INFO [crawler] beater/crawler.go:108 Loading and starting Inputs completed. Enabled inputs: 10
2021-03-09T09:57:39.413+0100 INFO cfgfile/reload.go:164 Config reloader started
2021-03-09T09:57:39.414+0100 INFO cfgfile/reload.go:224 Loading of config files completed.
2021-03-09T09:57:59.398+0100 DEBUG [publisher] pipeline/client.go:231 Pipeline client receives callback 'onFilteredOut' for event: {Timestamp:0001-01-01 00:00:00 +0000 UTC Meta:null Fields:null Private:{Id:native::50414591-64512 PrevId: Finished:false Fileinfo:0xc000c0f2b0 Source:/var/log/httpd/ssl_error_log-20210307 Offset:4028 Timestamp:2021-03-09 09:57:59.39810365 +0100 CET m=+20.528680756 TTL:-1ns Type:log Meta:map[] FileStateOS:50414591-64512 IdentifierName:native} TimeSeries:false}
2021-03-09T09:57:59.398+0100 INFO log/harvester.go:302 Harvester started for file: /var/log/httpd/ssl_error_log-20210307
2021-03-09T09:58:00.399+0100 INFO [publisher_pipeline_output] pipeline/output.go:143 Connecting to backoff(elasticsearch(http://xxx:9200))
2021-03-09T09:58:00.400+0100 INFO [publisher] pipeline/retry.go:219 retryer: send unwait signal to consumer
2021-03-09T09:58:00.400+0100 INFO [publisher] pipeline/retry.go:223 done
2021-03-09T09:58:00.401+0100 INFO [esclientleg] eslegclient/connection.go:314 Attempting to connect to Elasticsearch version 7.10.2
2021-03-09T09:58:00.452+0100 INFO [license] licenser/es_callback.go:51 Elasticsearch license: Basic
2021-03-09T09:58:00.452+0100 INFO [publisher_pipeline_output] pipeline/output.go:151 Connection to backoff(elasticsearch(http://xxx:9200)) established
2021-03-09T09:58:00.456+0100 DEBUG [publisher] memqueue/ackloop.go:160 ackloop: receive ack [0: 0, 1]
2021-03-09T09:58:00.456+0100 DEBUG [publisher] memqueue/eventloop.go:535 broker ACK events: count=1, start-seq=1, end-seq=1

2021-03-09T09:58:00.456+0100 DEBUG [publisher] memqueue/ackloop.go:128 ackloop: return ack to broker loop:1
2021-03-09T09:58:00.456+0100 DEBUG [publisher] memqueue/ackloop.go:131 ackloop: done send ack

```

---

<div class="post-metadata">

**Author:** ![Edi1](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/edi1/32/84053_2.png) [@Edi1](https://discuss.elastic.co/u/Edi1)\
**Post date:** [March 9, 2021, 9:03am UTC](https://discuss.elastic.co/t/filebeat-to-index-with-different-structures/266565/5 "2021-03-09T09:03:25Z")

</div>

Thank you for reply, I have added the code from filebeat output

---

<div class="post-metadata">

**Author:** ![Edi1](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/edi1/32/84053_2.png) [@Edi1](https://discuss.elastic.co/u/Edi1)\
**Post date:** [March 9, 2021, 9:56am UTC](https://discuss.elastic.co/t/filebeat-to-index-with-different-structures/266565/6 "2021-03-09T09:56:03Z")

</div>

I think I havent also explained well what my goal is.

I want to get multiple different logs in same index as follows:

Index:

```auto
"log": {
"a": "something a",
"b": "something b",
"c": "something c"
}

```

Filebeat log 1

- this will go inside index "log"

```auto
{
"a": "something a",
"b": ------> this filebeat log doesnt have field b and inside proseccor dissect it does not exist,
"c": ----> same for "c"
}

```

Filebeat log b

```auto
{
"a": ----> doesnt exist in this log,
"b": "something b",
"c": "something c"
}

```

Final index:

```auto
"log": {
"a": "something a",
"b": "-",
"c": "-"
},
"log": {
"a": "-",
"b": "something b",
"c": "something c"
}

```

---

<div class="post-metadata">

**Author:** ![Edi1](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/edi1/32/84053_2.png) [@Edi1](https://discuss.elastic.co/u/Edi1)\
**Post date:** [March 9, 2021, 2:00pm UTC](https://discuss.elastic.co/t/filebeat-to-index-with-different-structures/266565/7 "2021-03-09T14:00:01Z")

</div>

Solved. Had some minor mistakes in filebeat

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [April 6, 2021, 4:00pm UTC](https://discuss.elastic.co/t/filebeat-to-index-with-different-structures/266565/8 "2021-04-06T16:00:53Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.