Simple scenario where Filebeat transfers log lines into Elastic search - does not work

Hi,

my goal is to load Tshark logs into Elasticsearch using Filebeat. So here is my test file (for some lines port column is empty):

2024-11-21 19:29:28 192.168.1.144	142.250.186.196	52873\443
2024-11-21 19:29:28 142.250.186.196	192.168.1.144	443\52873
2024-11-21 19:29:28 192.168.1.185	224.0.0.7
2024-11-21 19:29:28 192.168.1.140	239.255.255.250

My filebeat config:

filebeat.inputs:
  - type: log
    enabled: true
    paths:
      - "G:/xx/test.csv" 

processors:
  - decode_csv_fields:
     fields:
       message: "message"
     separator: "\t"
     ignore_missing: false
     overwrite_keys: true
     trim_leading_space: true
     fail_on_error: true

  - extract_array:
      field: message
      mappings:
        created_at: 0
        ip_src: 1
        ip_dst: 2
        port: 3

  - drop_fields:
      fields: [ "message" ]

  - drop_event:
      when:
        equals:
          ip_src: ""

  
output.elasticsearch:
  hosts: ["http://localhost:9200"] 
  username: "elastic"             
  password: "elastic"             
  index: "packets-%{+yyyy.MM.dd}" 
  scan_frequency: 5s              
  data_stream.enabled: true 

setup.template.enabled: true
setup.template.name: "wireshark_template"
setup.template.pattern: "wireshark_template*"
setup.template.overwrite: true  

logging:
  level: info            #(debug, info, warn, error)
  to_files: true         
  files:
    path: "G:/xx/logtxt"  
    name: filebeat        
    keepfiles: 7          # No of logs

And my template

PUT _index_template/wireshark_template
{
  "index_patterns": ["packets-*"], 
  "data_stream": {},  
  "template": {
    "mappings": {
      "properties": {
        "@timestamp": {
          "type": "date"
        },
        "packet_date": {
          "type": "date",
          "format": "yyyy-MM-dd HH:mm:ss"
        },
        "ip_src": {
          "type": "ip"
        },
        "ip_dst": {
          "type": "ip"
        },
        "port": {
          "type": "text"
        }
      }
    }
  }
}

OK, let's start it:

filebeat.exe -c filebeat.yml

It creates no indices in Elasticsearch=>Indices.

When I enter Stack Management=>Index Management=>Data streams and click on it then on 'Indices' on left pane I see '.ds-packets-2024.11.22-2024.11.22-000001'.
But wen I 'discover index' I see only some trash in documents created like 'agent.id', 'agent.type'...'log.offset'. All my fields like 'ip_src' or 'port' are empty.
I see nothing wrong in log file.

What I'm doing wrong?

Can u give more info about OS and logs of elasticsearch

OS: Windows

There's nothing bad in Elasticsearch log:

[2024-11-22T10:07:01,099][INFO ][o.e.c.m.MetadataIndexTemplateService] [SOME-PC] adding index template [wireshark_template] for index patterns [wireshark_template*]
[2024-11-22T10:07:01,854][INFO ][o.e.c.m.MetadataCreateIndexService] [SOME-PC] [.ds-wireshark_template-2024.11.22-000001] creating index, cause [initialize_data_stream], templates [wireshark_template], shards [1]/[1]
[2024-11-22T10:07:01,856][INFO ][o.e.c.m.MetadataCreateDataStreamService] [SOME-PC] adding data stream [wireshark_template] with write index [.ds-wireshark_template-2024.11.22-000001], backing indices [], and aliases []
[2024-11-22T10:07:02,614][INFO ][o.e.i.m.MapperService    ] [SOME-PC] [.ds-wireshark_template-2024.11.22-000001] reloading search analyzers
[2024-11-22T10:07:02,617][INFO ][o.e.x.i.IndexLifecycleTransition] [SOME-PC] moving index [.ds-wireshark_template-2024.11.22-000001] from [null] to [{"phase":"new","action":"complete","name":"complete"}] in policy [filebeat]
[2024-11-22T10:07:03,528][INFO ][o.e.x.i.IndexLifecycleTransition] [SOME-PC] moving index [.ds-wireshark_template-2024.11.22-000001] from [{"phase":"new","action":"complete","name":"complete"}] to [{"phase":"hot","action":"unfollow","name":"branch-check-unfollow-prerequisites"}] in policy [filebeat]
[2024-11-22T10:07:03,758][INFO ][o.e.c.m.MetadataCreateIndexService] [SOME-PC] [packets-2024.11.22] creating index, cause [auto(bulk api)], templates [], shards [1]/[1]
[2024-11-22T10:07:04,182][INFO ][o.e.i.m.MapperService    ] [SOME-PC] [packets-2024.11.22] reloading search analyzers
[2024-11-22T10:07:04,184][INFO ][o.e.x.i.IndexLifecycleTransition] [SOME-PC] moving index [.ds-wireshark_template-2024.11.22-000001] from [{"phase":"hot","action":"unfollow","name":"branch-check-unfollow-prerequisites"}] to [{"phase":"hot","action":"rollover","name":"check-rollover-ready"}] in policy [filebeat]
[2024-11-22T10:07:05,972][INFO ][o.e.c.m.MetadataMappingService] [SOME-PC] [packets-2024.11.22/gxiVA-L3SYeRLiAR7SVTnA] create_mapping
[2024-11-22T10:09:00,784][INFO ][o.e.c.m.MetadataDeleteIndexService] [SOME-PC] [packets-2024.11.22/gxiVA-L3SYeRLiAR7SVTnA] deleting index
[2024-11-22T10:13:49,783][INFO ][o.e.c.m.MetadataIndexTemplateService] [SOME-PC] adding index template [whireshark_template] for index patterns [packets-*]
[2024-11-22T10:14:11,156][INFO ][o.e.c.m.MetadataCreateIndexService] [SOME-PC] [.ds-packets-2024.11.22-2024.11.22-000001] creating index, cause [initialize_data_stream], templates [whireshark_template], shards [1]/[1]
[2024-11-22T10:14:11,156][INFO ][o.e.c.m.MetadataCreateDataStreamService] [SOME-PC] adding data stream [packets-2024.11.22] with write index [.ds-packets-2024.11.22-2024.11.22-000001], backing indices [], and aliases []
[2024-11-22T10:14:11,880][INFO ][o.e.i.m.MapperService    ] [SOME-PC] [.ds-packets-2024.11.22-2024.11.22-000001] reloading search analyzers
[2024-11-22T10:14:12,665][INFO ][o.e.c.m.MetadataMappingService] [SOME-PC] [.ds-packets-2024.11.22-2024.11.22-000001/mivHZpsrT3SPnMclsdqRyg] update_mapping [_doc]
[2024-11-22T10:36:22,051][WARN ][o.e.t.ThreadPool         ] [SOME-PC] timer thread slept for [9.7s/9748ms] on absolute clock which is above the warn threshold of [5000ms]

No more lines even after running
filebeat.exe -c filebeat.yml

Do you have anything in Filebeat logs?

Also, did you test your processors using the console output before configuring the elasticsearch output?

I would remove all processors and see if you start receving the raw message, after that would add one by one.

I've changed template name to 'packets_template' and it started working. What ***? Filebeat has hardcoded only one template name?

Now in filebeat config:

setup.template.name: "packets_template"
setup.template.pattern: "packets_template*"

There is as well directive:

setup.template.overwrite: true

I see that it can really create template if not exists or recreate it (damage template I created?). I do not get what is the sense of this command; templates are created on Elasticsearch side, I do not think filebeat gives us possibility to create templates.

OK, now time to play with mappings...

OK, some progress. I found these mapping comes from 'fields.yml'. So defined mine:

# Wireshark fields mapping [Tshark]
- key: wireshark
  title: WIRESHARK
  description: Wireshark debug fields.
  fields:
  - name: '@timestamp'
    level: core
    required: true
    type: date
    description: 'Date/time when the event originated.
      This is the date/time extracted from the event, typically representing when
      the event was generated by the source.
      If the event source has no original timestamp, this value is typically populated
      by the first time the event was received by the pipeline.
      Required field for all events. 2016-05-23T08:05:34.853Z'
    example: '2024-11-22T18:36:22.853Z'
    default_field: true
  - name: ip_src
    level: core
    type: ip
    description: Source IP.
    example: 1.2.3.4
    default_field: true
  - name: ip_dst
    level: core
    type: ip
    description: Destination IP.
    example: 1.2.3.4
    default_field: true
  - name: ports
    level: core
    required: true
    type: text
    description: Source and destination ports

But no matter I do I have IP as 'text' in Elasticsearch that turns into... keyword in Dashboard.
Yes, I removed all mappings on Elasticsearch side and I'm using these from Filebeat:

setup.template.enabled: true
setup.template.fields: "wireshark_fields.yml"
setup.template.name: "packets_template"
setup.template.pattern: "packets_template*"
setup.template.overwrite: true

I do not know if using IP format brings any advantage but would be nice if it works as suppose to.
So NO TEMPLATE is required in Elasticsearch if Filebeat has one? And why only one name, "packets_template", works