Problem deleting fields or renaming (from filebeat)

Elastic Stack Logstash
#1

Hello there,

I'm configuring filebeat. before i used logstashforwarder.
In logstash i'm getting logs BUT, if i set filebeat to add a new fields, i receive it like field.nameIchoose
If i use mutate filter to rename that field it doesn't work. Moreover if i try to delete it i can't.

It means that fields, created in filebeat can't be deleted?

Thank you all

#2

You're not giving your configuration so we can only guess, and my guess is that you're not using the current field reference syntax for nested fields. Read more at https://www.elastic.co/guide/en/logstash/current/event-dependent-configuration.html#logstash-config-field-references.

To get your fields at the top level from the start, use the fields_under_root configuration option in Filebeat.

#3

Here is my config
There's something i'm doing wrong, because it avoids some configurations, such as document_type..

filebeat:

List of prospectors to fetch data.

prospectors:

paths:

  • /opt/tomcat/logs/tomcat.*.log
    input_type: log
    document_type: filebeat
    scan_frequency: 60s
    fields:
    host: xxxx
    fields_under_root: true
    multiline:
    #pattern: "(^%{TIMESTAMP_ISO8601})|(^%{IPV4})"
    pattern: "()|(^(?:[0-9]{1,3}.){3}[0-9]{1,3}$)"
    negate: true
    match: after
    max_lines: 200
    timeout: 4s
    tail_files: true
    registry_file: /var/lib/filebeat/registry

output:
logstash:
hosts: ["x.x.x.x:5044"]
shipper:
logging:
files:
rotateeverybytes: 10485760 # = 10MB

#4

Please format your config as code so the indentation isn't stripped away. There's a toolbar button for that.

#5 
filebeat:
 # List of prospectors to fetch data.
 prospectors:
 -
  paths:
  - /opt/tomcat/logs/tomcat.*.log
 input_type: log
 document_type: filebeat
 scan_frequency: 60s
 fields:
  host: xxxx
 fields_under_root: true
 multiline:
    pattern: (^([0-9]{4})(-?)(1[0-2]|0[1-9])\2(3[01]|0[1-9]|[12][0-9])\s(2[0-3]|[01][0-9]):?([0-5][0-9]):?([0-5][0-9])$)|(^(?:[0-9]{1,3}\.){3}[0-9]{1,3}$)
    negate: true
    match: after
    max_lines: 200
    timeout: 4s

 tail_files: true
 registry_file: /var/lib/filebeat/registry

output:
   logstash:
    hosts: ["x.x.x.x:5044"]
shipper:
 logging:
   files:
     rotateeverybytes: 10485760 # = 10MB
#6

The indentation is off. input_type and document_type should be on the same level as paths. See the example in the docs.

filebeat:
  # List of prospectors to fetch data.
  prospectors:
  -
    paths:
      - /opt/tomcat/logs/tomcat.*.log
    input_type: log
    document_type: filebeat
    scan_frequency: 60s
    fields:
      host: aleadigital
    fields_under_root: true
#7

Hi Magnus,

right, that was the problem. Thanks

#8