Filebeat for just forwarding Full text log

Elastic Stack Beats
#1

Hello team,

Is there any way using filebat (7.10), for just forwarding logs ?
The main goals is :

  • shipping log from differents sources (file log text)
  • forward them to another (file log text), without do any changes inside

i will also remove the metadata from filebeat !!
here my config file :

    filebeat.inputs:
- type: log
  paths:
    - /ines-logs-nfs/syslog-ng/uaaifnd11rou01/LogFonct.log
    - /ines-logs-nfs/syslog-ng/uaaifnd11rou02/LogFonct.log

output.file:
  path: "/ines-logs-nfs/syslog-ng/"
  filename: all.log
  permissions: 0600

Any ideas about this ??

#2

Sure, you can do this. And you can use the drop_fields processor to remove any unwanted fields from events before they are sent to the output.

#3

@shaunak thanks for replying !
i have tested this in my config file, but didn't work !! filebeat service failed

processors:
  - drop_fields:
      when:
        has_fields:
      fields: ["log.offset", "input.type", "fileset.name"]
      ignore_missing: false

any idea ?

Regards
sai

#4

i have changed the configuration (drop_fields) to

processors:
  - drop_fields:
        fields: ["log.offset", "input.type", "fileset.name", "log.source.address", "log.flags"]

but didn't works, i still show on the output file the medatada (tomestamps version, beat etc ....) :confused:

#5

i can summarize here, what i need .
see below the output generated by filebeat :

{"@timestamp":"2020-11-26T13:19:24.149Z","@metadata":{"beat":"filebeat","type":"_doc","version":"7.10.0"},"log":{"file":{"path":"/logs-nfs/syslog-ng/XXXX/appli.log"}},"message":"2020-11-26T14:19:23+01:00 servermaster CORE: TimeStamp=\"2020-11-26 14:19:23.560\" TraceID=\"TIDcb28e32379fd73bdbe896cqsdsdd979ffc\" LogType=\"APPLI_LOG\" MessType=\"QUEING\" StateInd=\"0\" ErrCode=\"0\" ErrMess=\"Operation successful\" Serial=\"TID04b55dab01dsdfdsac13141b\" TicketNBR=\"4\"","input":{},"agent":{"hostname":"filebeatserver","ephemeral_id":"2488c327-71c6-4bcb-a747-eee5b86190a0","id":"91f74f15-737a-42df-b5d7-c629b3dd9e74","name":"filebeatserver","type":"filebeat","version":"7.10.0"},"ecs":{"version":"1.6.0"},"host":{"name":"filebeatserver"}}

the needed is to keep just this part of output , and remove the rest :

2020-11-26T14:19:23+01:00 servermaster CORE: TimeStamp=\"2020-11-26 14:19:23.560\" TraceID=\"TIDcb28e32379fd73bdbe896cqsdsdd979ffc\" LogType=\"APPLI_LOG\" MessType=\"QUEING\" StateInd=\"0\" ErrCode=\"0\" ErrMess=\"Operation successful\" Serial=\"TID04b55dab01dsdfdsac13141b\" TicketNBR=\"4\"

many thanks for helps

sai

#6

any help plz ??

sai

#7

I think you need to include @metadata in your drop_fields processor's fields setting? Also, the @timestamp field cannot be dropped, as noted in the processor's documentation: https://www.elastic.co/guide/en/beats/filebeat/current/drop-fields.html

#8

hi ,
@metadata in drop_fields didn't resolve the pb :confused:
you think it is a bug ?

regards

#9

any with this plz ?

Regards

#10

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.