Filebeat show unwanted logs

[abu.sayeed]

#cat /etc/filebeat/filebeat.yml

filebeat.modules:
filebeat.prospectors:

• input_type: log
  paths:

  • /home/local/user/nsms/logs/Logs/nsms-server.log
    #include_lines: ["GP+"]
    #- c:\programdata\elasticsearch\logs*
    output.logstash:
    hosts: ["10.33.254.83:5044"]

  ssl.certificate_authorities: ["/etc/pki/tls/certs/logstash-forwarder.crt"]

logging.to_files: true
logging.files:
rotateeverybytes: 10485760 # = 10MB

keepfiles: 7

##########

Kibana discover show:

July 31st 2017, 16:54:28.274
@timestamp:July 31st 2017, 16:54:28.27 offset: 10,067,175 @version:
1 input_type: beat.hostname:vNTDACLSWEBP005 beat.name:vNTDACLSWEBP005
beat.version:5.4.3 host: vNTDACLSWEBP005 source: /home/local/user/nsms/logs/Logs/nsms-server.log message: 2017-07-31 16:54:26,865 : [INFO ] http-nio-8080-exec-57 [c.n.s.g.SmsGatewayGPHttpGet:281] Setting cmp param value for smsGateway [GP:DAC:GET}
[GP:DAC:GET} | SmsResult

But I want:
July 31st 2017, 16:54:28.274

host: vNTDACLSWEBP005
source: /home/local/user/nsms/logs/Logs/nsms-server.log
message: 2017-07-31 16:54:26,865 : [INFO ] http-nio-8080-exec-57 [c.n.s.g.SmsGatewayGPHttpGet:281] Setting cmp param value for smsGateway [GP:DAC:GET}
[GP:DAC:GET} | SmsResult

How can remove unwanted logs. Like:
@timestamp:July 31st 2017, 16:54:28.27
offset: 10,067,175
@version: 5.4.3
beat.hostname:vNTDACLSWEBP005
beat.name:vNTDACLSWEBP005
beat.version:5.4.3

Please anybody help me.

Thanks all.

[leandrojmp (Leandro Pereira)]

I don't think you can remove the @timestamp, it is a necessary field, and I'm not sure about the @version field either, but the beat fields and the offset can be removed using mutate in your filter pipeline on logstash.

filter {
    mutate {
       remove_field => [ "beat", "offset"]
    }
}

[abhiroyg]

Except @timestamp, you can remove any other field including @version.
I tried removing @timestamp, but I got errors. So, I ended up keeping it.
Please use the code suggested by @leandrojmp for removing fields.

[abu.sayeed]

@abhiroyg

@version

no change after using mutate in my filter pipeline on logstash.

mutate {
remove_field => [ "beat", "offset"]
}

[abhiroyg]

Hi,

Please add the field name you want to remove to that list. Here, since you want to remove @version. Your mutate filter would look like this.

mutate {
    remove_field => [ "@version", "beat", "offset" ]
}

Hope it helps.

[abu.sayeed]

Logstash-pipeline conf

filter {
mutate {
remove_field => [ "_id", "_type", "_index", "_score", "@version", "beat", "offset"]
}
}

@timestamp August 1st 2017, 12:56:54.893
t _id AV2clVn9SwgmMqjSZLw1
t _index filebeat-2017.08.01

_score -

t _type log
t host vNTDACLSWEBP005
t input_type log
t message 2017-08-01 12:56:54,133 : [INFO ] http-nio-8080-exec-7 [c.n.j.JdbcService:-1] Executing SP ACT_sms with Action [UPDATE]
t source /home/local/user/nsms/logs/Logs/nsms-server.log
t tags beats_input_codec_plain_applied
t type log

how can remove highlights log.
thanks all

[abhiroyg]

Hi,

You can't remove _id, _index, _score, _type. They are required by elasticsearch. You can try reading elasticsearch documentation to know why.

You can remove input_type, type, tags using the syntax I mentioned in my previous reply.

Generally tags can be helpful. So, if you want to disable that particular tag (beats_input_codec_plain_applied). Add include_codec_tag => false to your beats plugin.
Example:

input {
    beats {
        include_codec_tag => false
        <other configuration>
    }
}

Hope it helps.

[abu.sayeed]

It's works. Thanks all for helping me. I'm very very glad for your help.

[abu.sayeed]

How can split log in column permanently?

Kibana Discovery logs format:

host:VM_5 source:/home/user/user.log message:smsStatus : [SENT]

But I wish log format:

host: source: message:
VM_5 /home/user/user.log smsStatus : [SENT]

[abhiroyg]

Hi,

Can you please post this as a new question in Kibana ?

Thanks

[abu.sayeed]

OK Thanks