# How to send Json logs to Elastic Search using File Beats without extra fields

**URL:** <https://discuss.elastic.co/t/how-to-send-json-logs-to-elastic-search-using-file-beats-without-extra-fields/239972>\
**Category:** Beats\
**Tags:** filebeat\
**Created:** [July 6, 2020, 8:21am UTC](https://discuss.elastic.co/t/how-to-send-json-logs-to-elastic-search-using-file-beats-without-extra-fields/239972 "2020-07-06T08:21:10Z")\
**Posts on this page:** 6\
**Page:** 1

<div class="post-metadata">

**Author:** ![Ashish\_kapoor](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ashish_kapoor/32/45171_2.png) [@Ashish\_kapoor](https://discuss.elastic.co/u/Ashish_kapoor)\
**Post date:** [July 6, 2020, 8:21am UTC](https://discuss.elastic.co/t/how-to-send-json-logs-to-elastic-search-using-file-beats-without-extra-fields/239972/1 "2020-07-06T08:21:10Z")

</div>

Hi there,

I am trying to send JSON logs to Elastic Search using file beats. My logs file looks like this

{"timestamp":1581386084780,"message":"User 'Test' connected","eventId":107,"metadata":{"userID":"Test","serviceID":"instance-6"}}

I am using file beat to read this log and send it to elastic search. Below is the Filebeat config

\<  
filebeat.inputs:

- type: log  
enabled: true  
paths:
  - C:/Users/Ashish/Downloads/logs/test/audit.log  
json.keys\_under\_root: true  
json.add\_error\_key: true

setup.template.name: "auditbeatasdsadsa-%{[beat.version]}"  
setup.template.pattern: "auditbeatdsadsa-%{[beat.version]}-\*"  
setup.ilm.overwrite: true  
setup.ilm.enabled: auto  
setup.ilm.rollover\_alias: "auditbeatasdsad-%{[beat.version]}"  
setup.ilm.pattern: "{now/M{yyyy.MM}}-000008"

output.elasticsearch:  
hosts: ["[http://localhost:9200](http://localhost:9200)"]  
template.name: filebeat  
template.path: filebeat.template.json  
/\>

I am able to process logs with this configuration, but when I am viewing this data in Kibana. There are so many extra fields that are automatically generated by filebeat. Is there is any way we can control these extra fields?  
\<  
{  
"\_index": "filebeat-7.8.0-2020.07.05-000001",  
"\_type": "\_doc",  
"\_id": "oR0CIHMB6CBCVUUG-yDL",  
"\_version": 1,  
"\_score": 1,  
"\_source": {  
"@timestamp": "2020-07-05T17:25:31.415Z",  
"timestamp": 1581411268592,  
"message": "Performing search for http traffic information over a 120h interval",  
"metadata": {  
"serviceID": "instance-4",  
"userID": "monitor"  
},  
"host": {  
"name": "LAPTOP-I0BND0BP"  
},  
"agent": {  
"ephemeral\_id": "22074664-b74e-4b70-b6dc-8d68d6212953",  
"id": "91f3c264-5bbf-4aed-b63d-d092ca6f3f4b",  
"name": "LAPTOP-I0BND0BP",  
"type": "filebeat",  
"version": "7.8.0",  
"hostname": "LAPTOP-I0BND0BP"  
},  
"log": {  
"offset": 8190,  
"file": {  
"path": "C:\Users\Ashish\Downloads\logs\test\audit.log"  
}  
},  
"eventId": 9,  
"input": {  
"type": "log"  
},  
"ecs": {  
"version": "1.5.0"  
}  
},  
"fields": {  
"cef.extensions.flexDate1": ,  
"netflow.flow\_end\_microseconds": ,  
"netflow.system\_init\_time\_milliseconds": ,  
"netflow.flow\_end\_nanoseconds": ,  
"misp.observed\_data.last\_observed": ,  
"netflow.max\_flow\_end\_microseconds": ,  
"file.mtime": ,  
"aws.cloudtrail.user\_identity.session\_context.creation\_date": ,  
"netflow.min\_flow\_start\_seconds": ,  
"misp.intrusion\_set.first\_seen": ,  
"file.created": ,  
"misp.threat\_indicator.valid\_from": ,  
"process.parent.start": ,  
"azure.auditlogs.properties.activity\_datetime": ,  
"crowdstrike.event.ProcessStartTime": ,  
"zeek.ocsp.update.this": ,  
"crowdstrike.event.IncidentStartTime": ,  
"netflow.observation\_time\_microseconds": ,  
"event.start": ,  
"cef.extensions.agentReceiptTime": ,  
"cef.extensions.oldFileModificationTime": ,  
"checkpoint.subs\_exp": ,  
"event.end": ,  
"netflow.max\_flow\_end\_milliseconds": ,  
"netflow.min\_flow\_start\_nanoseconds": ,  
"zeek.smb\_files.times.changed": ,  
"crowdstrike.event.StartTimestamp": ,  
"netflow.flow\_start\_nanoseconds": ,  
"netflow.flow\_start\_seconds": ,  
"crowdstrike.event.ProcessEndTime": ,  
"zeek.x509.certificate.valid.until": ,  
"misp.observed\_data.first\_observed": ,  
"netflow.exporter.timestamp": ,  
"netflow.monitoring\_interval\_start\_milli\_seconds": ,  
"cef.extensions.oldFileCreateTime": ,  
"event.ingested": ,  
"@timestamp": [  
"2020-07-05T17:25:31.415Z"  
],  
"zeek.ocsp.update.next": ,  
"crowdstrike.event.UTCTimestamp": ,  
"tls.server.not\_before": ,  
"cef.extensions.startTime": ,  
"netflow.min\_flow\_start\_milliseconds": ,  
"azure.signinlogs.properties.created\_at": ,  
"cef.extensions.endTime": ,  
"suricata.eve.tls.notbefore": ,  
"zeek.kerberos.valid.from": ,  
"cef.extensions.fileCreateTime": ,  
"misp.threat\_indicator.valid\_until": ,  
"crowdstrike.event.EndTimestamp": ,  
"misp.campaign.last\_seen": ,  
"cef.extensions.deviceReceiptTime": ,  
"netflow.observation\_time\_seconds": ,  
"crowdstrike.metadata.eventCreationTime": ,  
"cef.extensions.fileModificationTime": ,  
"tls.client.not\_before": ,  
"zeek.smb\_files.times.created": ,  
"zeek.smtp.date": ,  
"netflow.collection\_time\_milliseconds": ,  
"zeek.pe.compile\_time": ,  
"netflow.max\_flow\_end\_seconds": ,  
"tls.client.not\_after": ,  
"netflow.flow\_start\_milliseconds": ,  
"event.created": ,  
"package.installed": ,  
"zeek.kerberos.valid.until": ,  
"suricata.eve.flow.end": ,  
"netflow.observation\_time\_milliseconds": ,  
"netflow.flow\_start\_microseconds": ,  
"tls.server.not\_after": ,  
"netflow.flow\_end\_seconds": ,  
"process.start": ,  
"suricata.eve.tls.notafter": ,  
"zeek.snmp.up\_since": ,  
"azure.enqueued\_time": ,  
"netflow.max\_flow\_end\_nanoseconds": ,  
"misp.intrusion\_set.last\_seen": ,  
"netflow.min\_flow\_start\_microseconds": ,  
"netflow.observation\_time\_nanoseconds": ,  
"cef.extensions.managerReceiptTime": ,  
"file.accessed": ,  
"netflow.flow\_end\_milliseconds": ,  
"misp.campaign.first\_seen": ,  
"netflow.min\_export\_seconds": ,  
"suricata.eve.flow.start": ,  
"suricata.eve.timestamp": [  
"2020-07-05T17:25:31.415Z"  
],  
"cef.extensions.deviceCustomDate1": ,  
"cef.extensions.deviceCustomDate2": ,  
"netflow.monitoring\_interval\_end\_milli\_seconds": ,  
"file.ctime": ,  
"crowdstrike.event.IncidentEndTime": ,  
"zeek.smb\_files.times.accessed": ,  
"zeek.ocsp.revoke.time": ,  
"zeek.x509.certificate.valid.from": ,  
"netflow.max\_export\_seconds": ,  
"zeek.smb\_files.times.modified": ,  
"kafka.block\_timestamp": ,  
"misp.report.published":   
}  
}  
/\>  
How to remove these extra fields in fields tag, so that i can have only required fields.

---

<div class="post-metadata">

**Author:** ![kvch](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kvch/32/72058_2.png) [@kvch](https://discuss.elastic.co/u/kvch)\
**Post date:** [July 6, 2020, 3:34pm UTC](https://discuss.elastic.co/t/how-to-send-json-logs-to-elastic-search-using-file-beats-without-extra-fields/239972/2 "2020-07-06T15:34:05Z")

</div>

Could you please format your configuration `</>`?

---

<div class="post-metadata">

**Author:** ![Ashish\_kapoor](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ashish_kapoor/32/45171_2.png) [@Ashish\_kapoor](https://discuss.elastic.co/u/Ashish_kapoor)\
**Post date:** [July 7, 2020, 7:27am UTC](https://discuss.elastic.co/t/how-to-send-json-logs-to-elastic-search-using-file-beats-without-extra-fields/239972/3 "2020-07-07T07:27:39Z")

</div>

```auto
    filebeat.inputs:
    - type: log
      paths:
       - C:/Users/Ashish/Downloads/Basefarm_audit_logs/Basefarm_audit_logs/OAG/audit.log
      json.keys_under_root: true
      json.message_key: event
      json.add_error_key: true
       
    output.elasticsearch:
      hosts: ["http://localhost:9200"]

```

---

<div class="post-metadata">

**Author:** ![Ashish\_kapoor](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ashish_kapoor/32/45171_2.png) [@Ashish\_kapoor](https://discuss.elastic.co/u/Ashish_kapoor)\
**Post date:** [July 7, 2020, 7:28am UTC](https://discuss.elastic.co/t/how-to-send-json-logs-to-elastic-search-using-file-beats-without-extra-fields/239972/4 "2020-07-07T07:28:38Z")

</div>

```auto
filebeat.inputs:

# Each - is an input. Most options can be set at the input level, so
# you can use different inputs for various configurations.
# Below are the input specific configurations.

- type: log

  # Change to true to enable this input configuration.
  paths:
   - C:/Users/Ashish/Downloads/Basefarm_audit_logs/Basefarm_audit_logs/OAG/audit.log
  
  json.keys_under_root: true
  json.message_key: event
  json.add_error_key: true
      
output.elasticsearch:
  hosts: ["http://localhost:9200"]

```

---

<div class="post-metadata">

**Author:** ![kumarabhi](https://avatars.discourse-cdn.com/v4/letter/k/6a8cbe/32.png) [@kumarabhi](https://discuss.elastic.co/u/kumarabhi)\
**Post date:** [July 9, 2020, 4:50am UTC](https://discuss.elastic.co/t/how-to-send-json-logs-to-elastic-search-using-file-beats-without-extra-fields/239972/5 "2020-07-09T04:50:16Z")

</div>

You should use the drop\_fields processor to remove unwanted fields.

Please search for drop\_fields in [https://www.elastic.co/guide/en/beats/filebeat/current/filebeat-reference-yml.html](https://www.elastic.co/guide/en/beats/filebeat/current/filebeat-reference-yml.html)  
You will probably be adding something provided below

```auto
#================================ Processors =====================================

# Configure processors to enhance or manipulate events generated by the beat.

    processors:
      - drop_fields:
          fields: ["host.name", "ecs.version", "agent.version", "agent.type", "agent.id", "agent.ephemeral_id", "agent.hostname", "input.type"]
    # - add_host_metadata: ~
    # - add_cloud_metadata: ~

```

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [August 6, 2020, 6:50am UTC](https://discuss.elastic.co/t/how-to-send-json-logs-to-elastic-search-using-file-beats-without-extra-fields/239972/6 "2020-08-06T06:50:18Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.