Unable to insert log data using filebeat

[NileshG (Nilesh)]

Hi,
I am trying to insert log data using filebeat in an ElascticSearch.
The index is created but the data differ than the actual one.

My Log is

2020-01-17T17:13:43.218+0530	INFO	crawler/crawler.go:106	Loading and starting Inputs completed. Enabled inputs: 1
2020-01-17T17:13:43.218+0530	INFO	cfgfile/reload.go:171	Config reloader started
2020-01-17T17:13:43.218+0530	INFO	cfgfile/reload.go:226	Loading of config files completed.
2020-01-17T17:13:43.229+0530	INFO	log/harvester.go:251	Harvester started for file: C:\Users\Nilesh.Gunjkar\Desktop\ElasticSearch\LogsData\Book3.xlsx
2020-01-17T17:13:46.182+0530	INFO	add_cloud_metadata/add_cloud_metadata.go:89	add_cloud_metadata: hosting provider type not detected.

My Config File is
###################### Filebeat Configuration Example #########################

This file is an example configuration file highlighting only the most common

options. The filebeat.reference.yml file from the same directory contains all the

supported options with more comments. You can use it as a reference.

You can find the full configuration reference here:

https://www.elastic.co/guide/en/beats/filebeat/index.html

For more available modules and options, please see the filebeat.reference.yml sample

configuration file.

#=========================== Filebeat inputs =============================

filebeat.inputs:

Each - is an input. Most options can be set at the input level, so

you can use different inputs for various configurations.

Below are the input specific configurations.

• type: log

  Change to true to enable this input configuration.

  enabled: true

  Paths that should be crawled and fetched. Glob based paths.

  paths:

  • D:\Desktop\ElasticSearch\LogsData*
    #- c:\programdata\elasticsearch\logs*

  Exclude lines. A list of regular expressions to match. It drops the lines that are

  matching any regular expression from the list.

  #exclude_lines: ['^DBG']

  Include lines. A list of regular expressions to match. It exports the lines that are

  matching any regular expression from the list.

  #include_lines: ['^ERR', '^WARN']

  Exclude files. A list of regular expressions to match. Filebeat drops the files that

  are matching any regular expression from the list. By default, no files are dropped.

  #exclude_files: ['.gz$']

#================================ Outputs =====================================

Configure what output to use when sending the data collected by the beat.

#-------------------------- Elasticsearch output ------------------------------
output.elasticsearch:

Array of hosts to connect to.

hosts: ["elk.dev.echdev1.com"]

Optional protocol and basic auth credentials.

#protocol: "http"
#username: ""
#password: ""

[shaunak (Shaunak Kashyap)]

Hi @NileshG, welcome to the Elastic community forums!

@NileshG

The index is created but the data differ than the actual one.

Would you mind telling us what data you're seeing in the index and what you expect to see? Perhaps you could post a sample document in the index and the corresponding expected data?

Also, what version of Filebeat and Elasticsearch are you running?

Thanks,

Shaunak

[NileshG (Nilesh)]

Hi @Shaunak,

Thank you for your assistance.

I am using Filebeat version 7.5.1 and Elasticsearch version 6.8.3.

The below screenshot contains the data which I am trying to insert.

Please check the following screenshot to see the actual data:

Following screenshot shows the data which we got in Elasticsearch:

Some additional parameters are automatically inserted with the data (highlighted in a yellow color box)