Incorrect mapping of fields by Kibanna send from logstash

[asad_ali]

Problem
I have very simple log file for testing purposes.

Below as:-

2008-09-15T11:30:00Z sarah
2008-09-15T12:18:00Z jessica
2008-09-15T13:20:00Z parker lee

On testing

stdout of logstash is

{
"type" => "log",
"input" => {
"type" => "log"
},
"tags" => [
[0] "beats_input_codec_plain_applied"
],
"@timestamp" => 2008-09-15T11:30:00.000Z,
"ecs" => {
"version" => "1.0.0"
},
"host" => {
"name" => "elk"
},
"agent" => {
"ephemeral_id" => "de724cef-e507-4b82-909d-700ab2f14f0c",
"type" => "filebeat",
"id" => "02437fe5-f069-405f-bd06-14b5501db678",
"version" => "7.1.1",
"hostname" => "elk"
},
"@version" => "1",
"log" => {
"file" => {
"path" => "/root/filebeat-7.1.1-linux-x86_64/sample.log"
},
"offset" => 0
},
"line" => "sarah"
}
{
"type" => "log",
"input" => {
"type" => "log"
},
"tags" => [
[0] "beats_input_codec_plain_applied"
],
"@timestamp" => 2008-09-15T12:18:00.000Z,
"ecs" => {
"version" => "1.0.0"
},
"host" => {
"name" => "elk"
},
"agent" => {
"ephemeral_id" => "de724cef-e507-4b82-909d-700ab2f14f0c",
"type" => "filebeat",
"id" => "02437fe5-f069-405f-bd06-14b5501db678",
"version" => "7.1.1",
"hostname" => "elk"
},
"@version" => "1",
"log" => {
"file" => {
"path" => "/root/filebeat-7.1.1-linux-x86_64/sample.log"
},
"offset" => 27
},
"line" => "jessica"
}
{
"type" => "log",
"input" => {
"type" => "log"
},
"tags" => [
[0] "beats_input_codec_plain_applied"
],
"@timestamp" => 2008-09-15T13:20:00.000Z,
"ecs" => {
"version" => "1.0.0"
},
"host" => {
"name" => "elk"
},
"agent" => {
"ephemeral_id" => "de724cef-e507-4b82-909d-700ab2f14f0c",
"type" => "filebeat",
"id" => "02437fe5-f069-405f-bd06-14b5501db678",
"version" => "7.1.1",
"hostname" => "elk"
},
"@version" => "1",
"log" => {
"file" => {
"path" => "/root/filebeat-7.1.1-linux-x86_64/sample.log"
},
"offset" => 56
},
"line" => "parker lee"
}

curl -XGET 'localhost:9200/logstash-2015.01.04/_search?pretty&q=response=200'

{
  "took" : 3,
  "timed_out" : false,
  "_shards" : {
    "total" : 1,
    "successful" : 1,
    "skipped" : 0,
    "failed" : 0
  },
  "hits" : {
    "total" : {
      "value" : 0,
      "relation" : "eq"
    },
    "max_score" : null,
    "hits" : [ ]
  }
}

curl 'http://localhost:9200/_search?pretty'

output see file

Filebin | hdnbfbhn0xtrzitp

---

My expectation, is that beside timestamp and line it should not match or add other fields the other fields its matching is more then 30 given below and also shown in complete file in link above

` "fields" : """[{"name":"@timestamp","type":"date","count":0,"scripted":false,"searchable":true,"aggregatable":true,"readFromDocValues":true},{"name":"@version","type":"string","count":0,"scripted":false,"searchable":true,"aggregatable":true,"readFromDocValues":true},{"name":"_id","type":"string","count":0,"scripted":false,"searchable":true,"aggregatable":true,"readFromDocValues":false},{"name":"_index","type":"string","count":0,"scripted":false,"searchable":true,"aggregatable":true,"readFromDocValues":false},{"name":"_score","type":"number","count":0,"scripted":false,"searchable":false,"aggregatable":false,"readFromDocValues":false},{"name":"_source","type":"_source","count":0,"scripted":false,"searchable":false,"aggregatable":false,"readFromDocValues":false},{"name":"_type","type":"string","count":0,"scripted":false,"searchable":true,"aggregatable":true,"readFromDocValues":false},{"name":"agent.ephemeral_id","type":"string","count":0,"scripted":false,"searchable":true,"aggregatable":false,"readFromDocValues":false},{"name":"agent.ephemeral_id.keyword","type":"string","count":0,"scripted":false,"searchable":true,"aggregatable":true,"readFromDocValues":true},{"name":"agent.hostname","type":"string","count":0,"scripted":false,"searchable":true,"aggregatable":false,"readFromDocValues":false},{"name":"agent.hostname.keyword","type":"string","count":0,"scripted":false,"searchable":true,"aggregatable":true,"readFromDocValues":true},{"name":"agent.id","type":"string","count":0,"scripted":false,"searchable":true,"aggregatable":false,"readFromDocValues":false},{"name":"agent.id.keyword","type":"string","count":0,"scripted":false,"searchable":true,"aggregatable":true,"readFromDocValues":true},{"name":"agent.type","type":"string","count":0,"scripted":false,"searchable":true,"aggregatable":false,"readFromDocValues":false},{"name":"agent.type.keyword","type":"string","count":0,"scripted":false,"searchable":true,"aggregatable":true,"readFromDocValues":true},{"name":"agent.version","type":"string","count":0,"scripted":false,"searchable":true,"aggregatable":false,"readFromDocValues":false},...........

---

On elastic-search logs I get

[2019-06-18T03:34:21,099][DEBUG][o.e.x.s.a.a.OptOutQueryCache] [elk] [.kibana_task_manager] not opting out of the query cache; authorization is not allowed.

[asad_ali]

Logstash.yml

filebeat.inputs:
- type: log
  enabled: true
  paths:
    - /root/filebeat-7.1.1-linux-x86_64/sample.log
  fields_under_root: true
  fields:
    type: log
filebeat.config.modules:
  path: ${path.config}/modules.d/*.yml
  reload.enabled: false
setup.template.settings:
  index.number_of_shards: 1
setup.kibana:
output.logstash:
 hosts: ["localhost:5044"]
processors:

[Mario_Castro]

Hi @asad_ali

I'm not sure about the problem itself. Are you having issues with Logstash or with Filebeat? That Logstash.yml looks like a Filebeat config, not Logstash.

[sjabiulla (JB)]

please post your Logstash.yml