Number of docs count does not match number of json file records

Elastic Stack Beats
1

Hi. I am trying to load a json file with 8 records from filebeats to logstash (latest verion 7.13). It looks like i have no errors but when i check number of records at Kibana Index Management i see docs counts = 1 (also i checked at Discover and there is indeed just one document/event).

json file

Filebeat yml file is pretty straight forward:

# ============================== Filebeat inputs ===============================

filebeat.inputs:

# Each - is an input. Most options can be set at the input level, so
# you can use different inputs for various configurations.
# Below are the input specific configurations.

- type: log

  # Change to true to enable this input configuration.
  enabled: true

  # Paths that should be crawled and fetched. Glob based paths.
  paths:
    - /home/dashboards/ingest_data/*.json
    #- c:\programdata\elasticsearch\logs\*

Logstash conf file follows:

input {
   beats {
      port => "5044"
      include_codec_tag => false
   }
}
  
filter {
	 json {
	    source => "message"
	    enable_metric => false
	    skip_on_invalid_json => true
	    remove_field => ["message", "hostname"]
	 }
}


output 
{           
   stdout { codec => rubydebug }
   elasticsearch
   { 
      hosts =>['localhost:9200']
      document_id => "%{[@metadata][document_id]}"
      index =>"testjson"
   }
   file {
      path => '/home/dashboards/logstash_outfile.log'
   }
}

So, i was expecting to see 8 doc counts instead of just 1. Any idea what could be wrong.

Thanks in advance.

Regards

2

Please check the Filebeat log. It provides the stats about the number of records transmitted. Once you confirm that 8 records were transmitted then look into Logstash logs.

Better start with a new input file. If you add record to the existing file, then only the new records will be transmitted.

3

Hi, thanks for prompt reply.
I am trying to produce the filebeat log file but unable to creat it:

# ================================== Logging ===================================

# Sets log level. The default log level is info.
# Available log levels are: error, warning, info, debug
logging.level: debug
logging.to_files: true
logging.files:
  path: /root/logs/
  name: filebeat
  keepfiles: 7
  permissions: 0644
# At debug level, you can selectively enable logging only for some components.
# To enable all selectors use ["*"]. Examples of other selectors are "beat",
# "publisher", "service".
#logging.selectors: ["*"]

Anyhow, i am sending the output from console:

2021-06-13T11:08:17.809-0300    INFO    instance/beat.go:309    Setup Beat: filebeat; Version: 7.13.1
2021-06-13T11:08:17.810-0300    INFO    [publisher]     pipeline/module.go:113  Beat name: mvdElastick1.verifone.com
2021-06-13T11:08:17.810-0300    WARN    beater/filebeat.go:178  Filebeat is unable to load the Ingest Node pipelines for the configured modules because the Elasticsearch output is not configured/enabled. If you have already loaded the Ingest Node pipelines or are using Logstash pipelines, you can ignore this warning.
2021-06-13T11:08:17.810-0300    INFO    [monitoring]    log/log.go:117  Starting metrics logging every 30s
2021-06-13T11:08:17.810-0300    INFO    instance/beat.go:473    filebeat start running.
2021-06-13T11:08:17.813-0300    INFO    memlog/store.go:119     Loading data file of '/var/lib/filebeat/registry/filebeat' succeeded. Active transaction id=0
2021-06-13T11:08:17.813-0300    INFO    memlog/store.go:124     Finished loading transaction log file for '/var/lib/filebeat/registry/filebeat'. Active transaction id=0
2021-06-13T11:08:17.813-0300    WARN    beater/filebeat.go:381  Filebeat is unable to load the Ingest Node pipelines for the configured modules because the Elasticsearch output is not configured/enabled. If you have already loaded the Ingest Node pipelines or are using Logstash pipelines, you can ignore this warning.
2021-06-13T11:08:17.813-0300    INFO    [registrar]     registrar/registrar.go:109      States Loaded from registrar: 0
2021-06-13T11:08:17.813-0300    INFO    [crawler]       beater/crawler.go:71    Loading Inputs: 2
2021-06-13T11:08:17.813-0300    INFO    log/input.go:157        Configured paths: [/home/dashboards/ingest_data/*.json]
2021-06-13T11:08:17.813-0300    INFO    [crawler]       beater/crawler.go:141   Starting input (ID: 10382931307721444929)
2021-06-13T11:08:17.814-0300    INFO    [crawler]       beater/crawler.go:108   Loading and starting Inputs completed. Enabled inputs: 1
2021-06-13T11:08:17.814-0300    INFO    cfgfile/reload.go:164   Config reloader started
2021-06-13T11:08:17.814-0300    INFO    cfgfile/reload.go:224   Loading of config files completed.
2021-06-13T11:08:17.814-0300    INFO    log/harvester.go:302    Harvester started for file: /home/dashboards/ingest_data/cities3.json
2021-06-13T11:08:18.814-0300    INFO    [publisher_pipeline_output]     pipeline/output.go:143  Connecting to backoff(async(tcp://localhost:5044))
2021-06-13T11:08:18.814-0300    INFO    [publisher]     pipeline/retry.go:219   retryer: send unwait signal to consumer
2021-06-13T11:08:18.814-0300    INFO    [publisher]     pipeline/retry.go:223     done
2021-06-13T11:08:18.815-0300    INFO    [publisher_pipeline_output]     pipeline/output.go:151  Connection to backoff(async(tcp://localhost:5044)) established
2021-06-13T11:08:47.813-0300    INFO    [monitoring]    log/log.go:144  Non-zero metrics in the last 30s        {"monitoring": {"metrics": {"beat":{"cgroup":{"cpu":{"cfs":{"period":{"us":100000}},"id":"user.slice"},"cpuacct":{"id":"user.slice","total":{"ns":1626467099961}},"memory":{"id":"session-122.scope","mem":{"limit":{"bytes":9223372036854771712},"usage":{"bytes":251543552}}}},"cpu":{"system":{"ticks":30,"time":{"ms":32}},"total":{"ticks":140,"time":{"ms":147},"value":140},"user":{"ticks":110,"time":{"ms":115}}},"handles":{"limit":{"hard":262144,"soft":1024},"open":14},"info":{"ephemeral_id":"861f8e4f-9d78-47d6-9b1f-b5ac7bb46e24","uptime":{"ms":30040}},"memstats":{"gc_next":17458608,"memory_alloc":13547576,"memory_sys":76104704,"memory_total":45222704,"rss":125607936},"runtime":{"goroutines":30}},"filebeat":{"events":{"added":8,"done":8},"harvester":{"open_files":1,"running":1,"started":1}},"libbeat":{"config":{"module":{"running":0},"reloads":1,"scans":1},"output":{"events":{"acked":7,"active":0,"batches":1,"total":7},"read":{"bytes":6},"type":"logstash","write":{"bytes":673}},"pipeline":{"clients":1,"events":{"active":0,"filtered":1,"published":7,"retry":7,"total":8},"queue":{"acked":7,"max_events":4096}}},"registrar":{"states":{"current":1,"update":8},"writes":{"success":2,"total":2}},"system":{"cpu":{"cores":8},"load":{"1":0.49,"15":0.18,"5":0.29,"norm":{"1":0.0613,"15":0.0225,"5":0.0363}}}}}}

At logstash log i can see 7 records instead of the original 8:

{"log":{"offset":44,"file":{"path":"/home/dashboards/ingest_data/cities3.json"}},"ecs":{"version":"1.8.0"},"city":"Sydney","province":"New South Wales","input":{"type":"log"},"@timestamp":"2021-06-13T14:08:17.814Z","host":{"name":"mvdElastick1.company.com"},"@version":"1","agent":{"type":"filebeat","version":"7.13.1","ephemeral_id":"861f8e4f-9d78-47d6-9b1f-b5ac7bb46e24","name":"mvdElastick1.company.com","hostname":"mvdElastick1.company.com","id":"54d7d3c2-2b0e-4e0f-a27d-d6c79d4db85c"}}
{"agent":{"type":"filebeat","version":"7.13.1","ephemeral_id":"861f8e4f-9d78-47d6-9b1f-b5ac7bb46e24","name":"mvdElastick1.company.com","hostname":"mvdElastick1.company.com","id":"54d7d3c2-2b0e-4e0f-a27d-d6c79d4db85c"},"ecs":{"version":"1.8.0"},"city":"South Brunswick","province":"New Jersey","input":{"type":"log"},"@timestamp":"2021-06-13T14:08:17.814Z","host":{"name":"mvdElastick1.company.com"},"log":{"offset":211,"file":{"path":"/home/dashboards/ingest_data/cities3.json"}},"@version":"1"}
{"agent":{"type":"filebeat","version":"7.13.1","ephemeral_id":"861f8e4f-9d78-47d6-9b1f-b5ac7bb46e24","name":"mvdElastick1.company.com","hostname":"mvdElastick1.company.com","id":"54d7d3c2-2b0e-4e0f-a27d-d6c79d4db85c"},"ecs":{"version":"1.8.0"},"city":"Southlake","province":"Texas","input":{"type":"log"},"@timestamp":"2021-06-13T14:08:17.814Z","host":{"name":"mvdElastick1.company.com"},"@version":"1","log":{"offset":114,"file":{"path":"/home/dashboards/ingest_data/cities3.json"}}}
{"agent":{"type":"filebeat","version":"7.13.1","ephemeral_id":"861f8e4f-9d78-47d6-9b1f-b5ac7bb46e24","name":"mvdElastick1.company.com","hostname":"mvdElastick1.company.com","id":"54d7d3c2-2b0e-4e0f-a27d-d6c79d4db85c"},"ecs":{"version":"1.8.0"},"city":"Singapore","input":{"type":"log"},"@timestamp":"2021-06-13T14:08:17.814Z","host":{"name":"mvdElastick1.company.com"},"@version":"1","log":{"offset":92,"file":{"path":"/home/dashboards/ingest_data/cities3.json"}}}
{"log":{"offset":155,"file":{"path":"/home/dashboards/ingest_data/cities3.json"}},"ecs":{"version":"1.8.0"},"city":"South San Francisco","province":"California","input":{"type":"log"},"@timestamp":"2021-06-13T14:08:17.814Z","host":{"name":"mvdElastick1.company.com"},"@version":"1","agent":{"type":"filebeat","version":"7.13.1","ephemeral_id":"861f8e4f-9d78-47d6-9b1f-b5ac7bb46e24","name":"mvdElastick1.company.com","hostname":"mvdElastick1.company.com","id":"54d7d3c2-2b0e-4e0f-a27d-d6c79d4db85c"}}
{"log":{"offset":0,"file":{"path":"/home/dashboards/ingest_data/cities3.json"}},"ecs":{"version":"1.8.0"},"city":"Seattle","province":"Washington","input":{"type":"log"},"@timestamp":"2021-06-13T14:08:17.814Z","host":{"name":"mvdElastick1.company.com"},"@version":"1","agent":{"type":"filebeat","version":"7.13.1","ephemeral_id":"861f8e4f-9d78-47d6-9b1f-b5ac7bb46e24","name":"mvdElastick1.company.com","hostname":"mvdElastick1.company.com","id":"54d7d3c2-2b0e-4e0f-a27d-d6c79d4db85c"}}
{"agent":{"type":"filebeat","version":"7.13.1","ephemeral_id":"861f8e4f-9d78-47d6-9b1f-b5ac7bb46e24","name":"mvdElastick1.company.com","hostname":"mvdElastick1.company.com","id":"54d7d3c2-2b0e-4e0f-a27d-d6c79d4db85c"},"ecs":{"version":"1.8.0"},"city":"Stretford","province":"Manchester","input":{"type":"log"},"@timestamp":"2021-06-13T14:08:17.814Z","host":{"name":"mvdElastick1.company.com"},"log":{"offset":263,"file":{"path":"/home/dashboards/ingest_data/cities3.json"}},"@version":"1"}

Still, i can see only one document at Kibana:

kibana idx management
kibana idx management1536×586 91.5 KB

Appreciate any help.

Regards

4

If you rerun filebeat on a file it has already read it will not load again it keeps track of what it has read.

Delete the filebeat registry files in the filebeat data directory or the whole data directory and try again.

5

Hi, i always delete the registry folder as i undertsand is the one used by filebeat to keep track of processed files:

[root@mvdElastick1 filebeat]# pwd
/var/lib/filebeat
[root@mvdElastick1 filebeat]# ls -l
total 0
[root@mvdElastick1 filebeat]#

Log and Data folder are empty before rerun:

[root@mvdElastick1 ~]# cd /var/log/filebeat
[root@mvdElastick1 filebeat]# ls -l
total 0
[root@mvdElastick1 filebeat]#

After running filebeat and logstash still no log at all:

[root@mvdElastick1 filebeat]# ls -l /root/logs
total 0
[root@mvdElastick1 filebeat]#
[root@mvdElastick1 filebeat]#
[root@mvdElastick1 filebeat]#
[root@mvdElastick1 filebeat]# pwd
/var/log/filebeat
[root@mvdElastick1 filebeat]# ls -l
total 0
[root@mvdElastick1 filebeat]#

I have set log conf option at filebeat.yml as follows:

# ================================== Logging ===================================

# Sets log level. The default log level is info.
# Available log levels are: error, warning, info, debug
logging.level: debug
logging.to_files: true
logging.files:
  path: /root/logs/
  name: filebeat
  keepfiles: 7
  permissions: 0644

Anyhow, i believe that some records are passed to logstash (at leas i can count 7 records):

[INFO ] 2021-06-13 15:31:35.416 [[main]-pipeline-manager] beats - Starting input listener {:address=>"0.0.0.0:5044"}
[INFO ] 2021-06-13 15:31:35.430 [[main]-pipeline-manager] javapipeline - Pipeline started {"pipeline.id"=>"main"}
[INFO ] 2021-06-13 15:31:35.498 [Agent thread] agent - Pipelines running {:count=>1, :running_pipelines=>[:main], :non_running_pipelines=>[]}
[INFO ] 2021-06-13 15:31:35.514 [[main]<beats] Server - Starting server on port: 5044
{
    "@timestamp" => 2021-06-13T18:31:49.807Z,
           "ecs" => {
        "version" => "1.8.0"
    },
          "city" => "Stretford",
      "province" => "Manchester",
         "input" => {
        "type" => "log"
    },
          "host" => {
        "name" => "mvdElastick1.company.com"
    },
         "agent" => {
                "type" => "filebeat",
             "version" => "7.13.1",
        "ephemeral_id" => "7ddc1ae5-2443-4a93-9e60-6028cd2b0d50",
                "name" => "mvdElastick1.company.com",
                  "id" => "10e326a0-8d02-4339-b9cd-34fa64d7b7f9",
            "hostname" => "mvdElastick1.company.com"
    },
      "@version" => "1",
           "log" => {
        "offset" => 263,
          "file" => {
            "path" => "/home/dashboards/ingest_data/cities4.json"
        }
    }
}
{
    "@timestamp" => 2021-06-13T18:31:49.807Z,
           "ecs" => {
        "version" => "1.8.0"
    },
          "city" => "Singapore",
         "input" => {
        "type" => "log"
    },
         "agent" => {
                "type" => "filebeat",
             "version" => "7.13.1",
        "ephemeral_id" => "7ddc1ae5-2443-4a93-9e60-6028cd2b0d50",
                "name" => "mvdElastick1.company.com",
                  "id" => "10e326a0-8d02-4339-b9cd-34fa64d7b7f9",
            "hostname" => "mvdElastick1.company.com"
    },
          "host" => {
        "name" => "mvdElastick1.company.com"
    },
      "@version" => "1",
           "log" => {
        "offset" => 92,
          "file" => {
            "path" => "/home/dashboards/ingest_data/cities4.json"
        }
    }
}
{
    "@timestamp" => 2021-06-13T18:31:49.807Z,
           "ecs" => {
        "version" => "1.8.0"
    },
          "city" => "Sydney",
      "province" => "New South Wales",
         "input" => {
        "type" => "log"
    },
          "host" => {
        "name" => "mvdElastick1.company.com"
    },
         "agent" => {
                "type" => "filebeat",
             "version" => "7.13.1",
        "ephemeral_id" => "7ddc1ae5-2443-4a93-9e60-6028cd2b0d50",
                "name" => "mvdElastick1.company.com",
                  "id" => "10e326a0-8d02-4339-b9cd-34fa64d7b7f9",
            "hostname" => "mvdElastick1.company.com"
    },
      "@version" => "1",
           "log" => {
          "file" => {
            "path" => "/home/dashboards/ingest_data/cities4.json"
        },
        "offset" => 44
    }
}
{
    "@timestamp" => 2021-06-13T18:31:49.807Z,
           "ecs" => {
        "version" => "1.8.0"
    },
          "city" => "Seattle",
      "province" => "Washington",
         "input" => {
        "type" => "log"
    },
         "agent" => {
                "type" => "filebeat",
             "version" => "7.13.1",
        "ephemeral_id" => "7ddc1ae5-2443-4a93-9e60-6028cd2b0d50",
                "name" => "mvdElastick1.company.com",
                  "id" => "10e326a0-8d02-4339-b9cd-34fa64d7b7f9",
            "hostname" => "mvdElastick1.company.com"
    },
          "host" => {
        "name" => "mvdElastick1.company.com"
    },
      "@version" => "1",
           "log" => {
        "offset" => 0,
          "file" => {
            "path" => "/home/dashboards/ingest_data/cities4.json"
        }
    }
}
{
    "@timestamp" => 2021-06-13T18:31:49.807Z,
           "ecs" => {
        "version" => "1.8.0"
    },
          "city" => "Southlake",
      "province" => "Texas",
         "input" => {
        "type" => "log"
    },
          "host" => {
        "name" => "mvdElastick1.company.com"
    },
         "agent" => {
                "type" => "filebeat",
             "version" => "7.13.1",
        "ephemeral_id" => "7ddc1ae5-2443-4a93-9e60-6028cd2b0d50",
                "name" => "mvdElastick1.company.com",
                  "id" => "10e326a0-8d02-4339-b9cd-34fa64d7b7f9",
            "hostname" => "mvdElastick1.company.com"
    },
      "@version" => "1",
           "log" => {
        "offset" => 114,
          "file" => {
            "path" => "/home/dashboards/ingest_data/cities4.json"
        }
    }
}
{
    "@timestamp" => 2021-06-13T18:31:49.807Z,
           "ecs" => {
        "version" => "1.8.0"
    },
          "city" => "South San Francisco",
      "province" => "California",
          "host" => {
        "name" => "mvdElastick1.company.com"
    },
         "agent" => {
                "type" => "filebeat",
             "version" => "7.13.1",
        "ephemeral_id" => "7ddc1ae5-2443-4a93-9e60-6028cd2b0d50",
                "name" => "mvdElastick1.company.com",
                  "id" => "10e326a0-8d02-4339-b9cd-34fa64d7b7f9",
            "hostname" => "mvdElastick1.company.com"
    },
         "input" => {
        "type" => "log"
    },
      "@version" => "1",
           "log" => {
        "offset" => 155,
          "file" => {
            "path" => "/home/dashboards/ingest_data/cities4.json"
        }
    }
}
{
    "@timestamp" => 2021-06-13T18:31:49.807Z,
           "ecs" => {
        "version" => "1.8.0"
    },
          "city" => "South Brunswick",
      "province" => "New Jersey",
         "input" => {
        "type" => "log"
    },
          "host" => {
        "name" => "mvdElastick1.company.com"
    },
         "agent" => {
                "type" => "filebeat",
             "version" => "7.13.1",
        "ephemeral_id" => "7ddc1ae5-2443-4a93-9e60-6028cd2b0d50",
                "name" => "mvdElastick1.company.com",
                  "id" => "10e326a0-8d02-4339-b9cd-34fa64d7b7f9",
            "hostname" => "mvdElastick1.company.com"
    },
      "@version" => "1",
           "log" => {
          "file" => {
            "path" => "/home/dashboards/ingest_data/cities4.json"
        },
        "offset" => 211
    }
}
[INFO ] 2021-06-13 15:31:51.431 [[main]>worker0] file - Opening file {:path=>"/home/dashboards/logstash_outfile.log"}
[INFO ] 2021-06-13 15:32:10.471 [[main]>worker6] file - Closing file /home/dashboards/logstash_outfile.log

So, logstash i saying /showing 7 out of 8, but at kibana i can see only 1 document:

This is the json file i have used:

json file
json file922×255 59.2 KB

I was checking some similar cases, and someone said that json records should have an index json record before each of my json records. Not sure if this could be the case.

I assume that my original assumptions, are ok, that after processing this json file i should see 8 documents inside the index, right ?

Thanks in advance.

Regards

6

@rafaell2 Can you post the actual text of the json please not a screen shot.
You are probably not getting the last line as it would require a newline, otherwise filebeat does not know the line is complete. I suspect the last line is the line that is missing.

Also try taking out this line In your logstash

      document_id => "%{[@metadata][document_id]}"
7

My Completes Solution

Input notice the last newline

{ "name" : "value1", "name2" : "value21"}
{ "name" : "value2", "name2" : "value22"}
{ "name" : "value3", "name2" : "value23"}
{ "name" : "value4", "name2" : "value24"}
{ "name" : "value5", "name2" : "value25"}
{ "name" : "value6", "name2" : "value26"}

Complete minimal filebeat-minimal.yml

filebeat.inputs:
- type: log
  enabled: true
  paths:
    - /Users/sbrown/workspace/elastic-install/7.13.0/filebeat-7.13.0-darwin-x86_64/test.ndjson
- type: filestream
  enabled: false
  paths:
    - /var/log/*.log
filebeat.config.modules:
  path: ${path.config}/modules.d/*.yml
  reload.enabled: false
setup.template.settings:
  index.number_of_shards: 1
output.logstash:
  hosts: ["localhost:5044"]
processors:
  - add_host_metadata:
      when.not.contains.tags: forwarded
  - add_cloud_metadata: ~
  - add_docker_metadata: ~
  - add_kubernetes_metadata: ~

Complete logstash.conf NOTE where the remove is outside the JSON block.

# simple beats (ndjson) -> logsash -> elasticsearch
input {
   beats {
      port => "5044"
      include_codec_tag => false
   }
}
  
filter {
   json {
     source => "message"
   }
   mutate 
   {
      remove_field => ["message", "host", "agent"]
   }
}

output 
{           
   stdout { codec => rubydebug }
   elasticsearch
   { 
      hosts =>['localhost:9200']
      index =>"my-discuss-index"
   }
}

Output looks good!

GET my-discuss-index/_search

{
  "took" : 3,
  "timed_out" : false,
  "_shards" : {
    "total" : 1,
    "successful" : 1,
    "skipped" : 0,
    "failed" : 0
  },
  "hits" : {
    "total" : {
      "value" : 6,   <!------- All Records
      "relation" : "eq"
    },
    "max_score" : 1.0,
    "hits" : [
      {
        "_index" : "my-discuss-index",
        "_type" : "_doc",
        "_id" : "_UziBnoBhPeJjpPZ_R56",
        "_score" : 1.0,
        "_source" : {
          "container" : {
            "id" : "7.13.0"
          },
          "name2" : "value21",
          "@version" : "1",
          "@timestamp" : "2021-06-13T19:39:32.488Z",
          "ecs" : {
            "version" : "1.8.0"
          },
          "input" : {
            "type" : "log"
          },
          "log" : {
            "offset" : 0,
            "file" : {
              "path" : "/Users/sbrown/workspace/elastic-install/7.13.0/filebeat-7.13.0-darwin-x86_64/test.ndjson"
            }
          },
          "name" : "value1"
        }
      },
      {
        "_index" : "my-discuss-index",
        "_type" : "_doc",
        "_id" : "AEziBnoBhPeJjpPZ_R98",
        "_score" : 1.0,
        "_source" : {
          "container" : {
            "id" : "7.13.0"
          },
          "name2" : "value23",
          "@version" : "1",
          "@timestamp" : "2021-06-13T19:39:35.438Z",
          "ecs" : {
            "version" : "1.8.0"
          },
          "input" : {
            "type" : "log"
          },
          "log" : {
            "offset" : 84,
            "file" : {
              "path" : "/Users/sbrown/workspace/elastic-install/7.13.0/filebeat-7.13.0-darwin-x86_64/test.ndjson"
            }
          },
          "name" : "value3"
        }
      },
      {
        "_index" : "my-discuss-index",
        "_type" : "_doc",
        "_id" : "_0ziBnoBhPeJjpPZ_R58",
        "_score" : 1.0,
        "_source" : {
          "container" : {
            "id" : "7.13.0"
          },
          "name2" : "value25",
          "@version" : "1",
          "@timestamp" : "2021-06-13T19:39:35.438Z",
          "ecs" : {
            "version" : "1.8.0"
          },
          "input" : {
            "type" : "log"
          },
          "log" : {
            "file" : {
              "path" : "/Users/sbrown/workspace/elastic-install/7.13.0/filebeat-7.13.0-darwin-x86_64/test.ndjson"
            },
            "offset" : 168
          },
          "name" : "value5"
        }
      },
      {
        "_index" : "my-discuss-index",
        "_type" : "_doc",
        "_id" : "_kziBnoBhPeJjpPZ_R56",
        "_score" : 1.0,
        "_source" : {
          "container" : {
            "id" : "7.13.0"
          },
          "name2" : "value22",
          "@version" : "1",
          "@timestamp" : "2021-06-13T19:39:35.438Z",
          "ecs" : {
            "version" : "1.8.0"
          },
          "input" : {
            "type" : "log"
          },
          "log" : {
            "offset" : 42,
            "file" : {
              "path" : "/Users/sbrown/workspace/elastic-install/7.13.0/filebeat-7.13.0-darwin-x86_64/test.ndjson"
            }
          },
          "name" : "value2"
        }
      },
      {
        "_index" : "my-discuss-index",
        "_type" : "_doc",
        "_id" : "-0ziBnoBhPeJjpPZ_R56",
        "_score" : 1.0,
        "_source" : {
          "container" : {
            "id" : "7.13.0"
          },
          "name2" : "value24",
          "@version" : "1",
          "@timestamp" : "2021-06-13T19:39:35.438Z",
          "ecs" : {
            "version" : "1.8.0"
          },
          "input" : {
            "type" : "log"
          },
          "log" : {
            "offset" : 126,
            "file" : {
              "path" : "/Users/sbrown/workspace/elastic-install/7.13.0/filebeat-7.13.0-darwin-x86_64/test.ndjson"
            }
          },
          "name" : "value4"
        }
      },
      {
        "_index" : "my-discuss-index",
        "_type" : "_doc",
        "_id" : "_EziBnoBhPeJjpPZ_R56",
        "_score" : 1.0,
        "_source" : {
          "container" : {
            "id" : "7.13.0"
          },
          "name2" : "value26",
          "@version" : "1",
          "@timestamp" : "2021-06-13T19:39:35.438Z",
          "ecs" : {
            "version" : "1.8.0"
          },
          "input" : {
            "type" : "log"
          },
          "log" : {
            "offset" : 210,
            "file" : {
              "path" : "/Users/sbrown/workspace/elastic-install/7.13.0/filebeat-7.13.0-darwin-x86_64/test.ndjson"
            }
          },
          "name" : "value6"
        }
      }
    ]
  }
}
8

Hi Stephen !!! hate to say it BUT IT WORKED !!!

This line was causing the whole mess !!! Not sure from where i got this example, but i will need to go back to basis and figure out what is this for !!! But if i leave it i get just one document, but without it it works just fine !!!

document_id => "%{[@metadata][document_id]}"

Regarding the missing line, indeed was because the lack of a new line at the end of the file !!!

So, thanks a million to all of you who helped me out with this issue !!!

Stay safe !!! Take care & regards

Rafael

1 Like
9

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.