# Number of docs count does not match number of json file records

**URL:** <https://discuss.elastic.co/t/number-of-docs-count-does-not-match-number-of-json-file-records/275739>\
**Category:** Beats\
**Tags:** filebeat\
**Created:** [June 12, 2021, 9:28pm UTC](https://discuss.elastic.co/t/number-of-docs-count-does-not-match-number-of-json-file-records/275739 "2021-06-12T21:28:38Z")\
**Posts on this page:** 9\
**Page:** 1

<div class="post-metadata">

**Author:** ![rafaell2](https://avatars.discourse-cdn.com/v4/letter/r/94ad74/32.png) [@rafaell2](https://discuss.elastic.co/u/rafaell2)\
**Post date:** [June 12, 2021, 9:28pm UTC](https://discuss.elastic.co/t/number-of-docs-count-does-not-match-number-of-json-file-records/275739/1 "2021-06-12T21:28:38Z")

</div>

Hi. I am trying to load a json file with 8 records from filebeats to logstash (latest verion 7.13). It looks like i have no errors but when i check number of records at Kibana Index Management i see docs counts = 1 (also i checked at Discover and there is indeed just one document/event).

![json file](https://us1.discourse-cdn.com/elastic/original/3X/b/6/b6205982f48a87048599a0c63985dd9e73676c48.jpeg)

Filebeat yml file is pretty straight forward:

```auto
# ============================== Filebeat inputs ===============================

filebeat.inputs:

# Each - is an input. Most options can be set at the input level, so
# you can use different inputs for various configurations.
# Below are the input specific configurations.

- type: log

  # Change to true to enable this input configuration.
  enabled: true

  # Paths that should be crawled and fetched. Glob based paths.
  paths:
    - /home/dashboards/ingest_data/*.json
    #- c:\programdata\elasticsearch\logs\*

```

Logstash conf file follows:

```auto
input {
   beats {
      port => "5044"
      include_codec_tag => false
   }
}
  
filter {
	 json {
	    source => "message"
	    enable_metric => false
	    skip_on_invalid_json => true
	    remove_field => ["message", "hostname"]
	 }
}

output 
{           
   stdout { codec => rubydebug }
   elasticsearch
   { 
      hosts =>['localhost:9200']
      document_id => "%{[@metadata][document_id]}"
      index =>"testjson"
   }
   file {
      path => '/home/dashboards/logstash_outfile.log'
   }
}

```

So, i was expecting to see 8 doc counts instead of just 1. Any idea what could be wrong.

Thanks in advance.

Regards

---

<div class="post-metadata">

**Author:** ![kumarabhi](https://avatars.discourse-cdn.com/v4/letter/k/6a8cbe/32.png) [@kumarabhi](https://discuss.elastic.co/u/kumarabhi)\
**Post date:** [June 13, 2021, 6:12am UTC](https://discuss.elastic.co/t/number-of-docs-count-does-not-match-number-of-json-file-records/275739/2 "2021-06-13T06:12:10Z")

</div>

Please check the Filebeat log. It provides the stats about the number of records transmitted. Once you confirm that 8 records were transmitted then look into Logstash logs.

Better start with a new input file. If you add record to the existing file, then only the new records will be transmitted.

---

<div class="post-metadata">

**Author:** ![rafaell2](https://avatars.discourse-cdn.com/v4/letter/r/94ad74/32.png) [@rafaell2](https://discuss.elastic.co/u/rafaell2)\
**Post date:** [June 13, 2021, 2:11pm UTC](https://discuss.elastic.co/t/number-of-docs-count-does-not-match-number-of-json-file-records/275739/3 "2021-06-13T14:11:46Z")

</div>

Hi, thanks for prompt reply.  
I am trying to produce the filebeat log file but unable to creat it:

```auto
# ================================== Logging ===================================

# Sets log level. The default log level is info.
# Available log levels are: error, warning, info, debug
logging.level: debug
logging.to_files: true
logging.files:
  path: /root/logs/
  name: filebeat
  keepfiles: 7
  permissions: 0644
# At debug level, you can selectively enable logging only for some components.
# To enable all selectors use ["*"]. Examples of other selectors are "beat",
# "publisher", "service".
#logging.selectors: ["*"]

```

Anyhow, i am sending the output from console:

```auto
2021-06-13T11:08:17.809-0300 INFO instance/beat.go:309 Setup Beat: filebeat; Version: 7.13.1
2021-06-13T11:08:17.810-0300 INFO [publisher] pipeline/module.go:113 Beat name: mvdElastick1.verifone.com
2021-06-13T11:08:17.810-0300 WARN beater/filebeat.go:178 Filebeat is unable to load the Ingest Node pipelines for the configured modules because the Elasticsearch output is not configured/enabled. If you have already loaded the Ingest Node pipelines or are using Logstash pipelines, you can ignore this warning.
2021-06-13T11:08:17.810-0300 INFO [monitoring] log/log.go:117 Starting metrics logging every 30s
2021-06-13T11:08:17.810-0300 INFO instance/beat.go:473 filebeat start running.
2021-06-13T11:08:17.813-0300 INFO memlog/store.go:119 Loading data file of '/var/lib/filebeat/registry/filebeat' succeeded. Active transaction id=0
2021-06-13T11:08:17.813-0300 INFO memlog/store.go:124 Finished loading transaction log file for '/var/lib/filebeat/registry/filebeat'. Active transaction id=0
2021-06-13T11:08:17.813-0300 WARN beater/filebeat.go:381 Filebeat is unable to load the Ingest Node pipelines for the configured modules because the Elasticsearch output is not configured/enabled. If you have already loaded the Ingest Node pipelines or are using Logstash pipelines, you can ignore this warning.
2021-06-13T11:08:17.813-0300 INFO [registrar] registrar/registrar.go:109 States Loaded from registrar: 0
2021-06-13T11:08:17.813-0300 INFO [crawler] beater/crawler.go:71 Loading Inputs: 2
2021-06-13T11:08:17.813-0300 INFO log/input.go:157 Configured paths: [/home/dashboards/ingest_data/*.json]
2021-06-13T11:08:17.813-0300 INFO [crawler] beater/crawler.go:141 Starting input (ID: 10382931307721444929)
2021-06-13T11:08:17.814-0300 INFO [crawler] beater/crawler.go:108 Loading and starting Inputs completed. Enabled inputs: 1
2021-06-13T11:08:17.814-0300 INFO cfgfile/reload.go:164 Config reloader started
2021-06-13T11:08:17.814-0300 INFO cfgfile/reload.go:224 Loading of config files completed.
2021-06-13T11:08:17.814-0300 INFO log/harvester.go:302 Harvester started for file: /home/dashboards/ingest_data/cities3.json
2021-06-13T11:08:18.814-0300 INFO [publisher_pipeline_output] pipeline/output.go:143 Connecting to backoff(async(tcp://localhost:5044))
2021-06-13T11:08:18.814-0300 INFO [publisher] pipeline/retry.go:219 retryer: send unwait signal to consumer
2021-06-13T11:08:18.814-0300 INFO [publisher] pipeline/retry.go:223 done
2021-06-13T11:08:18.815-0300 INFO [publisher_pipeline_output] pipeline/output.go:151 Connection to backoff(async(tcp://localhost:5044)) established
2021-06-13T11:08:47.813-0300 INFO [monitoring] log/log.go:144 Non-zero metrics in the last 30s {"monitoring": {"metrics": {"beat":{"cgroup":{"cpu":{"cfs":{"period":{"us":100000}},"id":"user.slice"},"cpuacct":{"id":"user.slice","total":{"ns":1626467099961}},"memory":{"id":"session-122.scope","mem":{"limit":{"bytes":9223372036854771712},"usage":{"bytes":251543552}}}},"cpu":{"system":{"ticks":30,"time":{"ms":32}},"total":{"ticks":140,"time":{"ms":147},"value":140},"user":{"ticks":110,"time":{"ms":115}}},"handles":{"limit":{"hard":262144,"soft":1024},"open":14},"info":{"ephemeral_id":"861f8e4f-9d78-47d6-9b1f-b5ac7bb46e24","uptime":{"ms":30040}},"memstats":{"gc_next":17458608,"memory_alloc":13547576,"memory_sys":76104704,"memory_total":45222704,"rss":125607936},"runtime":{"goroutines":30}},"filebeat":{"events":{"added":8,"done":8},"harvester":{"open_files":1,"running":1,"started":1}},"libbeat":{"config":{"module":{"running":0},"reloads":1,"scans":1},"output":{"events":{"acked":7,"active":0,"batches":1,"total":7},"read":{"bytes":6},"type":"logstash","write":{"bytes":673}},"pipeline":{"clients":1,"events":{"active":0,"filtered":1,"published":7,"retry":7,"total":8},"queue":{"acked":7,"max_events":4096}}},"registrar":{"states":{"current":1,"update":8},"writes":{"success":2,"total":2}},"system":{"cpu":{"cores":8},"load":{"1":0.49,"15":0.18,"5":0.29,"norm":{"1":0.0613,"15":0.0225,"5":0.0363}}}}}}

```

At logstash log i can see 7 records instead of the original 8:

```auto
{"log":{"offset":44,"file":{"path":"/home/dashboards/ingest_data/cities3.json"}},"ecs":{"version":"1.8.0"},"city":"Sydney","province":"New South Wales","input":{"type":"log"},"@timestamp":"2021-06-13T14:08:17.814Z","host":{"name":"mvdElastick1.company.com"},"@version":"1","agent":{"type":"filebeat","version":"7.13.1","ephemeral_id":"861f8e4f-9d78-47d6-9b1f-b5ac7bb46e24","name":"mvdElastick1.company.com","hostname":"mvdElastick1.company.com","id":"54d7d3c2-2b0e-4e0f-a27d-d6c79d4db85c"}}
{"agent":{"type":"filebeat","version":"7.13.1","ephemeral_id":"861f8e4f-9d78-47d6-9b1f-b5ac7bb46e24","name":"mvdElastick1.company.com","hostname":"mvdElastick1.company.com","id":"54d7d3c2-2b0e-4e0f-a27d-d6c79d4db85c"},"ecs":{"version":"1.8.0"},"city":"South Brunswick","province":"New Jersey","input":{"type":"log"},"@timestamp":"2021-06-13T14:08:17.814Z","host":{"name":"mvdElastick1.company.com"},"log":{"offset":211,"file":{"path":"/home/dashboards/ingest_data/cities3.json"}},"@version":"1"}
{"agent":{"type":"filebeat","version":"7.13.1","ephemeral_id":"861f8e4f-9d78-47d6-9b1f-b5ac7bb46e24","name":"mvdElastick1.company.com","hostname":"mvdElastick1.company.com","id":"54d7d3c2-2b0e-4e0f-a27d-d6c79d4db85c"},"ecs":{"version":"1.8.0"},"city":"Southlake","province":"Texas","input":{"type":"log"},"@timestamp":"2021-06-13T14:08:17.814Z","host":{"name":"mvdElastick1.company.com"},"@version":"1","log":{"offset":114,"file":{"path":"/home/dashboards/ingest_data/cities3.json"}}}
{"agent":{"type":"filebeat","version":"7.13.1","ephemeral_id":"861f8e4f-9d78-47d6-9b1f-b5ac7bb46e24","name":"mvdElastick1.company.com","hostname":"mvdElastick1.company.com","id":"54d7d3c2-2b0e-4e0f-a27d-d6c79d4db85c"},"ecs":{"version":"1.8.0"},"city":"Singapore","input":{"type":"log"},"@timestamp":"2021-06-13T14:08:17.814Z","host":{"name":"mvdElastick1.company.com"},"@version":"1","log":{"offset":92,"file":{"path":"/home/dashboards/ingest_data/cities3.json"}}}
{"log":{"offset":155,"file":{"path":"/home/dashboards/ingest_data/cities3.json"}},"ecs":{"version":"1.8.0"},"city":"South San Francisco","province":"California","input":{"type":"log"},"@timestamp":"2021-06-13T14:08:17.814Z","host":{"name":"mvdElastick1.company.com"},"@version":"1","agent":{"type":"filebeat","version":"7.13.1","ephemeral_id":"861f8e4f-9d78-47d6-9b1f-b5ac7bb46e24","name":"mvdElastick1.company.com","hostname":"mvdElastick1.company.com","id":"54d7d3c2-2b0e-4e0f-a27d-d6c79d4db85c"}}
{"log":{"offset":0,"file":{"path":"/home/dashboards/ingest_data/cities3.json"}},"ecs":{"version":"1.8.0"},"city":"Seattle","province":"Washington","input":{"type":"log"},"@timestamp":"2021-06-13T14:08:17.814Z","host":{"name":"mvdElastick1.company.com"},"@version":"1","agent":{"type":"filebeat","version":"7.13.1","ephemeral_id":"861f8e4f-9d78-47d6-9b1f-b5ac7bb46e24","name":"mvdElastick1.company.com","hostname":"mvdElastick1.company.com","id":"54d7d3c2-2b0e-4e0f-a27d-d6c79d4db85c"}}
{"agent":{"type":"filebeat","version":"7.13.1","ephemeral_id":"861f8e4f-9d78-47d6-9b1f-b5ac7bb46e24","name":"mvdElastick1.company.com","hostname":"mvdElastick1.company.com","id":"54d7d3c2-2b0e-4e0f-a27d-d6c79d4db85c"},"ecs":{"version":"1.8.0"},"city":"Stretford","province":"Manchester","input":{"type":"log"},"@timestamp":"2021-06-13T14:08:17.814Z","host":{"name":"mvdElastick1.company.com"},"log":{"offset":263,"file":{"path":"/home/dashboards/ingest_data/cities3.json"}},"@version":"1"}

```

Still, i can see only one document at Kibana:

 ![kibana idx management](https://us1.discourse-cdn.com/elastic/original/3X/8/1/813f0d273a3f0a0d9554df32b210b9aa784ae79c.jpeg)

Appreciate any help.

Regards

---

<div class="post-metadata">

**Author:** ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)\
**Post date:** [June 13, 2021, 2:59pm UTC](https://discuss.elastic.co/t/number-of-docs-count-does-not-match-number-of-json-file-records/275739/4 "2021-06-13T14:59:12Z")

</div>

If you rerun filebeat on a file it has already read it will not load again it keeps track of what it has read.

Delete the filebeat registry files in the filebeat `data` directory or the whole `data` directory and try again.

---

<div class="post-metadata">

**Author:** ![rafaell2](https://avatars.discourse-cdn.com/v4/letter/r/94ad74/32.png) [@rafaell2](https://discuss.elastic.co/u/rafaell2)\
**Post date:** [June 13, 2021, 6:48pm UTC](https://discuss.elastic.co/t/number-of-docs-count-does-not-match-number-of-json-file-records/275739/5 "2021-06-13T18:48:48Z")

</div>

Hi, i always delete the registry folder as i undertsand is the one used by filebeat to keep track of processed files:

```auto
[root@mvdElastick1 filebeat]# pwd
/var/lib/filebeat
[root@mvdElastick1 filebeat]# ls -l
total 0
[root@mvdElastick1 filebeat]#

```

Log and Data folder are empty before rerun:

```auto
[root@mvdElastick1 ~]# cd /var/log/filebeat
[root@mvdElastick1 filebeat]# ls -l
total 0
[root@mvdElastick1 filebeat]#

```

After running filebeat and logstash still no log at all:

```auto
[root@mvdElastick1 filebeat]# ls -l /root/logs
total 0
[root@mvdElastick1 filebeat]#
[root@mvdElastick1 filebeat]#
[root@mvdElastick1 filebeat]#
[root@mvdElastick1 filebeat]# pwd
/var/log/filebeat
[root@mvdElastick1 filebeat]# ls -l
total 0
[root@mvdElastick1 filebeat]#

```

I have set log conf option at filebeat.yml as follows:

```auto
# ================================== Logging ===================================

# Sets log level. The default log level is info.
# Available log levels are: error, warning, info, debug
logging.level: debug
logging.to_files: true
logging.files:
  path: /root/logs/
  name: filebeat
  keepfiles: 7
  permissions: 0644

```

Anyhow, i believe that some records are passed to logstash (at leas i can count 7 records):

```auto
[INFO] 2021-06-13 15:31:35.416 [[main]-pipeline-manager] beats - Starting input listener {:address=>"0.0.0.0:5044"}
[INFO] 2021-06-13 15:31:35.430 [[main]-pipeline-manager] javapipeline - Pipeline started {"pipeline.id"=>"main"}
[INFO] 2021-06-13 15:31:35.498 [Agent thread] agent - Pipelines running {:count=>1, :running_pipelines=>[:main], :non_running_pipelines=>[]}
[INFO] 2021-06-13 15:31:35.514 [[main]<beats] Server - Starting server on port: 5044
{
    "@timestamp" => 2021-06-13T18:31:49.807Z,
           "ecs" => {
        "version" => "1.8.0"
    },
          "city" => "Stretford",
      "province" => "Manchester",
         "input" => {
        "type" => "log"
    },
          "host" => {
        "name" => "mvdElastick1.company.com"
    },
         "agent" => {
                "type" => "filebeat",
             "version" => "7.13.1",
        "ephemeral_id" => "7ddc1ae5-2443-4a93-9e60-6028cd2b0d50",
                "name" => "mvdElastick1.company.com",
                  "id" => "10e326a0-8d02-4339-b9cd-34fa64d7b7f9",
            "hostname" => "mvdElastick1.company.com"
    },
      "@version" => "1",
           "log" => {
        "offset" => 263,
          "file" => {
            "path" => "/home/dashboards/ingest_data/cities4.json"
        }
    }
}
{
    "@timestamp" => 2021-06-13T18:31:49.807Z,
           "ecs" => {
        "version" => "1.8.0"
    },
          "city" => "Singapore",
         "input" => {
        "type" => "log"
    },
         "agent" => {
                "type" => "filebeat",
             "version" => "7.13.1",
        "ephemeral_id" => "7ddc1ae5-2443-4a93-9e60-6028cd2b0d50",
                "name" => "mvdElastick1.company.com",
                  "id" => "10e326a0-8d02-4339-b9cd-34fa64d7b7f9",
            "hostname" => "mvdElastick1.company.com"
    },
          "host" => {
        "name" => "mvdElastick1.company.com"
    },
      "@version" => "1",
           "log" => {
        "offset" => 92,
          "file" => {
            "path" => "/home/dashboards/ingest_data/cities4.json"
        }
    }
}
{
    "@timestamp" => 2021-06-13T18:31:49.807Z,
           "ecs" => {
        "version" => "1.8.0"
    },
          "city" => "Sydney",
      "province" => "New South Wales",
         "input" => {
        "type" => "log"
    },
          "host" => {
        "name" => "mvdElastick1.company.com"
    },
         "agent" => {
                "type" => "filebeat",
             "version" => "7.13.1",
        "ephemeral_id" => "7ddc1ae5-2443-4a93-9e60-6028cd2b0d50",
                "name" => "mvdElastick1.company.com",
                  "id" => "10e326a0-8d02-4339-b9cd-34fa64d7b7f9",
            "hostname" => "mvdElastick1.company.com"
    },
      "@version" => "1",
           "log" => {
          "file" => {
            "path" => "/home/dashboards/ingest_data/cities4.json"
        },
        "offset" => 44
    }
}
{
    "@timestamp" => 2021-06-13T18:31:49.807Z,
           "ecs" => {
        "version" => "1.8.0"
    },
          "city" => "Seattle",
      "province" => "Washington",
         "input" => {
        "type" => "log"
    },
         "agent" => {
                "type" => "filebeat",
             "version" => "7.13.1",
        "ephemeral_id" => "7ddc1ae5-2443-4a93-9e60-6028cd2b0d50",
                "name" => "mvdElastick1.company.com",
                  "id" => "10e326a0-8d02-4339-b9cd-34fa64d7b7f9",
            "hostname" => "mvdElastick1.company.com"
    },
          "host" => {
        "name" => "mvdElastick1.company.com"
    },
      "@version" => "1",
           "log" => {
        "offset" => 0,
          "file" => {
            "path" => "/home/dashboards/ingest_data/cities4.json"
        }
    }
}
{
    "@timestamp" => 2021-06-13T18:31:49.807Z,
           "ecs" => {
        "version" => "1.8.0"
    },
          "city" => "Southlake",
      "province" => "Texas",
         "input" => {
        "type" => "log"
    },
          "host" => {
        "name" => "mvdElastick1.company.com"
    },
         "agent" => {
                "type" => "filebeat",
             "version" => "7.13.1",
        "ephemeral_id" => "7ddc1ae5-2443-4a93-9e60-6028cd2b0d50",
                "name" => "mvdElastick1.company.com",
                  "id" => "10e326a0-8d02-4339-b9cd-34fa64d7b7f9",
            "hostname" => "mvdElastick1.company.com"
    },
      "@version" => "1",
           "log" => {
        "offset" => 114,
          "file" => {
            "path" => "/home/dashboards/ingest_data/cities4.json"
        }
    }
}
{
    "@timestamp" => 2021-06-13T18:31:49.807Z,
           "ecs" => {
        "version" => "1.8.0"
    },
          "city" => "South San Francisco",
      "province" => "California",
          "host" => {
        "name" => "mvdElastick1.company.com"
    },
         "agent" => {
                "type" => "filebeat",
             "version" => "7.13.1",
        "ephemeral_id" => "7ddc1ae5-2443-4a93-9e60-6028cd2b0d50",
                "name" => "mvdElastick1.company.com",
                  "id" => "10e326a0-8d02-4339-b9cd-34fa64d7b7f9",
            "hostname" => "mvdElastick1.company.com"
    },
         "input" => {
        "type" => "log"
    },
      "@version" => "1",
           "log" => {
        "offset" => 155,
          "file" => {
            "path" => "/home/dashboards/ingest_data/cities4.json"
        }
    }
}
{
    "@timestamp" => 2021-06-13T18:31:49.807Z,
           "ecs" => {
        "version" => "1.8.0"
    },
          "city" => "South Brunswick",
      "province" => "New Jersey",
         "input" => {
        "type" => "log"
    },
          "host" => {
        "name" => "mvdElastick1.company.com"
    },
         "agent" => {
                "type" => "filebeat",
             "version" => "7.13.1",
        "ephemeral_id" => "7ddc1ae5-2443-4a93-9e60-6028cd2b0d50",
                "name" => "mvdElastick1.company.com",
                  "id" => "10e326a0-8d02-4339-b9cd-34fa64d7b7f9",
            "hostname" => "mvdElastick1.company.com"
    },
      "@version" => "1",
           "log" => {
          "file" => {
            "path" => "/home/dashboards/ingest_data/cities4.json"
        },
        "offset" => 211
    }
}
[INFO] 2021-06-13 15:31:51.431 [[main]>worker0] file - Opening file {:path=>"/home/dashboards/logstash_outfile.log"}
[INFO] 2021-06-13 15:32:10.471 [[main]>worker6] file - Closing file /home/dashboards/logstash_outfile.log

```

So, logstash i saying /showing 7 out of 8, but at kibana i can see only 1 document:

This is the json file i have used:

 ![json file](https://us1.discourse-cdn.com/elastic/original/3X/f/8/f8587ea5ab131b59e116f265e7727cbfd897fed1.jpeg)

I was checking some similar cases, and someone said that json records should have an index json record before each of my json records. Not sure if this could be the case.

I assume that my original assumptions, are ok, that after processing this json file i should see 8 documents inside the index, right ?

Thanks in advance.

Regards

---

<div class="post-metadata">

**Author:** ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)\
**Post date:** [June 13, 2021, 7:08pm UTC](https://discuss.elastic.co/t/number-of-docs-count-does-not-match-number-of-json-file-records/275739/6 "2021-06-13T19:08:48Z")

</div>

@rafaell2 Can you post the actual text of the json please not a screen shot.  
You are probably not getting the last line as it would require a newline, otherwise filebeat does not know the line is complete. I suspect the last line is the line that is missing.

Also try taking out this line In your logstash

```auto
      document_id => "%{[@metadata][document_id]}"

```

---

<div class="post-metadata">

**Author:** ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)\
**Post date:** [June 13, 2021, 7:44pm UTC](https://discuss.elastic.co/t/number-of-docs-count-does-not-match-number-of-json-file-records/275739/7 "2021-06-13T19:44:49Z")

</div>

My Completes Solution

Input notice the last newline

```auto
{ "name" : "value1", "name2" : "value21"}
{ "name" : "value2", "name2" : "value22"}
{ "name" : "value3", "name2" : "value23"}
{ "name" : "value4", "name2" : "value24"}
{ "name" : "value5", "name2" : "value25"}
{ "name" : "value6", "name2" : "value26"}

```

Complete minimal filebeat-minimal.yml

```auto
filebeat.inputs:
- type: log
  enabled: true
  paths:
    - /Users/sbrown/workspace/elastic-install/7.13.0/filebeat-7.13.0-darwin-x86_64/test.ndjson
- type: filestream
  enabled: false
  paths:
    - /var/log/*.log
filebeat.config.modules:
  path: ${path.config}/modules.d/*.yml
  reload.enabled: false
setup.template.settings:
  index.number_of_shards: 1
output.logstash:
  hosts: ["localhost:5044"]
processors:
  - add_host_metadata:
      when.not.contains.tags: forwarded
  - add_cloud_metadata: ~
  - add_docker_metadata: ~
  - add_kubernetes_metadata: ~

```

Complete logstash.conf NOTE where the remove is outside the JSON block.

```auto
# simple beats (ndjson) -> logsash -> elasticsearch
input {
   beats {
      port => "5044"
      include_codec_tag => false
   }
}
  
filter {
   json {
     source => "message"
   }
   mutate 
   {
      remove_field => ["message", "host", "agent"]
   }
}

output 
{           
   stdout { codec => rubydebug }
   elasticsearch
   { 
      hosts =>['localhost:9200']
      index =>"my-discuss-index"
   }
}

```

Output looks good!

```auto
GET my-discuss-index/_search

{
  "took" : 3,
  "timed_out" : false,
  "_shards" : {
    "total" : 1,
    "successful" : 1,
    "skipped" : 0,
    "failed" : 0
  },
  "hits" : {
    "total" : {
      "value" : 6, <!------- All Records
      "relation" : "eq"
    },
    "max_score" : 1.0,
    "hits" : [
      {
        "_index" : "my-discuss-index",
        "_type" : "_doc",
        "_id" : "_UziBnoBhPeJjpPZ_R56",
        "_score" : 1.0,
        "_source" : {
          "container" : {
            "id" : "7.13.0"
          },
          "name2" : "value21",
          "@version" : "1",
          "@timestamp" : "2021-06-13T19:39:32.488Z",
          "ecs" : {
            "version" : "1.8.0"
          },
          "input" : {
            "type" : "log"
          },
          "log" : {
            "offset" : 0,
            "file" : {
              "path" : "/Users/sbrown/workspace/elastic-install/7.13.0/filebeat-7.13.0-darwin-x86_64/test.ndjson"
            }
          },
          "name" : "value1"
        }
      },
      {
        "_index" : "my-discuss-index",
        "_type" : "_doc",
        "_id" : "AEziBnoBhPeJjpPZ_R98",
        "_score" : 1.0,
        "_source" : {
          "container" : {
            "id" : "7.13.0"
          },
          "name2" : "value23",
          "@version" : "1",
          "@timestamp" : "2021-06-13T19:39:35.438Z",
          "ecs" : {
            "version" : "1.8.0"
          },
          "input" : {
            "type" : "log"
          },
          "log" : {
            "offset" : 84,
            "file" : {
              "path" : "/Users/sbrown/workspace/elastic-install/7.13.0/filebeat-7.13.0-darwin-x86_64/test.ndjson"
            }
          },
          "name" : "value3"
        }
      },
      {
        "_index" : "my-discuss-index",
        "_type" : "_doc",
        "_id" : "_0ziBnoBhPeJjpPZ_R58",
        "_score" : 1.0,
        "_source" : {
          "container" : {
            "id" : "7.13.0"
          },
          "name2" : "value25",
          "@version" : "1",
          "@timestamp" : "2021-06-13T19:39:35.438Z",
          "ecs" : {
            "version" : "1.8.0"
          },
          "input" : {
            "type" : "log"
          },
          "log" : {
            "file" : {
              "path" : "/Users/sbrown/workspace/elastic-install/7.13.0/filebeat-7.13.0-darwin-x86_64/test.ndjson"
            },
            "offset" : 168
          },
          "name" : "value5"
        }
      },
      {
        "_index" : "my-discuss-index",
        "_type" : "_doc",
        "_id" : "_kziBnoBhPeJjpPZ_R56",
        "_score" : 1.0,
        "_source" : {
          "container" : {
            "id" : "7.13.0"
          },
          "name2" : "value22",
          "@version" : "1",
          "@timestamp" : "2021-06-13T19:39:35.438Z",
          "ecs" : {
            "version" : "1.8.0"
          },
          "input" : {
            "type" : "log"
          },
          "log" : {
            "offset" : 42,
            "file" : {
              "path" : "/Users/sbrown/workspace/elastic-install/7.13.0/filebeat-7.13.0-darwin-x86_64/test.ndjson"
            }
          },
          "name" : "value2"
        }
      },
      {
        "_index" : "my-discuss-index",
        "_type" : "_doc",
        "_id" : "-0ziBnoBhPeJjpPZ_R56",
        "_score" : 1.0,
        "_source" : {
          "container" : {
            "id" : "7.13.0"
          },
          "name2" : "value24",
          "@version" : "1",
          "@timestamp" : "2021-06-13T19:39:35.438Z",
          "ecs" : {
            "version" : "1.8.0"
          },
          "input" : {
            "type" : "log"
          },
          "log" : {
            "offset" : 126,
            "file" : {
              "path" : "/Users/sbrown/workspace/elastic-install/7.13.0/filebeat-7.13.0-darwin-x86_64/test.ndjson"
            }
          },
          "name" : "value4"
        }
      },
      {
        "_index" : "my-discuss-index",
        "_type" : "_doc",
        "_id" : "_EziBnoBhPeJjpPZ_R56",
        "_score" : 1.0,
        "_source" : {
          "container" : {
            "id" : "7.13.0"
          },
          "name2" : "value26",
          "@version" : "1",
          "@timestamp" : "2021-06-13T19:39:35.438Z",
          "ecs" : {
            "version" : "1.8.0"
          },
          "input" : {
            "type" : "log"
          },
          "log" : {
            "offset" : 210,
            "file" : {
              "path" : "/Users/sbrown/workspace/elastic-install/7.13.0/filebeat-7.13.0-darwin-x86_64/test.ndjson"
            }
          },
          "name" : "value6"
        }
      }
    ]
  }
}

```

---

<div class="post-metadata">

**Author:** ![rafaell2](https://avatars.discourse-cdn.com/v4/letter/r/94ad74/32.png) [@rafaell2](https://discuss.elastic.co/u/rafaell2)\
**Post date:** [June 13, 2021, 9:58pm UTC](https://discuss.elastic.co/t/number-of-docs-count-does-not-match-number-of-json-file-records/275739/8 "2021-06-13T21:58:26Z")

</div>

Hi Stephen !!! hate to say it BUT IT WORKED !!!

This line was causing the whole mess !!! Not sure from where i got this example, but i will need to go back to basis and figure out what is this for !!! But if i leave it i get just one document, but without it it works just fine !!!

```auto
document_id => "%{[@metadata][document_id]}"

```

Regarding the missing line, indeed was because the lack of a new line at the end of the file !!!

So, thanks a million to all of you who helped me out with this issue !!!

Stay safe !!! Take care & regards

Rafael

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [July 11, 2021, 11:58pm UTC](https://discuss.elastic.co/t/number-of-docs-count-does-not-match-number-of-json-file-records/275739/9 "2021-07-11T23:58:46Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.