Some entries missing when uploading json file in kibana

[Kanav_Ghai]

I have 153 entries in my json file, but when i upload the corresponding log file in kibana, it shows 121 entires (sometimes lesser) . I don't understand why is it missing out a few entries. Below is my conf file

    input {
    	file {
    		path => "C:/Users/ADMIN/Desktop/BTP/tweets.json"
    		start_position => "beginning"
    		sincedb_path => "NULL"
    		type => "json"
            file_completed_log_path => "NULL"
    	}
    }

    filter {
    	json {
    		source => "message"
    	}
    	mutate {
    		remove_field => "message"
    	}
    }

    output {
    	elasticsearch {
    		hosts => "http://localhost:9200"
    		index => "test_json1"
    	}
    }

please help

[aaron-nimocks]

Windows - sincedb_path => "nul"

Linux - sincedb_path => "/dev/null"

You probably don't need file_completed_log_path also.

If that doesn't fix it I would try to validate your entire JSON file to ensure it's valid.

[Kanav_Ghai]

Thanks for the quick reply!

I tried your suggestion, it didn't work (displayed 113 entries).

[2020-09-17T14:07:18,388][WARN ][logstash.outputs.elasticsearch][main][d89aae7cfe77d2a0efbee2a8145cb6c61774697e6c27700ab729173ced2aa465] Could not index event to Elasticsearch. {:status=>400, :action=>["index", {:_id=>nil, :_index=>"test_json1", :routing=>nil, :_type=>"_doc"}, #<LogStash::Event:0x12d7d7aa>], :response=>{"index"=>{"_index"=>"test_json1", "_type"=>"_doc", "_id"=>"epE1m3QBrwJIRxVUweSS", "status"=>400, "error"=>{"type"=>"illegal_argument_exception", "reason"=>"Limit of total fields [1000] in index [test_json1] has been exceeded"}}}}

This is my json file

tweets.json

and this is my conf file

jsonconf.conf

[aaron-nimocks]

@Kanav_Ghai I would verify your json is valid. When I run it in a validator it fails.