# Difference between Doc count and Number of events sent by a collector **URL:** \ **Category:** Elasticsearch\ **Created:** [April 14, 2020, 11:12am UTC](https://discuss.elastic.co/t/difference-between-doc-count-and-number-of-events-sent-by-a-collector/227896 "2020-04-14T11:12:23Z")\ **Posts on this page:** 6\ **Page:** 1
**Author:** ![hilo21](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/hilo21/32/66272_2.png) [@hilo21](https://discuss.elastic.co/u/hilo21)\ **Post date:** [April 14, 2020, 11:12am UTC](https://discuss.elastic.co/t/difference-between-doc-count-and-number-of-events-sent-by-a-collector/227896/1 "2020-04-14T11:12:23Z")
For sizing purposes, I am studying how much data size would be taken by different logs. I have ingested more than 10000 events (PANOS events automatically generated by a script) but i only see 4472 documents in my index. the reason I am asking this question is, with my understanding of this, while I was slowly increasing the number of ingested events I noticed that the number of events i generate is always equal to the number of documents : `yellow open traffic 7rMZiJ6BS5W4aW0zNEH6Jg 1 1 4472 0 2.7mb 2.7mb` But when i started ingesting more and more the number of documents stopped matching the number of events. You should know that my script only changes the receiving date and destination IP randomly the rest is the same. I wanna know if there is something else going on under elasticsearch that limits the number of documents. Thank you ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [April 14, 2020, 11:19am UTC](https://discuss.elastic.co/t/difference-between-doc-count-and-number-of-events-sent-by-a-collector/227896/2 "2020-04-14T11:19:11Z")
Are you by any chance specifying a document ID when you index? ---
**Author:** ![hilo21](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/hilo21/32/66272_2.png) [@hilo21](https://discuss.elastic.co/u/hilo21)\ **Post date:** [April 14, 2020, 11:28am UTC](https://discuss.elastic.co/t/difference-between-doc-count-and-number-of-events-sent-by-a-collector/227896/3 "2020-04-14T11:28:29Z")
no, i am not specifying anything about adding a document id in my logstash config file ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [April 14, 2020, 11:29am UTC](https://discuss.elastic.co/t/difference-between-doc-count-and-number-of-events-sent-by-a-collector/227896/4 "2020-04-14T11:29:35Z")
What does your Logstash config file look like? ---
**Author:** ![hilo21](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/hilo21/32/66272_2.png) [@hilo21](https://discuss.elastic.co/u/hilo21)\ **Post date:** [April 14, 2020, 11:38am UTC](https://discuss.elastic.co/t/difference-between-doc-count-and-number-of-events-sent-by-a-collector/227896/5 "2020-04-14T11:38:01Z")
```auto input { file { path => ["/home/elastic/test/*.log"] start_position => "beginning" tags => ["paloalto"] } } filter { if "paloalto" in [tags] { if ([message] =~ /TRAFFIC/) { grok { patterns_dir => ["/home/elastic/logstash/patterns"] match => { "message" => "ReceiveTime=%{PANDATE:ReceiveTime}\|SerialNumber=%{NUMBER:SerialNumber}\|Type=%{UPWORD:Type}\|Subtype=%{DATA:Subtype}\|devTime=%{DATA:devTime}\|src=%{IP:SoucreIP}\|dst=%{IP:DestinationIP}\|srcPostNAT=%{IP:NATSourceIP}\|dstPostNAT=%{IP:NATDestinationIP}\|RuleName=%{DATA:RuleName}\|usrName=%{DATA:userName}\|SourceUser=%{DATA:SourceUser}\|DestinationUser=%{DATA:DestinationUser}\|Application=%{DATA:Application}\|VirtualSystem=%{DATA:VirtualSystem}\|SourceZone=%{DATA:SourceZone}\|DestinationZone=%{DATA:DestinationZone}\|IngressInterface=%{DATA:InboundInterface}\|EgressInterface=%{DATA:OuboundInterface}\|LogForwardingProfile=%{DATA:LogForwardingProfile}\|SessionID=%{NUMBER:SessionID}\|RepeatCount=%{NUMBER:RepeatCount}\|srcPort=%{NUMBER:SourcePort}\|dstPort=%{NUMBER:DestinationPort}\|srcPostNATPort=%{NUMBER:NATSourcePort}\|dstPostNATPort=%{NUMBER:NATDestinationPort}\|Flags=%{DATA:Flags}\|proto=%{DATA:Protocol}\|action=%{DATA:Action}\|totalBytes=%{NUMBER:Bytes}\|dstBytes=%{NUMBER:BytesSent}\|srcBytes=%{NUMBER:BytesReceived}\|totalPackets=%{NUMBER:Packets}\|StartTime=%{PANDATE:StartTime}\|ElapsedTime=%{NUMBER:ElapsedTime}\|URLCategory=%{DATA:URLCategory}\|sequence=%{NUMBER:SequenceNumber}\|ActionFlags=%{DATA:ActionFlags}\|SourceLocation=%{DATA:SourceLocation}\|DestinationLocation=%{DATA:DestinationLocation}\|dstPackets=%{NUMBER:PacketsSent}\|srcPackets=%{NUMBER:PacketsReceived}\|SessionEndReason=%{DATA:SessionEndReason}\|DeviceGroupHierarchyL1=%{DATA:DeviceGroupHierarchyL1}\|DeviceGroupHierarchyL2=%{DATA:DeviceGroupHierarchyL2}\|DeviceGroupHierarchyL3=%{DATA:DeviceGroupHierarchyL3}\|DeviceGroupHierarchyL4=%{DATA:DeviceGroupHierarchyL4}\|vSrcName=%{DATA:VirtualSystemName}\|DeviceName=%{DATA:DeviceName}\|ActionSource=%{DATA:ActionSource}\|SrcUUID=%{DATA:SourceVMUUID}\|DstUUID=%{DATA:DestinationVMUUID}\|TunnelID=%{DATA:TunnelID_IMSI}\|MonitorTag=%{DATA:MonitorTag_IMEI}\|ParentSessionID=%{DATA:ParentSessionID}\|ParentStartTime=%{DATA:ParentSartTime}\|TunnelType=%{DATA:TunnelType}" } } mutate { convert => ["Bytes", "integer"] convert => ["BytesReceived", "integer"] convert => ["BytesSent", "integer"] convert => ["ElapsedTime", "integer"] convert => ["GeoIP.dma_code", "integer"] convert => ["GeoIP.latitude", "float"] convert => ["GeoIP.longitude", "float"] convert => ["NATDestinationPort", "integer"] convert => ["NATSourcePort", "integer"] convert => ["Packets", "integer"] convert => ["PacketsReceived", "integer"] convert => ["PacketsSent", "integer"] convert => ["SequenceNumber", "integer"] add_tag => ["PAN-OS_traffic"] } } mutate { # Original message has been fully parsed, so remove it. remove_field => ["message"] } if [SourceIP] and [SourceIP] !~ "(^127\.0\.0\.1)|(^10\.)|(^172\.1[6-9]\.)|(^172\.2[0-9]\.)|(^172\.3[0-1]\.)|(^192\.168\.)|(^169\.254\.)" { geoip { source => "SourceIP" target => "SourceIPGeo" } if ([SourceIPGeo.location] and [SourceIPGeo.location] =~ "0,0") { mutate { replace => ["SourceIPGeo.location", ""] } } } if [DestinationIP] and [DestinationIP] !~ "(^127\.0\.0\.1)|(^10\.)|(^172\.1[6-9]\.)|(^172\.2[0-9]\.)|(^172\.3[0-1]\.)|(^192\.168\.)|(^169\.254\.)" { geoip { source => "DestinationIP" target => "DestinationIPGeo" } if ([DestinationIPGeo.location] and [DestinationIPGeo.location] =~ "0,0") { mutate { replace => ["DestinationIPGeo.location", ""] } } } if [SourceIP] and [DestinationIP] { fingerprint { concatenate_sources => true method => "SHA1" key => "logstash" source => ["SourceIP", "SourcePort", "DestinationIP", "DestinationPort", "Protocol"] } } } } output { if "PAN-OS_traffic" in [tags] { elasticsearch { index => "traffic" hosts => ["localhost:9200"] } } } ``` ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [May 12, 2020, 11:38am UTC](https://discuss.elastic.co/t/difference-between-doc-count-and-number-of-events-sent-by-a-collector/227896/6 "2020-05-12T11:38:06Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.