File copy size that appears multiple times

[learner]

I downloaded a 2.4 GB file from the Internet and the same 2.4 GB size appears multiple times in my query result, as below.

Is this normal? Why does ES show the total size multiple times rather than just once at the end of the download?

My query: dest.ip:10.10.243.236 AND source.stats.net_bytes_total:>2590000000

Thanks.

• Young

[Brandon_Kobel]

Hey @learner the source.state.net_bytes_total that you're seeing is coming from the document that is indexed into Elasticsearch, you have multiple documents in Elasticsearch and each of them have that net_bytes_total field.

[learner]

Hi Brandon,

Yeah, apparently those search results are coming from ES but do you know of any source document where it explains why ES gets multiple documents for a single file transfer instead of just once when it gets completed?

Thank you for your help!!!

• Young

[Brandon_Kobel]

@learner how are you indexing your documents in Elasticsearch? Are you using Logstash, Beats or the Ingest Node?

[learner]

Hi Brandon,

I'm using Packetbeat for ES indexing.

Thanks.

• Young

[Brandon_Kobel]

Ah... Packetbeat's net_bytes_total is an always increasing value that is the total bytes that have been transferred over your network connection.

[learner]

OK, my searches came up with progressed sizes during the transfer, which made sense, but I was expecting just one final document when it's completed.

I guess it creates multiple documents in microscopic intervals while it's wrapping up the completion.

Thank you!

• Young

[Brandon_Kobel]

Yup @learner, it's constantly sampling and writing documents to Elasticsearch. Packetbeat doesn't know about the files that are being downloaded themselves, but instead it's logged the total packets being sent over your network connection.

[learner]

I got it now.

Thank you very much for the helpful responses and explanations, Brandon!!!