TShark and Rotating JSON Files

Elastic Stack Beats
1

Greetings ...

I am searching and searching and cannot seem to find a way to have TShark capture packets live to ElasticSearch JSON format and rotate them so I create a new JSON file, say, every hour or every 6 hours or whatever ...

Otherwise it seems that the TShark JSON output just grows and grows and grows until you run out of Disk space ...

If I could capture the JSON file and rotate it I could delete the ElasticSearch JSON files that have already been ingested or I could archive them or whatever ....

I've got the ElasticSearch Index created ... have the Ingest Pipeline created to rename fields and do GeoIP lookups ... the works ... and I've tested it manually from the console and it works like a charm!

Now I just need to feed the JSON file into Elastic Cloud using Filebeat and I'm off to the races!

Thanks in advance for any tips people can send my way!

Cheers!

2

This is more a tshark questions. Never tried myself, but have you tried with the -b flag?

$ tshark -h
...
Capture output:
  -b <ringbuffer opt.> ... duration:NUM - switch to next file after NUM secs
                           filesize:NUM - switch to next file after NUM KB
                              files:NUM - ringbuffer: replace after NUM files
...
3

Steffens ...

Thanks kindly for the reply ....

Absolutely ... it's purely a TShark question ... but since it has direct ties into ElasticSearch and because it's reference HERE I figured I'd post the question here in the off chance someone else has experienced this ...

If you try and run TShark with the -b option while outputting to JSON you get the attached error ..

55%20AM

If you try running the commands like the attached example it will properly rotate the PCAPs but the JSON file just keeps growing and growing ... the 2nd try in the attached didn't embed the date on the JSON but the JSON just kept growing as well ..

38%20AM
Screen Shot 2019-08-14 at 9.19.38 AM.png869×123 11.4 KB

There's gotta be a way to rotate the JSON ... I'll keep searching but if someone has a great idea I'm all ears ... I suppose we could write a CRON job of some sort to try and do the rotation but with that file filling up as quickly as it does I could a CRON job solution being fraught with issues ...

Thanks all ... cheers!

4

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.