Squid module does not parse logs

[stephenb (Stephen Brown)]

@it_dev

Why lots of bunch of empty and unnecessary fields are also shipped? aws.cloud, azure.activitylogs etc?

Show me.
There are many fields available in the mapping ... but they are not shipped in each squid document.

Show me the extra fields in one of the squid documents.

@it_dev

The real log size of squid ~25mb. But in Data Streams it shows 66mb.

What do you mean the "real" squid logs logs size the size of the file... show me where you the data stream size.

GET _cat/indices?v

Two things 1 when indices / data streams first get created there is some overhead overtime elasticsearch will optimizes that overhead and the average document size will shrink. These processes are in the background. 2nd 66mb is very small in terms of index size and some the level of the optimization is also small...

[it_dev]

On syslog server, size of access.log is 20 mb.
In ELK it is 66.6 mb.

[it_dev]

As you can see, there are many unnecessary fields (7278) in filebeat.

Only squid module is working. Why are there unnecessary fields?