Squid module does not parse logs

  1. I have Linux syslog server.
  2. My squid server (separate from Linux syslog) sends all its access.log files to Linux syslog server into /var/log/squid/*.log
  3. There, on my Linux syslog server I have installed filebeat 8.4.
  4. I have enabled Filebeat Squid module, showed its input as a file and var.paths as a /var/log/squid/*.log
  5. Elastic Stack is working on a separate server, ok.
  6. When I test filebeat test config, filebeat test output - OK, no problem.
  7. When I start filebeat (service filebeat start) it sends all logs in /var/log/squid folder to Elastic search. I can see it in Index Management.

If I discover my filebeat logs or create graph in Kibana —-> message is not parsed. It comes as a filestream. I mean that all Squid logs come in message format.
There is a special Squid module on my Linux syslog server enabled. Why logs come not parsed? It seems like Squid module has no effect.

What maybe the problem?

Hi @it_dev

Did you carefully follow all the steps in the Quick Start Guide?
And run the filebeat setup -e command? This is key... as shown here this setups up all the needed templates and ingest pipelines that parse the data.

A) If you did not do all that clean up everything and start over.

B) If you did all those things it is possible you squid log are non-standard and that is why they are not getting parsed...

If B can you provide 5 or 6 lines from your squid logs perhaps we can take a look.

Hello @stephenb !

Thank you for your reply.

A) I did all that.
B) Of course, I can provide logs:

1661367615.883 188607 10.1.32.17 TCP_TUNNEL/200 412492 CONNECT www.fotospor.com:443 - HIER_DIRECT/31.186.15.10 --
1661367621.886 186711 10.1.32.17 TCP_TUNNEL/200 2345 CONNECT cdnjs.cloudflare.com:443 - HIER_DIRECT/188.114.98.192 -
1661367622.019  65211 10.1.32.23 NONE/503 0 CONNECT mtalk.google.com:5228 - HIER_NONE/- -
1661367626.407 496760 10.1.32.50 TCP_TUNNEL/200 521 CONNECT dc1.ksn.kaspersky-labs.com:443 - HIER_DIRECT/62.128.100.45 -
1661367627.563 105758 10.1.32.18 TCP_TUNNEL/200 846 CONNECT emea1.views.cp.thomsonreuters.com:443 - HIER_DIRECT/159.220.1.49

Hope for your help.

I have 8.4.0 Elastic Stack

  1. I simply put those lines above in a file

  2. I configured the modules.d/squid.yml

- module: squid
  log:
    enabled: true

    # Set which input to use between udp (default), tcp or file.
    var.input: file
    # var.syslog_host: localhost
    # var.syslog_port: 9520

    # Set paths for the log files when file input is used.
    var.paths: ["/Users/sbrown/workspace/sample-data/discuss/squid.log"]

I ran

  1. filebeat setup -e <<<<Please confirm you ran this
    then I ran

  2. filebeat -e

And here are my results all the logs are parse.

So can you see they are properly parsed

GET filebeat-*/_search

{
  "took": 0,
  "timed_out": false,
  "_shards": {
    "total": 1,
    "successful": 1,
    "skipped": 0,
    "failed": 0
  },
  "hits": {
    "total": {
      "value": 5,
      "relation": "eq"
    },
    "max_score": 1,
    "hits": [
      {
        "_index": ".ds-filebeat-8.4.0-2022.08.29-000001",
        "_id": "w24F6IIBVm7wjCsj5KL6",
        "_score": 1,
        "_source": {
          "container": {
            "id": "discuss"
          },
          "rsa": {
            "internal": {
              "messageid": "CONNECT",
              "hcode": "HIER_DIRECT"
            },
            "web": {
              "alias_host": "www.fotospor.com"
            },
            "investigations": {
              "ec_subject": "NetworkComm",
              "ec_theme": "ALM"
            },
            "time": {
              "event_time_str": "1661367615",
              "duration_time": 188607,
              "event_time": "2022-08-24T19:00:15.000Z"
            },
            "misc": {
              "content_type": "--",
              "action": [
                "CONNECT",
                "TCP_TUNNEL"
              ],
              "result_code": "200"
            },
            "network": {
              "domain": "www.fotospor.com"
            }
          },
          "server": {
            "registered_domain": "fotospor.com",
            "top_level_domain": "com",
            "domain": "www.fotospor.com",
            "subdomain": "www"
          },
          "agent": {
            "name": "hyperion",
            "id": "f0dbfc56-c491-4f5e-920b-561fe2e54b2c",
            "type": "filebeat",
            "ephemeral_id": "b0b5d6a2-5f96-4a4b-9242-2b88d7e65fac",
            "version": "8.4.0"
          },
          "log": {
            "file": {
              "path": "/Users/sbrown/workspace/sample-data/discuss/squid.log"
            },
            "offset": 0
          },
          "destination": {
            "geo": {
              "continent_name": "Asia",
              "country_iso_code": "TR",
              "country_name": "Turkey",
              "location": {
                "lon": 28.9948,
                "lat": 41.0214
              }
            },
            "as": {
              "number": 197720,
              "organization": {
                "name": "TURKTICARET.NET YAZILIM HIZMETLERI SAN. ve TIC. A.S."
              }
            },
            "ip": "31.186.15.10"
          },
          "source": {
            "bytes": 412492,
            "ip": "10.1.32.17"
          },
          "fileset": {
            "name": "log"
          },
          "url": {
            "registered_domain": "fotospor.com",
            "original": "www.fotospor.com:443",
            "top_level_domain": "com",
            "domain": "www.fotospor.com",
            "subdomain": "www"
          },
          "tags": [
            "squid.log",
            "forwarded"
          ],
          "observer": {
            "product": "Proxy",
            "vendor": "Squid",
            "type": "Proxies"
          },
          "input": {
            "type": "log"
          },
          "@timestamp": "2022-08-24T19:00:15.000Z",
          "ecs": {
            "version": "1.12.0"
          },
          "related": {
            "hosts": [
              "www.fotospor.com"
            ],
            "ip": [
              "10.1.32.17",
              "31.186.15.10"
            ],
            "user": [
              "-"
            ]
          },
          "service": {
            "type": "squid"
          },
          "event": {
            "ingested": "2022-08-29T05:14:36.633267333Z",
            "original": "1661367615.883 188607 10.1.32.17 TCP_TUNNEL/200 412492 CONNECT www.fotospor.com:443 - HIER_DIRECT/31.186.15.10 --",
            "code": "CONNECT",
            "module": "squid",
            "action": "TCP_TUNNEL",
            "dataset": "squid.log"
          },
          "user": {
            "name": "-"
          }
        }
      },
      {
        "_index": ".ds-filebeat-8.4.0-2022.08.29-000001",
        "_id": "xG4F6IIBVm7wjCsj5KL6",
        "_score": 1,
        "_source": {
          "container": {
            "id": "discuss"
          },
          "rsa": {
            "internal": {
              "messageid": "CONNECT",
              "hcode": "HIER_DIRECT"
            },
            "web": {
              "alias_host": "cdnjs.cloudflare.com"
            },
            "investigations": {
              "ec_subject": "NetworkComm",
              "ec_theme": "ALM"
            },
            "time": {
              "event_time_str": "1661367621",
              "duration_time": 186711,
              "event_time": "2022-08-24T19:00:21.000Z"
            },
            "misc": {
              "content_type": "-",
              "action": [
                "CONNECT",
                "TCP_TUNNEL"
              ],
              "result_code": "200"
            },
            "network": {
              "domain": "cdnjs.cloudflare.com"
            }
          },
          "server": {
            "registered_domain": "cloudflare.com",
            "top_level_domain": "com",
            "domain": "cdnjs.cloudflare.com",
            "subdomain": "cdnjs"
          },
          "agent": {
            "name": "hyperion",
            "id": "f0dbfc56-c491-4f5e-920b-561fe2e54b2c",
            "type": "filebeat",
            "ephemeral_id": "b0b5d6a2-5f96-4a4b-9242-2b88d7e65fac",
            "version": "8.4.0"
          },
          "log": {
            "file": {
              "path": "/Users/sbrown/workspace/sample-data/discuss/squid.log"
            },
            "offset": 114
          },
          "destination": {
            "geo": {
              "continent_name": "Europe",
              "region_iso_code": "NL-NH",
              "city_name": "Amsterdam",
              "country_iso_code": "NL",
              "country_name": "Netherlands",
              "region_name": "North Holland",
              "location": {
                "lon": 4.8883,
                "lat": 52.3716
              }
            },
            "as": {
              "number": 13335,
              "organization": {
                "name": "CLOUDFLARENET"
              }
            },
            "ip": "188.114.98.192"
          },
          "source": {
            "bytes": 2345,
            "ip": "10.1.32.17"
          },
          "fileset": {
            "name": "log"
          },
          "url": {
            "registered_domain": "cloudflare.com",
            "original": "cdnjs.cloudflare.com:443",
            "top_level_domain": "com",
            "domain": "cdnjs.cloudflare.com",
            "subdomain": "cdnjs"
          },
          "tags": [
            "squid.log",
            "forwarded"
          ],
          "input": {
            "type": "log"
          },
          "observer": {
            "product": "Proxy",
            "vendor": "Squid",
            "type": "Proxies"
          },
          "@timestamp": "2022-08-24T19:00:21.000Z",
          "ecs": {
            "version": "1.12.0"
          },
          "related": {
            "hosts": [
              "cdnjs.cloudflare.com"
            ],
            "ip": [
              "10.1.32.17",
              "188.114.98.192"
            ],
            "user": [
              "-"
            ]
          },
          "service": {
            "type": "squid"
          },
          "event": {
            "ingested": "2022-08-29T05:14:36.660378441Z",
            "original": "1661367621.886 186711 10.1.32.17 TCP_TUNNEL/200 2345 CONNECT cdnjs.cloudflare.com:443 - HIER_DIRECT/188.114.98.192 -",
            "code": "CONNECT",
            "module": "squid",
            "action": "TCP_TUNNEL",
            "dataset": "squid.log"
          },
          "user": {
            "name": "-"
          }
        }
      },
      {
        "_index": ".ds-filebeat-8.4.0-2022.08.29-000001",
        "_id": "xW4F6IIBVm7wjCsj5KL6",
        "_score": 1,
        "_source": {
          "container": {
            "id": "discuss"
          },
          "rsa": {
            "internal": {
              "messageid": "CONNECT",
              "hcode": "HIER_NONE"
            },
            "web": {
              "alias_host": "mtalk.google.com"
            },
            "investigations": {
              "ec_subject": "NetworkComm",
              "ec_theme": "ALM"
            },
            "time": {
              "event_time_str": "1661367622",
              "duration_time": 65211,
              "event_time": "2022-08-24T19:00:22.000Z"
            },
            "misc": {
              "content_type": "-",
              "action": [
                "NONE",
                "CONNECT"
              ],
              "result_code": "503"
            },
            "network": {
              "domain": "mtalk.google.com"
            }
          },
          "server": {
            "registered_domain": "google.com",
            "top_level_domain": "com",
            "domain": "mtalk.google.com",
            "subdomain": "mtalk"
          },
          "agent": {
            "name": "hyperion",
            "id": "f0dbfc56-c491-4f5e-920b-561fe2e54b2c",
            "type": "filebeat",
            "ephemeral_id": "b0b5d6a2-5f96-4a4b-9242-2b88d7e65fac",
            "version": "8.4.0"
          },
          "log": {
            "file": {
              "path": "/Users/sbrown/workspace/sample-data/discuss/squid.log"
            },
            "offset": 231
          },
          "source": {
            "bytes": 0,
            "ip": "10.1.32.23"
          },
          "fileset": {
            "name": "log"
          },
          "url": {
            "registered_domain": "google.com",
            "original": "mtalk.google.com:5228",
            "top_level_domain": "com",
            "domain": "mtalk.google.com",
            "subdomain": "mtalk"
          },
          "tags": [
            "squid.log",
            "forwarded"
          ],
          "input": {
            "type": "log"
          },
          "observer": {
            "product": "Proxy",
            "vendor": "Squid",
            "type": "Proxies"
          },
          "@timestamp": "2022-08-24T19:00:22.000Z",
          "related": {
            "hosts": [
              "mtalk.google.com"
            ],
            "ip": [
              "10.1.32.23"
            ],
            "user": [
              "-"
            ]
          },
          "ecs": {
            "version": "1.12.0"
          },
          "service": {
            "type": "squid"
          },
          "event": {
            "ingested": "2022-08-29T05:14:36.663594492Z",
            "original": "1661367622.019  65211 10.1.32.23 NONE/503 0 CONNECT mtalk.google.com:5228 - HIER_NONE/- -",
            "code": "CONNECT",
            "module": "squid",
            "action": "NONE",
            "dataset": "squid.log"
          },
          "user": {
            "name": "-"
          }
        }
      },
      {
        "_index": ".ds-filebeat-8.4.0-2022.08.29-000001",
        "_id": "xm4F6IIBVm7wjCsj5KL6",
        "_score": 1,
        "_source": {
          "container": {
            "id": "discuss"
          },
          "rsa": {
            "internal": {
              "messageid": "CONNECT",
              "hcode": "HIER_DIRECT"
            },
            "web": {
              "alias_host": "dc1.ksn.kaspersky-labs.com"
            },
            "investigations": {
              "ec_subject": "NetworkComm",
              "ec_theme": "ALM"
            },
            "time": {
              "event_time_str": "1661367626",
              "duration_time": 496760,
              "event_time": "2022-08-24T19:00:26.000Z"
            },
            "network": {
              "domain": "dc1.ksn.kaspersky-labs.com"
            },
            "misc": {
              "content_type": "-",
              "action": [
                "TCP_TUNNEL",
                "CONNECT"
              ],
              "result_code": "200"
            }
          },
          "server": {
            "registered_domain": "kaspersky-labs.com",
            "top_level_domain": "com",
            "domain": "dc1.ksn.kaspersky-labs.com",
            "subdomain": "dc1.ksn"
          },
          "agent": {
            "name": "hyperion",
            "id": "f0dbfc56-c491-4f5e-920b-561fe2e54b2c",
            "type": "filebeat",
            "ephemeral_id": "b0b5d6a2-5f96-4a4b-9242-2b88d7e65fac",
            "version": "8.4.0"
          },
          "log": {
            "file": {
              "path": "/Users/sbrown/workspace/sample-data/discuss/squid.log"
            },
            "offset": 321
          },
          "destination": {
            "geo": {
              "continent_name": "Europe",
              "region_iso_code": "RU-MOW",
              "city_name": "Moscow",
              "country_iso_code": "RU",
              "country_name": "Russia",
              "region_name": "Moscow",
              "location": {
                "lon": 37.6171,
                "lat": 55.7483
              }
            },
            "as": {
              "number": 3327,
              "organization": {
                "name": "CITIC Telecom CPC Netherlands B.V."
              }
            },
            "ip": "62.128.100.45"
          },
          "source": {
            "bytes": 521,
            "ip": "10.1.32.50"
          },
          "fileset": {
            "name": "log"
          },
          "url": {
            "registered_domain": "kaspersky-labs.com",
            "original": "dc1.ksn.kaspersky-labs.com:443",
            "top_level_domain": "com",
            "domain": "dc1.ksn.kaspersky-labs.com",
            "subdomain": "dc1.ksn"
          },
          "tags": [
            "squid.log",
            "forwarded"
          ],
          "input": {
            "type": "log"
          },
          "observer": {
            "product": "Proxy",
            "vendor": "Squid",
            "type": "Proxies"
          },
          "@timestamp": "2022-08-24T19:00:26.000Z",
          "related": {
            "hosts": [
              "dc1.ksn.kaspersky-labs.com"
            ],
            "ip": [
              "10.1.32.50",
              "62.128.100.45"
            ],
            "user": [
              "-"
            ]
          },
          "ecs": {
            "version": "1.12.0"
          },
          "service": {
            "type": "squid"
          },
          "event": {
            "ingested": "2022-08-29T05:14:36.664195302Z",
            "original": "1661367626.407 496760 10.1.32.50 TCP_TUNNEL/200 521 CONNECT dc1.ksn.kaspersky-labs.com:443 - HIER_DIRECT/62.128.100.45 -",
            "code": "CONNECT",
            "module": "squid",
            "action": "TCP_TUNNEL",
            "dataset": "squid.log"
          },
          "user": {
            "name": "-"
          }
        }
      },
      {
        "_index": ".ds-filebeat-8.4.0-2022.08.29-000001",
        "_id": "x24F6IIBVm7wjCsj5KL6",
        "_score": 1,
        "_source": {
          "container": {
            "id": "discuss"
          },
          "rsa": {
            "internal": {
              "messageid": "CONNECT"
            }
          },
          "agent": {
            "name": "hyperion",
            "id": "f0dbfc56-c491-4f5e-920b-561fe2e54b2c",
            "ephemeral_id": "b0b5d6a2-5f96-4a4b-9242-2b88d7e65fac",
            "type": "filebeat",
            "version": "8.4.0"
          },
          "log": {
            "file": {
              "path": "/Users/sbrown/workspace/sample-data/discuss/squid.log"
            },
            "offset": 442,
            "flags": [
              "dissect_parsing_error"
            ]
          },
          "fileset": {
            "name": "log"
          },
          "tags": [
            "squid.log",
            "forwarded"
          ],
          "input": {
            "type": "log"
          },
          "observer": {
            "product": "Proxy",
            "vendor": "Squid",
            "type": "Proxies"
          },
          "@timestamp": "2022-08-29T05:14:35.586Z",
          "ecs": {
            "version": "1.12.0"
          },
          "service": {
            "type": "squid"
          },
          "event": {
            "ingested": "2022-08-29T05:14:36.666485929Z",
            "original": "1661367627.563 105758 10.1.32.18 TCP_TUNNEL/200 846 CONNECT emea1.views.cp.thomsonreuters.com:443 - HIER_DIRECT/159.220.1.49",
            "code": "CONNECT",
            "module": "squid",
            "dataset": "squid.log"
          }
        }
      }
    ]
  }
}

What happens when you run from the Kibana Dev Tools

GET _ingest/pipeline/filebeat-8.4.0-squid-log-pipeline

You are missing a step somewhere.

filebeat setup -e - yes I do this.

Still logs are not parsed.
And when I do filebeat -e, there is no output in console like yours.
Instead, there are many notifications.
One of them is this:

{"log.level":"warn","@timestamp":"2022-08-29T08:04:31.717Z","log.logger":"cfgwarn","log.origin":{"file.name":"log/input.go","file.line":90},"message":"DEPRECATED: Log input. Use Filestream input instead.","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-08-29T08:04:31.718Z","log.logger":"input","log.origin":{"file.name":"log/input.go","file.line":172},"message":"Configured paths: [/var/log/squid/*.log]","service.name":"filebeat","input_id":"93519840-1d95-464a-a208-67100856a47d","ec

My Elastic Stack is 8.3.3
My Filebeat version is 8.4.0

Can this be the reason?

I have updated my ELK to 8.4.0, problem still exists.

It is not a good idea to you a beat newer than the stack.

That is from Kibana -> Dev Tools

GET filebeat-*/_search

Please post your entire
filebeat.yml
And
modules.d/squid.yml

You have something miss-configured.

Did you enable the squid module

filebeat modules enable squid

GET filebeat-*/_search:

{
"took": 1,
"timed_out": false,
"_shards": {
"total": 1,
"successful": 1,
"skipped": 0,
"failed": 0
},
"hits": {
"total": {
"value": 0,
"relation": "eq"
},
"max_score": null,
"hits":
}
}

filebeat.yml:

###################### Filebeat Configuration Example #########################

# This file is an example configuration file highlighting only the most common
# options. The filebeat.reference.yml file from the same directory contains all the
# supported options with more comments. You can use it as a reference.
#
# You can find the full configuration reference here:
# https://www.elastic.co/guide/en/beats/filebeat/index.html

# For more available modules and options, please see the filebeat.reference.yml sample
# configuration file.

# ============================== Filebeat inputs ===============================

filebeat.inputs:

# Each - is an input. Most options can be set at the input level, so
# you can use different inputs for various configurations.
# Below are the input specific configurations.

# filestream is an input for collecting log messages from files.
- type: filestream

  # Unique ID among all inputs, an ID is required.
  id: my-filestream-id

  # Change to true to enable this input configuration.
  enabled: false

  # Paths that should be crawled and fetched. Glob based paths.
  paths:
    - /var/log/*.log
    #- c:\programdata\elasticsearch\logs\*
  # Exclude lines. A list of regular expressions to match. It drops the lines that are
  # matching any regular expression from the list.
  # Line filtering happens after the parsers pipeline. If you would like to filter lines
  # before parsers, use include_message parser.
  #exclude_lines: ['^DBG']

  # Include lines. A list of regular expressions to match. It exports the lines that are
  # matching any regular expression from the list.
  # Line filtering happens after the parsers pipeline. If you would like to filter lines
  # before parsers, use include_message parser.
  #include_lines: ['^ERR', '^WARN']

  # Exclude files. A list of regular expressions to match. Filebeat drops the files that
  # are matching any regular expression from the list. By default, no files are dropped.
  #prospector.scanner.exclude_files: ['.gz$']

  # Optional additional fields. These fields can be freely picked
  # to add additional information to the crawled log files for filtering
  #fields:
  #  level: debug
  #  review: 1

# ============================== Filebeat modules ==============================

filebeat.config.modules:
  # Glob pattern for configuration loading
  path: ${path.config}/modules.d/*.yml

  # Set to true to enable config reloading
  reload.enabled: true

  # Period on which files under path should be checked for changes
  #reload.period: 10s

# ======================= Elasticsearch template setting =======================

setup.template.settings:
  index.number_of_shards: 1
  #index.codec: best_compression
  #_source.enabled: false


# ================================== General ===================================

# The name of the shipper that publishes the network data. It can be used to group
# all the transactions sent by a single shipper in the web interface.
#name:

# The tags of the shipper are included in their own field with each
# transaction published.
#tags: ["service-X", "web-tier"]

# Optional fields that you can specify to add additional information to the
# output.
#fields:
#  env: staging

# ================================= Dashboards =================================
# These settings control loading the sample dashboards to the Kibana index. Loading
# the dashboards is disabled by default and can be enabled either by setting the
# options here or by using the `setup` command.
#setup.dashboards.enabled: false

# The URL from where to download the dashboards archive. By default this URL
# has a value which is computed based on the Beat name and version. For released
# versions, this URL points to the dashboard archive on the artifacts.elastic.co
# website.
#setup.dashboards.url:

# =================================== Kibana ===================================

# Starting with Beats version 6.0.0, the dashboards are loaded via the Kibana API.
# This requires a Kibana endpoint configuration.
setup.kibana:

  # Kibana Host
  # Scheme and port can be left out and will be set to the default (http and 5601)
  # In case you specify and additional path, the scheme is required: http://localhost:5601/path
  # IPv6 addresses should always be defined as: https://[2001:db8::1]:5601
  #host: "localhost:5601"
  # Kibana Space ID
  # ID of the Kibana Space into which the dashboards should be loaded. By default,
  # the Default Space will be used.
  #space.id:

# =============================== Elastic Cloud ================================

# These settings simplify using Filebeat with the Elastic Cloud (https://cloud.elastic.co/).

# The cloud.id setting overwrites the `output.elasticsearch.hosts` and
# `setup.kibana.host` options.
# You can find the `cloud.id` in the Elastic Cloud web UI.
#cloud.id:

# The cloud.auth setting overwrites the `output.elasticsearch.username` and
# `output.elasticsearch.password` settings. The format is `<user>:<pass>`.
#cloud.auth:

# ================================== Outputs ===================================

# Configure what output to use when sending the data collected by the beat.

# ---------------------------- Elasticsearch Output ----------------------------
output.elasticsearch:
  # Array of hosts to connect to.
#A  hosts: ["localhost:9200"]
    hosts: ["https://10.1.32.44:9200"]
    username: "test-user"
    password: "test-pass"
    ssl:
      enabled: true
      ca_trusted_fingerprint: "69bd0555b28fbc14527506352a7374ae3697b219456f900542dec288bc870b3c"
  # Protocol - either `http` (default) or `https`.
  #protocol: "https"

  # Authentication credentials - either API key or username/password.
  #api_key: "id:api_key"
  #username: "elastic"
  #password: "changeme"
#output.elasticsearch.allow_older_versions: true
# ------------------------------ Logstash Output -------------------------------
#output.logstash:
  # The Logstash hosts
  #hosts: ["localhost:5044"]

  # Optional SSL. By default is off.
  # List of root certificates for HTTPS server verifications
  #ssl.certificate_authorities: ["/etc/pki/root/ca.pem"]

  # Certificate for SSL client authentication
  #ssl.certificate: "/etc/pki/client/cert.pem"

  # Client Certificate Key
  #ssl.key: "/etc/pki/client/cert.key"

# ================================= Processors =================================
processors:
  - add_host_metadata:
      when.not.contains.tags: forwarded
  - add_cloud_metadata: ~
  - add_docker_metadata: ~
  - add_kubernetes_metadata: ~

# ================================== Logging ===================================

# Sets log level. The default log level is info.
# Available log levels are: error, warning, info, debug
#logging.level: debug

# At debug level, you can selectively enable logging only for some components.
# To enable all selectors use ["*"]. Examples of other selectors are "beat",
# "publisher", "service".
#logging.selectors: ["*"]

# ============================= X-Pack Monitoring ==============================
# Filebeat can export internal metrics to a central Elasticsearch monitoring
# cluster.  This requires xpack monitoring to be enabled in Elasticsearch.  The
# reporting is disabled by default.

# Set to true to enable the monitoring reporter.
#monitoring.enabled: false

# Sets the UUID of the Elasticsearch cluster under which monitoring data for this
# Filebeat instance will appear in the Stack Monitoring UI. If output.elasticsearch
# is enabled, the UUID is derived from the Elasticsearch cluster referenced by output.elasticsearch.
#monitoring.cluster_uuid:

# Uncomment to send the metrics to Elasticsearch. Most settings from the
# Elasticsearch output are accepted here as well.
# Note that the settings should point to your Elasticsearch *monitoring* cluster.
# Any setting that is not set is automatically inherited from the Elasticsearch
# output configuration, so if you have the Elasticsearch output configured such
# that it is pointing to your Elasticsearch monitoring cluster, you can simply
# uncomment the following line.
#monitoring.elasticsearch:

# ============================== Instrumentation ===============================

# Instrumentation support for the filebeat.
#instrumentation:
    # Set to true to enable instrumentation of filebeat.
    #enabled: false

    # Environment in which filebeat is running on (eg: staging, production, etc.)
    #environment: ""

    # APM Server hosts to report instrumentation results to.
    #hosts:
    #  - http://localhost:8200

    # API Key for the APM Server(s).
    # If api_key is set then secret_token will be ignored.
    #api_key:

    # Secret token for the APM Server(s).
    #secret_token:


# ================================= Migration ==================================

# This allows to enable 6.7 migration aliases
#migration.6_to_7.enabled: true

modules.d/squid.yml:

# Module: squid
# Docs: https://www.elastic.co/guide/en/beats/filebeat/main/filebeat-module-squid.html

- module: squid
  log:
    enabled: true

    # Set which input to use between udp (default), tcp or file.
    # var.input: udp
    var.input: file
    # var.syslog_host: localhost
    # var.syslog_port: 9520
    # Set paths for the log files when file input is used.
    var.paths: ["/var/log/squid/access.log"]

    # Toggle output of non-ECS fields (default true).
    # var.rsa_fields: true

    # Set custom timezone offset.
    # "local" (default) for system timezone.
    # "+02:00" for GMT+02:00
    # var.tz_offset: local

Ok the logs are not loading at all... Your search shows that.

I suspect filebeat already read that log file once.. it will not re-read it again.

You need to clean out the filebeat data directory and try again. Look at the directory layout for the install you did clean the data directory and try again.

Tried to remove filebeat and install it again.
It says: Version 8.4.0 of Filebeat has not yet been released.

So I installed Filebeat 7.17.6.

Same thing, logs are not parsed and not fully sent.

In general your configs look pretty good

There is something basic wrong.. what does the filebeat logs show...

Did you clean up the data directory? and run again?

Version 8.4.0 of Filebeat has not yet been released.
Not sure where you saw the that message you are not giving context? 8.4.0 filebeat is certainly released. That could be a bug... so trying 7.17.6 is fine.

Please provide the filebeat startup logs... after you clean up everything... and run it... there is something basic wrong.. .this took me 5 mins end to end

Please Show the filebeat logs.

Please show messages look like inside elasticsearch.

filebeat setup -e logs:

{"log.level":"info","@timestamp":"2022-08-30T07:39:31.387Z","log.origin":{"file.name":"instance/beat.go","file.line":702},"message":"Home path: [/usr/share/filebeat] Config path: [/etc/filebeat] Data path: [/var/lib/filebeat] Logs path: [/var/log/filebeat]","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-08-30T07:39:31.387Z","log.origin":{"file.name":"instance/beat.go","file.line":710},"message":"Beat ID: 18f9fb7b-d419-4ad5-b357-158a87350fbf","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"warn","@timestamp":"2022-08-30T07:39:34.391Z","log.logger":"add_cloud_metadata","log.origin":{"file.name":"add_cloud_metadata/provider_aws_ec2.go","file.line":81},"message":"read token request for getting IMDSv2 token returns empty: Put \"http://169.254.169.254/latest/api/token\": context deadline exceeded (Client.Timeout exceeded while awaiting headers). No token in the metadata request will be used.","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-08-30T07:39:34.392Z","log.logger":"beat","log.origin":{"file.name":"instance/beat.go","file.line":1076},"message":"Beat info","service.name":"filebeat","system_info":{"beat":{"path":{"config":"/etc/filebeat","data":"/var/lib/filebeat","home":"/usr/share/filebeat","logs":"/var/log/filebeat"},"type":"filebeat","uuid":"18f9fb7b-d419-4ad5-b357-158a87350fbf"},"ecs.version":"1.6.0"}}
{"log.level":"info","@timestamp":"2022-08-30T07:39:34.392Z","log.logger":"beat","log.origin":{"file.name":"instance/beat.go","file.line":1085},"message":"Build info","service.name":"filebeat","system_info":{"build":{"commit":"ce383f1368ec7b3234de2dd4b1302be8db84fe1a","libbeat":"8.4.0","time":"2022-08-18T12:23:26.000Z","version":"8.4.0"},"ecs.version":"1.6.0"}}
{"log.level":"info","@timestamp":"2022-08-30T07:39:34.392Z","log.logger":"beat","log.origin":{"file.name":"instance/beat.go","file.line":1088},"message":"Go runtime info","service.name":"filebeat","system_info":{"go":{"os":"linux","arch":"amd64","max_procs":4,"version":"go1.17.12"},"ecs.version":"1.6.0"}}
{"log.level":"info","@timestamp":"2022-08-30T07:39:34.393Z","log.logger":"beat","log.origin":{"file.name":"instance/beat.go","file.line":1092},"message":"Host info","service.name":"filebeat","system_info":{"host":{"architecture":"x86_64","boot_time":"2022-08-30T07:32:31Z","containerized":false,"name":"websrv","ip":["127.0.0.1/8","::1/128","10.1.32.123/24","fe80::6c32:8aff:fe4a:2f47/64"],"kernel_version":"5.15.0-46-generic","mac":["6e:32:8a:4a:2f:47"],"os":{"type":"linux","family":"debian","platform":"ubuntu","name":"Ubuntu","version":"22.04.1 LTS (Jammy Jellyfish)","major":22,"minor":4,"patch":1,"codename":"jammy"},"timezone":"UTC","timezone_offset_sec":0,"id":"4d73804ce2ae43e196bdb2fba425d9af"},"ecs.version":"1.6.0"}}
{"log.level":"info","@timestamp":"2022-08-30T07:39:34.393Z","log.logger":"beat","log.origin":{"file.name":"instance/beat.go","file.line":1121},"message":"Process info","service.name":"filebeat","system_info":{"process":{"capabilities":{"inheritable":null,"permitted":["chown","dac_override","dac_read_search","fowner","fsetid","kill","setgid","setuid","setpcap","linux_immutable","net_bind_service","net_broadcast","net_admin","net_raw","ipc_lock","ipc_owner","sys_module","sys_rawio","sys_chroot","sys_ptrace","sys_pacct","sys_admin","sys_boot","sys_nice","sys_resource","sys_time","sys_tty_config","mknod","lease","audit_write","audit_control","setfcap","mac_override","mac_admin","syslog","wake_alarm","block_suspend","audit_read","38","39","40"],"effective":["chown","dac_override","dac_read_search","fowner","fsetid","kill","setgid","setuid","setpcap","linux_immutable","net_bind_service","net_broadcast","net_admin","net_raw","ipc_lock","ipc_owner","sys_module","sys_rawio","sys_chroot","sys_ptrace","sys_pacct","sys_admin","sys_boot","sys_nice","sys_resource","sys_time","sys_tty_config","mknod","lease","audit_write","audit_control","setfcap","mac_override","mac_admin","syslog","wake_alarm","block_suspend","audit_read","38","39","40"],"bounding":["chown","dac_override","dac_read_search","fowner","fsetid","kill","setgid","setuid","setpcap","linux_immutable","net_bind_service","net_broadcast","net_admin","net_raw","ipc_lock","ipc_owner","sys_module","sys_rawio","sys_chroot","sys_ptrace","sys_pacct","sys_admin","sys_boot","sys_nice","sys_resource","sys_time","sys_tty_config","mknod","lease","audit_write","audit_control","setfcap","mac_override","mac_admin","syslog","wake_alarm","block_suspend","audit_read","38","39","40"],"ambient":null},"cwd":"/etc/filebeat","exe":"/usr/share/filebeat/bin/filebeat","name":"filebeat","pid":1167,"ppid":1083,"seccomp":{"mode":"disabled","no_new_privs":false},"start_time":"2022-08-30T07:39:30.150Z"},"ecs.version":"1.6.0"}}
{"log.level":"info","@timestamp":"2022-08-30T07:39:34.394Z","log.origin":{"file.name":"instance/beat.go","file.line":293},"message":"Setup Beat: filebeat; Version: 8.4.0","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"warn","@timestamp":"2022-08-30T07:39:34.397Z","log.logger":"cfgwarn","log.origin":{"file.name":"tlscommon/config.go","file.line":102},"message":"DEPRECATED: Treating the CommonName field on X.509 certificates as a host name when no Subject Alternative Names are present is going to be removed. Please update your certificates if needed. Will be removed in version: 8.0.0","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-08-30T07:39:34.397Z","log.logger":"esclientleg","log.origin":{"file.name":"eslegclient/connection.go","file.line":108},"message":"elasticsearch url: https://10.1.32.44:9200","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-08-30T07:39:34.398Z","log.logger":"publisher","log.origin":{"file.name":"pipeline/module.go","file.line":113},"message":"Beat name: websrv","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-08-30T07:39:34.398Z","log.logger":"modules","log.origin":{"file.name":"fileset/modules.go","file.line":108},"message":"Enabled modules/filesets: ","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-08-30T07:39:34.398Z","log.logger":"esclientleg","log.origin":{"file.name":"eslegclient/connection.go","file.line":108},"message":"elasticsearch url: https://10.1.32.44:9200","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-08-30T07:39:34.426Z","log.logger":"esclientleg","log.origin":{"file.name":"eslegclient/connection.go","file.line":291},"message":"Attempting to connect to Elasticsearch version 8.4.0","service.name":"filebeat","ecs.version":"1.6.0"}
Overwriting ILM policy is disabled. Set `setup.ilm.overwrite: true` for enabling.

{"log.level":"info","@timestamp":"2022-08-30T07:39:34.426Z","log.logger":"index-management","log.origin":{"file.name":"idxmgmt/std.go","file.line":231},"message":"Auto ILM enable success.","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-08-30T07:39:34.433Z","log.logger":"index-management.ilm","log.origin":{"file.name":"ilm/std.go","file.line":118},"message":"ILM policy filebeat exists already.","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-08-30T07:39:34.433Z","log.logger":"index-management","log.origin":{"file.name":"idxmgmt/std.go","file.line":366},"message":"Set settings.index.lifecycle.name in template to {filebeat {\"policy\":{\"phases\":{\"hot\":{\"actions\":{\"rollover\":{\"max_age\":\"30d\",\"max_primary_shard_size\":\"50gb\"}}}}}}} as ILM is enabled.","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-08-30T07:39:34.465Z","log.logger":"template","log.origin":{"file.name":"template/load.go","file.line":245},"message":"Existing template will be overwritten, as overwrite is enabled.","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-08-30T07:39:37.312Z","log.logger":"template_loader","log.origin":{"file.name":"template/load.go","file.line":159},"message":"Try loading template filebeat-8.4.0 to Elasticsearch","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-08-30T07:39:37.393Z","log.logger":"add_cloud_metadata","log.origin":{"file.name":"add_cloud_metadata/add_cloud_metadata.go","file.line":102},"message":"add_cloud_metadata: hosting provider type not detected.","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-08-30T07:39:37.429Z","log.logger":"template_loader","log.origin":{"file.name":"template/load.go","file.line":127},"message":"Template with name \"filebeat-8.4.0\" loaded.","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-08-30T07:39:37.432Z","log.logger":"template_loader","log.origin":{"file.name":"template/load.go","file.line":184},"message":"Try loading data stream filebeat-8.4.0 to Elasticsearch","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-08-30T07:39:37.953Z","log.logger":"template_loader","log.origin":{"file.name":"template/load.go","file.line":150},"message":"Data stream with name \"filebeat-8.4.0\" loaded.","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-08-30T07:39:37.954Z","log.logger":"index-management","log.origin":{"file.name":"idxmgmt/std.go","file.line":267},"message":"Loaded index template.","service.name":"filebeat","ecs.version":"1.6.0"}
Index setup finished.
Loading dashboards (Kibana must be running and reachable)
{"log.level":"info","@timestamp":"2022-08-30T07:39:37.954Z","log.logger":"kibana","log.origin":{"file.name":"kibana/client.go","file.line":179},"message":"Kibana url: http://localhost:5601","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"error","@timestamp":"2022-08-30T07:39:37.955Z","log.origin":{"file.name":"instance/beat.go","file.line":1051},"message":"Exiting: error connecting to Kibana: fail to get the Kibana version: HTTP GET request to http://localhost:5601/api/status fails: fail to execute the HTTP GET request: Get \"http://localhost:5601/api/status\": dial tcp 127.0.0.1:5601: connect: connection refused. Response: ","service.name":"filebeat","ecs.version":"1.6.0"}
Exiting: error connecting to Kibana: fail to get the Kibana version: HTTP GET request to http://localhost:5601/api/status fails: fail to execute the HTTP GET request: Get "http://localhost:5601/api/status": dial tcp 127.0.0.1:5601: connect: connection refused. Response:

There is no /data directory in /etc/filebeat

Read the Docs as I suggested

data The location for persistent data files. /var/lib/filebeat

Clean out that directory

Also you did not set the Kibana configuration correct in your filebeat.yml

you need to set this

setup.kibana:

  # Kibana Host
  # Scheme and port can be left out and will be set to the default (http and 5601)
  # In case you specify and additional path, the scheme is required: http://localhost:5601/path
  # IPv6 addresses should always be defined as: https://[2001:db8::1]:5601
  host: "localhost:5601" <!----- SET THIS

Also where are the logs from when you actually run filebeat.

journalctl -u filebeat.service

Thank you very much, after cleaning that directory filebeat shipped squid logs with right parsing.

But:

  1. Why lots of bunch of empty and unnecessary fields are also shipped? aws.cloud, azure.activitylogs etc?
    I want only squid logs to be shipped.
  2. The real log size of squid ~25mb. But in Data Streams it shows 66mb.

What maybe the problem?
Again, thank you for your help!

Also you did not set the Kibana configuration correct in your filebeat.yml

Is it required?
I want to create my own dashboards.