# Squid module does not parse logs

**URL:** <https://discuss.elastic.co/t/squid-module-does-not-parse-logs/313088>\
**Category:** Beats\
**Tags:** filebeat\
**Created:** [August 28, 2022, 10:43am UTC](https://discuss.elastic.co/t/squid-module-does-not-parse-logs/313088 "2022-08-28T10:43:02Z")\
**Posts on this page:** 20\
**Page:** 1

<div class="post-metadata">

**Author:** ![it\_dev](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/it_dev/32/110019_2.png) [@it\_dev](https://discuss.elastic.co/u/it_dev)\
**Post date:** [August 28, 2022, 10:43am UTC](https://discuss.elastic.co/t/squid-module-does-not-parse-logs/313088/1 "2022-08-28T10:43:02Z")

</div>

1. I have Linux syslog server.
2. My squid server (separate from Linux syslog) sends all its access.log files to Linux syslog server into /var/log/squid/\*.log
3. There, on my Linux syslog server I have installed filebeat 8.4.
4. I have enabled Filebeat Squid module, showed its input as a file and var.paths as a /var/log/squid/\*.log
5. Elastic Stack is working on a separate server, ok.
6. When I test filebeat test config, filebeat test output - OK, no problem.
7. When I start filebeat (service filebeat start) it sends all logs in /var/log/squid folder to Elastic search. I can see it in Index Management.

If I discover my filebeat logs or create graph in Kibana —-\> message is not parsed. It comes as a filestream. I mean that all Squid logs come in message format.  
There is a special Squid module on my Linux syslog server enabled. Why logs come not parsed? It seems like Squid module has no effect.

What maybe the problem?

---

<div class="post-metadata">

**Author:** ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)\
**Post date:** [August 28, 2022, 3:47pm UTC](https://discuss.elastic.co/t/squid-module-does-not-parse-logs/313088/2 "2022-08-28T15:47:25Z")

</div>

Hi @it_dev

Did you carefully follow all the steps in the Quick Start Guide?  
And run the `filebeat setup -e` command? This is key... as shown [here](https://www.elastic.co/guide/en/beats/filebeat/master/filebeat-installation-configuration.html#setup-assets) this setups up all the needed templates and ingest pipelines that parse the data.

A) If you did not do all that clean up everything and start over.

B) If you did all those things it is possible you squid log are non-standard and that is why they are not getting parsed...

If B can you provide 5 or 6 lines from your squid logs perhaps we can take a look.

---

<div class="post-metadata">

**Author:** ![it\_dev](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/it_dev/32/110019_2.png) [@it\_dev](https://discuss.elastic.co/u/it_dev)\
**Post date:** [August 29, 2022, 4:42am UTC](https://discuss.elastic.co/t/squid-module-does-not-parse-logs/313088/3 "2022-08-29T04:42:43Z")

</div>

Hello @stephenb !

Thank you for your reply.

A) I did all that.  
B) Of course, I can provide logs:

```auto
1661367615.883 188607 10.1.32.17 TCP_TUNNEL/200 412492 CONNECT www.fotospor.com:443 - HIER_DIRECT/31.186.15.10 --
1661367621.886 186711 10.1.32.17 TCP_TUNNEL/200 2345 CONNECT cdnjs.cloudflare.com:443 - HIER_DIRECT/188.114.98.192 -
1661367622.019 65211 10.1.32.23 NONE/503 0 CONNECT mtalk.google.com:5228 - HIER_NONE/- -
1661367626.407 496760 10.1.32.50 TCP_TUNNEL/200 521 CONNECT dc1.ksn.kaspersky-labs.com:443 - HIER_DIRECT/62.128.100.45 -
1661367627.563 105758 10.1.32.18 TCP_TUNNEL/200 846 CONNECT emea1.views.cp.thomsonreuters.com:443 - HIER_DIRECT/159.220.1.49

```

Hope for your help.

---

<div class="post-metadata">

**Author:** ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)\
**Post date:** [August 29, 2022, 5:25am UTC](https://discuss.elastic.co/t/squid-module-does-not-parse-logs/313088/4 "2022-08-29T05:25:06Z")

</div>

> [@it\_dev](#):
>
> ```auto
> 1661367615.883 188607 10.1.32.17 TCP_TUNNEL/200 412492 CONNECT www.fotospor.com:443 - HIER_DIRECT/31.186.15.10 --
> 1661367621.886 186711 10.1.32.17 TCP_TUNNEL/200 2345 CONNECT cdnjs.cloudflare.com:443 - HIER_DIRECT/188.114.98.192 -
> 1661367622.019 65211 10.1.32.23 NONE/503 0 CONNECT mtalk.google.com:5228 - HIER_NONE/- -
> 1661367626.407 496760 10.1.32.50 TCP_TUNNEL/200 521 CONNECT dc1.ksn.kaspersky-labs.com:443 - HIER_DIRECT/62.128.100.45 -
> 1661367627.563 105758 10.1.32.18 TCP_TUNNEL/200 846 CONNECT emea1.views.cp.thomsonreuters.com:443 - HIER_DIRECT/159.220.1.49
> 
> ```

I have 8.4.0 Elastic Stack

1. I simply put those lines above in a file

2. I configured the modules.d/squid.yml

```auto
- module: squid
  log:
    enabled: true

    # Set which input to use between udp (default), tcp or file.
    var.input: file
    # var.syslog_host: localhost
    # var.syslog_port: 9520

    # Set paths for the log files when file input is used.
    var.paths: ["/Users/sbrown/workspace/sample-data/discuss/squid.log"]

```

I ran

1. `filebeat setup -e`_ **\<\<\<\<Please confirm you ran this** _  
then I ran

2. `filebeat -e`

And here are my results all the logs are parse.

So can you see they are properly parsed

```auto
GET filebeat-*/_search

{
  "took": 0,
  "timed_out": false,
  "_shards": {
    "total": 1,
    "successful": 1,
    "skipped": 0,
    "failed": 0
  },
  "hits": {
    "total": {
      "value": 5,
      "relation": "eq"
    },
    "max_score": 1,
    "hits": [
      {
        "_index": ".ds-filebeat-8.4.0-2022.08.29-000001",
        "_id": "w24F6IIBVm7wjCsj5KL6",
        "_score": 1,
        "_source": {
          "container": {
            "id": "discuss"
          },
          "rsa": {
            "internal": {
              "messageid": "CONNECT",
              "hcode": "HIER_DIRECT"
            },
            "web": {
              "alias_host": "www.fotospor.com"
            },
            "investigations": {
              "ec_subject": "NetworkComm",
              "ec_theme": "ALM"
            },
            "time": {
              "event_time_str": "1661367615",
              "duration_time": 188607,
              "event_time": "2022-08-24T19:00:15.000Z"
            },
            "misc": {
              "content_type": "--",
              "action": [
                "CONNECT",
                "TCP_TUNNEL"
              ],
              "result_code": "200"
            },
            "network": {
              "domain": "www.fotospor.com"
            }
          },
          "server": {
            "registered_domain": "fotospor.com",
            "top_level_domain": "com",
            "domain": "www.fotospor.com",
            "subdomain": "www"
          },
          "agent": {
            "name": "hyperion",
            "id": "f0dbfc56-c491-4f5e-920b-561fe2e54b2c",
            "type": "filebeat",
            "ephemeral_id": "b0b5d6a2-5f96-4a4b-9242-2b88d7e65fac",
            "version": "8.4.0"
          },
          "log": {
            "file": {
              "path": "/Users/sbrown/workspace/sample-data/discuss/squid.log"
            },
            "offset": 0
          },
          "destination": {
            "geo": {
              "continent_name": "Asia",
              "country_iso_code": "TR",
              "country_name": "Turkey",
              "location": {
                "lon": 28.9948,
                "lat": 41.0214
              }
            },
            "as": {
              "number": 197720,
              "organization": {
                "name": "TURKTICARET.NET YAZILIM HIZMETLERI SAN. ve TIC. A.S."
              }
            },
            "ip": "31.186.15.10"
          },
          "source": {
            "bytes": 412492,
            "ip": "10.1.32.17"
          },
          "fileset": {
            "name": "log"
          },
          "url": {
            "registered_domain": "fotospor.com",
            "original": "www.fotospor.com:443",
            "top_level_domain": "com",
            "domain": "www.fotospor.com",
            "subdomain": "www"
          },
          "tags": [
            "squid.log",
            "forwarded"
          ],
          "observer": {
            "product": "Proxy",
            "vendor": "Squid",
            "type": "Proxies"
          },
          "input": {
            "type": "log"
          },
          "@timestamp": "2022-08-24T19:00:15.000Z",
          "ecs": {
            "version": "1.12.0"
          },
          "related": {
            "hosts": [
              "www.fotospor.com"
            ],
            "ip": [
              "10.1.32.17",
              "31.186.15.10"
            ],
            "user": [
              "-"
            ]
          },
          "service": {
            "type": "squid"
          },
          "event": {
            "ingested": "2022-08-29T05:14:36.633267333Z",
            "original": "1661367615.883 188607 10.1.32.17 TCP_TUNNEL/200 412492 CONNECT www.fotospor.com:443 - HIER_DIRECT/31.186.15.10 --",
            "code": "CONNECT",
            "module": "squid",
            "action": "TCP_TUNNEL",
            "dataset": "squid.log"
          },
          "user": {
            "name": "-"
          }
        }
      },
      {
        "_index": ".ds-filebeat-8.4.0-2022.08.29-000001",
        "_id": "xG4F6IIBVm7wjCsj5KL6",
        "_score": 1,
        "_source": {
          "container": {
            "id": "discuss"
          },
          "rsa": {
            "internal": {
              "messageid": "CONNECT",
              "hcode": "HIER_DIRECT"
            },
            "web": {
              "alias_host": "cdnjs.cloudflare.com"
            },
            "investigations": {
              "ec_subject": "NetworkComm",
              "ec_theme": "ALM"
            },
            "time": {
              "event_time_str": "1661367621",
              "duration_time": 186711,
              "event_time": "2022-08-24T19:00:21.000Z"
            },
            "misc": {
              "content_type": "-",
              "action": [
                "CONNECT",
                "TCP_TUNNEL"
              ],
              "result_code": "200"
            },
            "network": {
              "domain": "cdnjs.cloudflare.com"
            }
          },
          "server": {
            "registered_domain": "cloudflare.com",
            "top_level_domain": "com",
            "domain": "cdnjs.cloudflare.com",
            "subdomain": "cdnjs"
          },
          "agent": {
            "name": "hyperion",
            "id": "f0dbfc56-c491-4f5e-920b-561fe2e54b2c",
            "type": "filebeat",
            "ephemeral_id": "b0b5d6a2-5f96-4a4b-9242-2b88d7e65fac",
            "version": "8.4.0"
          },
          "log": {
            "file": {
              "path": "/Users/sbrown/workspace/sample-data/discuss/squid.log"
            },
            "offset": 114
          },
          "destination": {
            "geo": {
              "continent_name": "Europe",
              "region_iso_code": "NL-NH",
              "city_name": "Amsterdam",
              "country_iso_code": "NL",
              "country_name": "Netherlands",
              "region_name": "North Holland",
              "location": {
                "lon": 4.8883,
                "lat": 52.3716
              }
            },
            "as": {
              "number": 13335,
              "organization": {
                "name": "CLOUDFLARENET"
              }
            },
            "ip": "188.114.98.192"
          },
          "source": {
            "bytes": 2345,
            "ip": "10.1.32.17"
          },
          "fileset": {
            "name": "log"
          },
          "url": {
            "registered_domain": "cloudflare.com",
            "original": "cdnjs.cloudflare.com:443",
            "top_level_domain": "com",
            "domain": "cdnjs.cloudflare.com",
            "subdomain": "cdnjs"
          },
          "tags": [
            "squid.log",
            "forwarded"
          ],
          "input": {
            "type": "log"
          },
          "observer": {
            "product": "Proxy",
            "vendor": "Squid",
            "type": "Proxies"
          },
          "@timestamp": "2022-08-24T19:00:21.000Z",
          "ecs": {
            "version": "1.12.0"
          },
          "related": {
            "hosts": [
              "cdnjs.cloudflare.com"
            ],
            "ip": [
              "10.1.32.17",
              "188.114.98.192"
            ],
            "user": [
              "-"
            ]
          },
          "service": {
            "type": "squid"
          },
          "event": {
            "ingested": "2022-08-29T05:14:36.660378441Z",
            "original": "1661367621.886 186711 10.1.32.17 TCP_TUNNEL/200 2345 CONNECT cdnjs.cloudflare.com:443 - HIER_DIRECT/188.114.98.192 -",
            "code": "CONNECT",
            "module": "squid",
            "action": "TCP_TUNNEL",
            "dataset": "squid.log"
          },
          "user": {
            "name": "-"
          }
        }
      },
      {
        "_index": ".ds-filebeat-8.4.0-2022.08.29-000001",
        "_id": "xW4F6IIBVm7wjCsj5KL6",
        "_score": 1,
        "_source": {
          "container": {
            "id": "discuss"
          },
          "rsa": {
            "internal": {
              "messageid": "CONNECT",
              "hcode": "HIER_NONE"
            },
            "web": {
              "alias_host": "mtalk.google.com"
            },
            "investigations": {
              "ec_subject": "NetworkComm",
              "ec_theme": "ALM"
            },
            "time": {
              "event_time_str": "1661367622",
              "duration_time": 65211,
              "event_time": "2022-08-24T19:00:22.000Z"
            },
            "misc": {
              "content_type": "-",
              "action": [
                "NONE",
                "CONNECT"
              ],
              "result_code": "503"
            },
            "network": {
              "domain": "mtalk.google.com"
            }
          },
          "server": {
            "registered_domain": "google.com",
            "top_level_domain": "com",
            "domain": "mtalk.google.com",
            "subdomain": "mtalk"
          },
          "agent": {
            "name": "hyperion",
            "id": "f0dbfc56-c491-4f5e-920b-561fe2e54b2c",
            "type": "filebeat",
            "ephemeral_id": "b0b5d6a2-5f96-4a4b-9242-2b88d7e65fac",
            "version": "8.4.0"
          },
          "log": {
            "file": {
              "path": "/Users/sbrown/workspace/sample-data/discuss/squid.log"
            },
            "offset": 231
          },
          "source": {
            "bytes": 0,
            "ip": "10.1.32.23"
          },
          "fileset": {
            "name": "log"
          },
          "url": {
            "registered_domain": "google.com",
            "original": "mtalk.google.com:5228",
            "top_level_domain": "com",
            "domain": "mtalk.google.com",
            "subdomain": "mtalk"
          },
          "tags": [
            "squid.log",
            "forwarded"
          ],
          "input": {
            "type": "log"
          },
          "observer": {
            "product": "Proxy",
            "vendor": "Squid",
            "type": "Proxies"
          },
          "@timestamp": "2022-08-24T19:00:22.000Z",
          "related": {
            "hosts": [
              "mtalk.google.com"
            ],
            "ip": [
              "10.1.32.23"
            ],
            "user": [
              "-"
            ]
          },
          "ecs": {
            "version": "1.12.0"
          },
          "service": {
            "type": "squid"
          },
          "event": {
            "ingested": "2022-08-29T05:14:36.663594492Z",
            "original": "1661367622.019 65211 10.1.32.23 NONE/503 0 CONNECT mtalk.google.com:5228 - HIER_NONE/- -",
            "code": "CONNECT",
            "module": "squid",
            "action": "NONE",
            "dataset": "squid.log"
          },
          "user": {
            "name": "-"
          }
        }
      },
      {
        "_index": ".ds-filebeat-8.4.0-2022.08.29-000001",
        "_id": "xm4F6IIBVm7wjCsj5KL6",
        "_score": 1,
        "_source": {
          "container": {
            "id": "discuss"
          },
          "rsa": {
            "internal": {
              "messageid": "CONNECT",
              "hcode": "HIER_DIRECT"
            },
            "web": {
              "alias_host": "dc1.ksn.kaspersky-labs.com"
            },
            "investigations": {
              "ec_subject": "NetworkComm",
              "ec_theme": "ALM"
            },
            "time": {
              "event_time_str": "1661367626",
              "duration_time": 496760,
              "event_time": "2022-08-24T19:00:26.000Z"
            },
            "network": {
              "domain": "dc1.ksn.kaspersky-labs.com"
            },
            "misc": {
              "content_type": "-",
              "action": [
                "TCP_TUNNEL",
                "CONNECT"
              ],
              "result_code": "200"
            }
          },
          "server": {
            "registered_domain": "kaspersky-labs.com",
            "top_level_domain": "com",
            "domain": "dc1.ksn.kaspersky-labs.com",
            "subdomain": "dc1.ksn"
          },
          "agent": {
            "name": "hyperion",
            "id": "f0dbfc56-c491-4f5e-920b-561fe2e54b2c",
            "type": "filebeat",
            "ephemeral_id": "b0b5d6a2-5f96-4a4b-9242-2b88d7e65fac",
            "version": "8.4.0"
          },
          "log": {
            "file": {
              "path": "/Users/sbrown/workspace/sample-data/discuss/squid.log"
            },
            "offset": 321
          },
          "destination": {
            "geo": {
              "continent_name": "Europe",
              "region_iso_code": "RU-MOW",
              "city_name": "Moscow",
              "country_iso_code": "RU",
              "country_name": "Russia",
              "region_name": "Moscow",
              "location": {
                "lon": 37.6171,
                "lat": 55.7483
              }
            },
            "as": {
              "number": 3327,
              "organization": {
                "name": "CITIC Telecom CPC Netherlands B.V."
              }
            },
            "ip": "62.128.100.45"
          },
          "source": {
            "bytes": 521,
            "ip": "10.1.32.50"
          },
          "fileset": {
            "name": "log"
          },
          "url": {
            "registered_domain": "kaspersky-labs.com",
            "original": "dc1.ksn.kaspersky-labs.com:443",
            "top_level_domain": "com",
            "domain": "dc1.ksn.kaspersky-labs.com",
            "subdomain": "dc1.ksn"
          },
          "tags": [
            "squid.log",
            "forwarded"
          ],
          "input": {
            "type": "log"
          },
          "observer": {
            "product": "Proxy",
            "vendor": "Squid",
            "type": "Proxies"
          },
          "@timestamp": "2022-08-24T19:00:26.000Z",
          "related": {
            "hosts": [
              "dc1.ksn.kaspersky-labs.com"
            ],
            "ip": [
              "10.1.32.50",
              "62.128.100.45"
            ],
            "user": [
              "-"
            ]
          },
          "ecs": {
            "version": "1.12.0"
          },
          "service": {
            "type": "squid"
          },
          "event": {
            "ingested": "2022-08-29T05:14:36.664195302Z",
            "original": "1661367626.407 496760 10.1.32.50 TCP_TUNNEL/200 521 CONNECT dc1.ksn.kaspersky-labs.com:443 - HIER_DIRECT/62.128.100.45 -",
            "code": "CONNECT",
            "module": "squid",
            "action": "TCP_TUNNEL",
            "dataset": "squid.log"
          },
          "user": {
            "name": "-"
          }
        }
      },
      {
        "_index": ".ds-filebeat-8.4.0-2022.08.29-000001",
        "_id": "x24F6IIBVm7wjCsj5KL6",
        "_score": 1,
        "_source": {
          "container": {
            "id": "discuss"
          },
          "rsa": {
            "internal": {
              "messageid": "CONNECT"
            }
          },
          "agent": {
            "name": "hyperion",
            "id": "f0dbfc56-c491-4f5e-920b-561fe2e54b2c",
            "ephemeral_id": "b0b5d6a2-5f96-4a4b-9242-2b88d7e65fac",
            "type": "filebeat",
            "version": "8.4.0"
          },
          "log": {
            "file": {
              "path": "/Users/sbrown/workspace/sample-data/discuss/squid.log"
            },
            "offset": 442,
            "flags": [
              "dissect_parsing_error"
            ]
          },
          "fileset": {
            "name": "log"
          },
          "tags": [
            "squid.log",
            "forwarded"
          ],
          "input": {
            "type": "log"
          },
          "observer": {
            "product": "Proxy",
            "vendor": "Squid",
            "type": "Proxies"
          },
          "@timestamp": "2022-08-29T05:14:35.586Z",
          "ecs": {
            "version": "1.12.0"
          },
          "service": {
            "type": "squid"
          },
          "event": {
            "ingested": "2022-08-29T05:14:36.666485929Z",
            "original": "1661367627.563 105758 10.1.32.18 TCP_TUNNEL/200 846 CONNECT emea1.views.cp.thomsonreuters.com:443 - HIER_DIRECT/159.220.1.49",
            "code": "CONNECT",
            "module": "squid",
            "dataset": "squid.log"
          }
        }
      }
    ]
  }
}

```

 ![Screen Shot 2022-08-28 at 10.21.37 PM](https://us1.discourse-cdn.com/elastic/original/3X/5/a/5a4eaeda2efa96b0f70f1a673390693afda6b68b.jpeg)

What happens when you run from the Kibana Dev Tools

`GET _ingest/pipeline/filebeat-8.4.0-squid-log-pipeline`

You are missing a step somewhere.

---

<div class="post-metadata">

**Author:** ![it\_dev](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/it_dev/32/110019_2.png) [@it\_dev](https://discuss.elastic.co/u/it_dev)\
**Post date:** [August 29, 2022, 6:06am UTC](https://discuss.elastic.co/t/squid-module-does-not-parse-logs/313088/5 "2022-08-29T06:06:19Z")

</div>

`filebeat setup -e` - yes I do this.

Still logs are not parsed.  
And when I do `filebeat -e`, there is no output in console like yours.  
Instead, there are many notifications.  
One of them is this:

```auto
{"log.level":"warn","@timestamp":"2022-08-29T08:04:31.717Z","log.logger":"cfgwarn","log.origin":{"file.name":"log/input.go","file.line":90},"message":"DEPRECATED: Log input. Use Filestream input instead.","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-08-29T08:04:31.718Z","log.logger":"input","log.origin":{"file.name":"log/input.go","file.line":172},"message":"Configured paths: [/var/log/squid/*.log]","service.name":"filebeat","input_id":"93519840-1d95-464a-a208-67100856a47d","ec

```

---

<div class="post-metadata">

**Author:** ![it\_dev](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/it_dev/32/110019_2.png) [@it\_dev](https://discuss.elastic.co/u/it_dev)\
**Post date:** [August 29, 2022, 6:10am UTC](https://discuss.elastic.co/t/squid-module-does-not-parse-logs/313088/6 "2022-08-29T06:10:08Z")

</div>

My Elastic Stack is 8.3.3  
My Filebeat version is 8.4.0

Can this be the reason?

---

<div class="post-metadata">

**Author:** ![it\_dev](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/it_dev/32/110019_2.png) [@it\_dev](https://discuss.elastic.co/u/it_dev)\
**Post date:** [August 29, 2022, 6:28am UTC](https://discuss.elastic.co/t/squid-module-does-not-parse-logs/313088/7 "2022-08-29T06:28:38Z")

</div>

I have updated my ELK to 8.4.0, problem still exists.

---

<div class="post-metadata">

**Author:** ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)\
**Post date:** [August 29, 2022, 6:29am UTC](https://discuss.elastic.co/t/squid-module-does-not-parse-logs/313088/8 "2022-08-29T06:29:31Z")

</div>

> [@it\_dev](#):
>
> My Elastic Stack is 8.3.3  
> My Filebeat version is 8.4.0

It is not a good idea to you a beat newer than the stack.

> [@it\_dev](#):
>
> there is no output in console like yours.

That is from Kibana -\> Dev Tools

`GET filebeat-*/_search`

Please post your entire  
filebeat.yml  
And  
modules.d/squid.yml

You have something miss-configured.

Did you enable the squid module

`filebeat modules enable squid`

---

<div class="post-metadata">

**Author:** ![it\_dev](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/it_dev/32/110019_2.png) [@it\_dev](https://discuss.elastic.co/u/it_dev)\
**Post date:** [August 29, 2022, 6:38am UTC](https://discuss.elastic.co/t/squid-module-does-not-parse-logs/313088/9 "2022-08-29T06:38:34Z")

</div>

`GET filebeat-*/_search:`

> {  
> "took": 1,  
> "timed\_out": false,  
> "\_shards": {  
> "total": 1,  
> "successful": 1,  
> "skipped": 0,  
> "failed": 0  
> },  
> "hits": {  
> "total": {  
> "value": 0,  
> "relation": "eq"  
> },  
> "max\_score": null,  
> "hits":   
> }  
> }

---

<div class="post-metadata">

**Author:** ![it\_dev](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/it_dev/32/110019_2.png) [@it\_dev](https://discuss.elastic.co/u/it_dev)\
**Post date:** [August 29, 2022, 6:39am UTC](https://discuss.elastic.co/t/squid-module-does-not-parse-logs/313088/10 "2022-08-29T06:39:20Z")

</div>

filebeat.yml:

```auto
###################### Filebeat Configuration Example #########################

# This file is an example configuration file highlighting only the most common
# options. The filebeat.reference.yml file from the same directory contains all the
# supported options with more comments. You can use it as a reference.
#
# You can find the full configuration reference here:
# https://www.elastic.co/guide/en/beats/filebeat/index.html

# For more available modules and options, please see the filebeat.reference.yml sample
# configuration file.

# ============================== Filebeat inputs ===============================

filebeat.inputs:

# Each - is an input. Most options can be set at the input level, so
# you can use different inputs for various configurations.
# Below are the input specific configurations.

# filestream is an input for collecting log messages from files.
- type: filestream

  # Unique ID among all inputs, an ID is required.
  id: my-filestream-id

  # Change to true to enable this input configuration.
  enabled: false

  # Paths that should be crawled and fetched. Glob based paths.
  paths:
    - /var/log/*.log
    #- c:\programdata\elasticsearch\logs\*
  # Exclude lines. A list of regular expressions to match. It drops the lines that are
  # matching any regular expression from the list.
  # Line filtering happens after the parsers pipeline. If you would like to filter lines
  # before parsers, use include_message parser.
  #exclude_lines: ['^DBG']

  # Include lines. A list of regular expressions to match. It exports the lines that are
  # matching any regular expression from the list.
  # Line filtering happens after the parsers pipeline. If you would like to filter lines
  # before parsers, use include_message parser.
  #include_lines: ['^ERR', '^WARN']

  # Exclude files. A list of regular expressions to match. Filebeat drops the files that
  # are matching any regular expression from the list. By default, no files are dropped.
  #prospector.scanner.exclude_files: ['.gz$']

  # Optional additional fields. These fields can be freely picked
  # to add additional information to the crawled log files for filtering
  #fields:
  # level: debug
  # review: 1

# ============================== Filebeat modules ==============================

filebeat.config.modules:
  # Glob pattern for configuration loading
  path: ${path.config}/modules.d/*.yml

  # Set to true to enable config reloading
  reload.enabled: true

  # Period on which files under path should be checked for changes
  #reload.period: 10s

# ======================= Elasticsearch template setting =======================

setup.template.settings:
  index.number_of_shards: 1
  #index.codec: best_compression
  #_source.enabled: false

# ================================== General ===================================

# The name of the shipper that publishes the network data. It can be used to group
# all the transactions sent by a single shipper in the web interface.
#name:

# The tags of the shipper are included in their own field with each
# transaction published.
#tags: ["service-X", "web-tier"]

# Optional fields that you can specify to add additional information to the
# output.
#fields:
# env: staging

# ================================= Dashboards =================================
# These settings control loading the sample dashboards to the Kibana index. Loading
# the dashboards is disabled by default and can be enabled either by setting the
# options here or by using the `setup` command.
#setup.dashboards.enabled: false

# The URL from where to download the dashboards archive. By default this URL
# has a value which is computed based on the Beat name and version. For released
# versions, this URL points to the dashboard archive on the artifacts.elastic.co
# website.
#setup.dashboards.url:

# =================================== Kibana ===================================

# Starting with Beats version 6.0.0, the dashboards are loaded via the Kibana API.
# This requires a Kibana endpoint configuration.
setup.kibana:

  # Kibana Host
  # Scheme and port can be left out and will be set to the default (http and 5601)
  # In case you specify and additional path, the scheme is required: http://localhost:5601/path
  # IPv6 addresses should always be defined as: https://[2001:db8::1]:5601
  #host: "localhost:5601"
  # Kibana Space ID
  # ID of the Kibana Space into which the dashboards should be loaded. By default,
  # the Default Space will be used.
  #space.id:

# =============================== Elastic Cloud ================================

# These settings simplify using Filebeat with the Elastic Cloud (https://cloud.elastic.co/).

# The cloud.id setting overwrites the `output.elasticsearch.hosts` and
# `setup.kibana.host` options.
# You can find the `cloud.id` in the Elastic Cloud web UI.
#cloud.id:

# The cloud.auth setting overwrites the `output.elasticsearch.username` and
# `output.elasticsearch.password` settings. The format is `<user>:<pass>`.
#cloud.auth:

# ================================== Outputs ===================================

# Configure what output to use when sending the data collected by the beat.

# ---------------------------- Elasticsearch Output ----------------------------
output.elasticsearch:
  # Array of hosts to connect to.
#A hosts: ["localhost:9200"]
    hosts: ["https://10.1.32.44:9200"]
    username: "test-user"
    password: "test-pass"
    ssl:
      enabled: true
      ca_trusted_fingerprint: "69bd0555b28fbc14527506352a7374ae3697b219456f900542dec288bc870b3c"
  # Protocol - either `http` (default) or `https`.
  #protocol: "https"

  # Authentication credentials - either API key or username/password.
  #api_key: "id:api_key"
  #username: "elastic"
  #password: "changeme"
#output.elasticsearch.allow_older_versions: true
# ------------------------------ Logstash Output -------------------------------
#output.logstash:
  # The Logstash hosts
  #hosts: ["localhost:5044"]

  # Optional SSL. By default is off.
  # List of root certificates for HTTPS server verifications
  #ssl.certificate_authorities: ["/etc/pki/root/ca.pem"]

  # Certificate for SSL client authentication
  #ssl.certificate: "/etc/pki/client/cert.pem"

  # Client Certificate Key
  #ssl.key: "/etc/pki/client/cert.key"

# ================================= Processors =================================
processors:
  - add_host_metadata:
      when.not.contains.tags: forwarded
  - add_cloud_metadata: ~
  - add_docker_metadata: ~
  - add_kubernetes_metadata: ~

# ================================== Logging ===================================

# Sets log level. The default log level is info.
# Available log levels are: error, warning, info, debug
#logging.level: debug

# At debug level, you can selectively enable logging only for some components.
# To enable all selectors use ["*"]. Examples of other selectors are "beat",
# "publisher", "service".
#logging.selectors: ["*"]

# ============================= X-Pack Monitoring ==============================
# Filebeat can export internal metrics to a central Elasticsearch monitoring
# cluster. This requires xpack monitoring to be enabled in Elasticsearch. The
# reporting is disabled by default.

# Set to true to enable the monitoring reporter.
#monitoring.enabled: false

# Sets the UUID of the Elasticsearch cluster under which monitoring data for this
# Filebeat instance will appear in the Stack Monitoring UI. If output.elasticsearch
# is enabled, the UUID is derived from the Elasticsearch cluster referenced by output.elasticsearch.
#monitoring.cluster_uuid:

# Uncomment to send the metrics to Elasticsearch. Most settings from the
# Elasticsearch output are accepted here as well.
# Note that the settings should point to your Elasticsearch *monitoring* cluster.
# Any setting that is not set is automatically inherited from the Elasticsearch
# output configuration, so if you have the Elasticsearch output configured such
# that it is pointing to your Elasticsearch monitoring cluster, you can simply
# uncomment the following line.
#monitoring.elasticsearch:

# ============================== Instrumentation ===============================

# Instrumentation support for the filebeat.
#instrumentation:
    # Set to true to enable instrumentation of filebeat.
    #enabled: false

    # Environment in which filebeat is running on (eg: staging, production, etc.)
    #environment: ""

    # APM Server hosts to report instrumentation results to.
    #hosts:
    # - http://localhost:8200

    # API Key for the APM Server(s).
    # If api_key is set then secret_token will be ignored.
    #api_key:

    # Secret token for the APM Server(s).
    #secret_token:

# ================================= Migration ==================================

# This allows to enable 6.7 migration aliases
#migration.6_to_7.enabled: true

```

---

<div class="post-metadata">

**Author:** ![it\_dev](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/it_dev/32/110019_2.png) [@it\_dev](https://discuss.elastic.co/u/it_dev)\
**Post date:** [August 29, 2022, 6:40am UTC](https://discuss.elastic.co/t/squid-module-does-not-parse-logs/313088/11 "2022-08-29T06:40:39Z")

</div>

modules.d/squid.yml:

```auto
# Module: squid
# Docs: https://www.elastic.co/guide/en/beats/filebeat/main/filebeat-module-squid.html

- module: squid
  log:
    enabled: true

    # Set which input to use between udp (default), tcp or file.
    # var.input: udp
    var.input: file
    # var.syslog_host: localhost
    # var.syslog_port: 9520
    # Set paths for the log files when file input is used.
    var.paths: ["/var/log/squid/access.log"]

    # Toggle output of non-ECS fields (default true).
    # var.rsa_fields: true

    # Set custom timezone offset.
    # "local" (default) for system timezone.
    # "+02:00" for GMT+02:00
    # var.tz_offset: local

```

---

<div class="post-metadata">

**Author:** ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)\
**Post date:** [August 29, 2022, 6:45am UTC](https://discuss.elastic.co/t/squid-module-does-not-parse-logs/313088/12 "2022-08-29T06:45:09Z")

</div>

Ok the logs are not loading at all... Your search shows that.

I suspect filebeat already read that log file once.. it will not re-read it again.

You need to clean out the filebeat data directory and try again. Look at the directory layout for the install you did clean the data directory and try again.

---

<div class="post-metadata">

**Author:** ![it\_dev](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/it_dev/32/110019_2.png) [@it\_dev](https://discuss.elastic.co/u/it_dev)\
**Post date:** [August 29, 2022, 7:17am UTC](https://discuss.elastic.co/t/squid-module-does-not-parse-logs/313088/13 "2022-08-29T07:17:59Z")

</div>

Tried to remove filebeat and install it again.  
It says: `Version 8.4.0 of Filebeat has not yet been released.`

So I installed Filebeat 7.17.6.

Same thing, logs are not parsed and not fully sent.

---

<div class="post-metadata">

**Author:** ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)\
**Post date:** [August 29, 2022, 3:16pm UTC](https://discuss.elastic.co/t/squid-module-does-not-parse-logs/313088/14 "2022-08-29T15:16:00Z")

</div>

In general your configs look pretty good

There is something basic wrong.. what does the filebeat logs show...

Did you clean up the data directory? and run again?

`Version 8.4.0 of Filebeat has not yet been released.`  
Not sure where you saw the that message you are not giving context? 8.4.0 filebeat is certainly released. That could be a bug... so trying 7.17.6 is fine.

Please provide the filebeat startup logs... after you clean up everything... and run it... there is something basic wrong.. .this took me 5 mins end to end

**Please Show the filebeat logs.**

Please show messages look like inside elasticsearch.

---

<div class="post-metadata">

**Author:** ![it\_dev](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/it_dev/32/110019_2.png) [@it\_dev](https://discuss.elastic.co/u/it_dev)\
**Post date:** [August 30, 2022, 5:40am UTC](https://discuss.elastic.co/t/squid-module-does-not-parse-logs/313088/15 "2022-08-30T05:40:42Z")

</div>

filebeat setup -e logs:

```auto
{"log.level":"info","@timestamp":"2022-08-30T07:39:31.387Z","log.origin":{"file.name":"instance/beat.go","file.line":702},"message":"Home path: [/usr/share/filebeat] Config path: [/etc/filebeat] Data path: [/var/lib/filebeat] Logs path: [/var/log/filebeat]","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-08-30T07:39:31.387Z","log.origin":{"file.name":"instance/beat.go","file.line":710},"message":"Beat ID: 18f9fb7b-d419-4ad5-b357-158a87350fbf","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"warn","@timestamp":"2022-08-30T07:39:34.391Z","log.logger":"add_cloud_metadata","log.origin":{"file.name":"add_cloud_metadata/provider_aws_ec2.go","file.line":81},"message":"read token request for getting IMDSv2 token returns empty: Put \"http://169.254.169.254/latest/api/token\": context deadline exceeded (Client.Timeout exceeded while awaiting headers). No token in the metadata request will be used.","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-08-30T07:39:34.392Z","log.logger":"beat","log.origin":{"file.name":"instance/beat.go","file.line":1076},"message":"Beat info","service.name":"filebeat","system_info":{"beat":{"path":{"config":"/etc/filebeat","data":"/var/lib/filebeat","home":"/usr/share/filebeat","logs":"/var/log/filebeat"},"type":"filebeat","uuid":"18f9fb7b-d419-4ad5-b357-158a87350fbf"},"ecs.version":"1.6.0"}}
{"log.level":"info","@timestamp":"2022-08-30T07:39:34.392Z","log.logger":"beat","log.origin":{"file.name":"instance/beat.go","file.line":1085},"message":"Build info","service.name":"filebeat","system_info":{"build":{"commit":"ce383f1368ec7b3234de2dd4b1302be8db84fe1a","libbeat":"8.4.0","time":"2022-08-18T12:23:26.000Z","version":"8.4.0"},"ecs.version":"1.6.0"}}
{"log.level":"info","@timestamp":"2022-08-30T07:39:34.392Z","log.logger":"beat","log.origin":{"file.name":"instance/beat.go","file.line":1088},"message":"Go runtime info","service.name":"filebeat","system_info":{"go":{"os":"linux","arch":"amd64","max_procs":4,"version":"go1.17.12"},"ecs.version":"1.6.0"}}
{"log.level":"info","@timestamp":"2022-08-30T07:39:34.393Z","log.logger":"beat","log.origin":{"file.name":"instance/beat.go","file.line":1092},"message":"Host info","service.name":"filebeat","system_info":{"host":{"architecture":"x86_64","boot_time":"2022-08-30T07:32:31Z","containerized":false,"name":"websrv","ip":["127.0.0.1/8","::1/128","10.1.32.123/24","fe80::6c32:8aff:fe4a:2f47/64"],"kernel_version":"5.15.0-46-generic","mac":["6e:32:8a:4a:2f:47"],"os":{"type":"linux","family":"debian","platform":"ubuntu","name":"Ubuntu","version":"22.04.1 LTS (Jammy Jellyfish)","major":22,"minor":4,"patch":1,"codename":"jammy"},"timezone":"UTC","timezone_offset_sec":0,"id":"4d73804ce2ae43e196bdb2fba425d9af"},"ecs.version":"1.6.0"}}
{"log.level":"info","@timestamp":"2022-08-30T07:39:34.393Z","log.logger":"beat","log.origin":{"file.name":"instance/beat.go","file.line":1121},"message":"Process info","service.name":"filebeat","system_info":{"process":{"capabilities":{"inheritable":null,"permitted":["chown","dac_override","dac_read_search","fowner","fsetid","kill","setgid","setuid","setpcap","linux_immutable","net_bind_service","net_broadcast","net_admin","net_raw","ipc_lock","ipc_owner","sys_module","sys_rawio","sys_chroot","sys_ptrace","sys_pacct","sys_admin","sys_boot","sys_nice","sys_resource","sys_time","sys_tty_config","mknod","lease","audit_write","audit_control","setfcap","mac_override","mac_admin","syslog","wake_alarm","block_suspend","audit_read","38","39","40"],"effective":["chown","dac_override","dac_read_search","fowner","fsetid","kill","setgid","setuid","setpcap","linux_immutable","net_bind_service","net_broadcast","net_admin","net_raw","ipc_lock","ipc_owner","sys_module","sys_rawio","sys_chroot","sys_ptrace","sys_pacct","sys_admin","sys_boot","sys_nice","sys_resource","sys_time","sys_tty_config","mknod","lease","audit_write","audit_control","setfcap","mac_override","mac_admin","syslog","wake_alarm","block_suspend","audit_read","38","39","40"],"bounding":["chown","dac_override","dac_read_search","fowner","fsetid","kill","setgid","setuid","setpcap","linux_immutable","net_bind_service","net_broadcast","net_admin","net_raw","ipc_lock","ipc_owner","sys_module","sys_rawio","sys_chroot","sys_ptrace","sys_pacct","sys_admin","sys_boot","sys_nice","sys_resource","sys_time","sys_tty_config","mknod","lease","audit_write","audit_control","setfcap","mac_override","mac_admin","syslog","wake_alarm","block_suspend","audit_read","38","39","40"],"ambient":null},"cwd":"/etc/filebeat","exe":"/usr/share/filebeat/bin/filebeat","name":"filebeat","pid":1167,"ppid":1083,"seccomp":{"mode":"disabled","no_new_privs":false},"start_time":"2022-08-30T07:39:30.150Z"},"ecs.version":"1.6.0"}}
{"log.level":"info","@timestamp":"2022-08-30T07:39:34.394Z","log.origin":{"file.name":"instance/beat.go","file.line":293},"message":"Setup Beat: filebeat; Version: 8.4.0","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"warn","@timestamp":"2022-08-30T07:39:34.397Z","log.logger":"cfgwarn","log.origin":{"file.name":"tlscommon/config.go","file.line":102},"message":"DEPRECATED: Treating the CommonName field on X.509 certificates as a host name when no Subject Alternative Names are present is going to be removed. Please update your certificates if needed. Will be removed in version: 8.0.0","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-08-30T07:39:34.397Z","log.logger":"esclientleg","log.origin":{"file.name":"eslegclient/connection.go","file.line":108},"message":"elasticsearch url: https://10.1.32.44:9200","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-08-30T07:39:34.398Z","log.logger":"publisher","log.origin":{"file.name":"pipeline/module.go","file.line":113},"message":"Beat name: websrv","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-08-30T07:39:34.398Z","log.logger":"modules","log.origin":{"file.name":"fileset/modules.go","file.line":108},"message":"Enabled modules/filesets: ","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-08-30T07:39:34.398Z","log.logger":"esclientleg","log.origin":{"file.name":"eslegclient/connection.go","file.line":108},"message":"elasticsearch url: https://10.1.32.44:9200","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-08-30T07:39:34.426Z","log.logger":"esclientleg","log.origin":{"file.name":"eslegclient/connection.go","file.line":291},"message":"Attempting to connect to Elasticsearch version 8.4.0","service.name":"filebeat","ecs.version":"1.6.0"}
Overwriting ILM policy is disabled. Set `setup.ilm.overwrite: true` for enabling.

{"log.level":"info","@timestamp":"2022-08-30T07:39:34.426Z","log.logger":"index-management","log.origin":{"file.name":"idxmgmt/std.go","file.line":231},"message":"Auto ILM enable success.","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-08-30T07:39:34.433Z","log.logger":"index-management.ilm","log.origin":{"file.name":"ilm/std.go","file.line":118},"message":"ILM policy filebeat exists already.","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-08-30T07:39:34.433Z","log.logger":"index-management","log.origin":{"file.name":"idxmgmt/std.go","file.line":366},"message":"Set settings.index.lifecycle.name in template to {filebeat {\"policy\":{\"phases\":{\"hot\":{\"actions\":{\"rollover\":{\"max_age\":\"30d\",\"max_primary_shard_size\":\"50gb\"}}}}}}} as ILM is enabled.","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-08-30T07:39:34.465Z","log.logger":"template","log.origin":{"file.name":"template/load.go","file.line":245},"message":"Existing template will be overwritten, as overwrite is enabled.","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-08-30T07:39:37.312Z","log.logger":"template_loader","log.origin":{"file.name":"template/load.go","file.line":159},"message":"Try loading template filebeat-8.4.0 to Elasticsearch","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-08-30T07:39:37.393Z","log.logger":"add_cloud_metadata","log.origin":{"file.name":"add_cloud_metadata/add_cloud_metadata.go","file.line":102},"message":"add_cloud_metadata: hosting provider type not detected.","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-08-30T07:39:37.429Z","log.logger":"template_loader","log.origin":{"file.name":"template/load.go","file.line":127},"message":"Template with name \"filebeat-8.4.0\" loaded.","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-08-30T07:39:37.432Z","log.logger":"template_loader","log.origin":{"file.name":"template/load.go","file.line":184},"message":"Try loading data stream filebeat-8.4.0 to Elasticsearch","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-08-30T07:39:37.953Z","log.logger":"template_loader","log.origin":{"file.name":"template/load.go","file.line":150},"message":"Data stream with name \"filebeat-8.4.0\" loaded.","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-08-30T07:39:37.954Z","log.logger":"index-management","log.origin":{"file.name":"idxmgmt/std.go","file.line":267},"message":"Loaded index template.","service.name":"filebeat","ecs.version":"1.6.0"}
Index setup finished.
Loading dashboards (Kibana must be running and reachable)
{"log.level":"info","@timestamp":"2022-08-30T07:39:37.954Z","log.logger":"kibana","log.origin":{"file.name":"kibana/client.go","file.line":179},"message":"Kibana url: http://localhost:5601","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"error","@timestamp":"2022-08-30T07:39:37.955Z","log.origin":{"file.name":"instance/beat.go","file.line":1051},"message":"Exiting: error connecting to Kibana: fail to get the Kibana version: HTTP GET request to http://localhost:5601/api/status fails: fail to execute the HTTP GET request: Get \"http://localhost:5601/api/status\": dial tcp 127.0.0.1:5601: connect: connection refused. Response: ","service.name":"filebeat","ecs.version":"1.6.0"}
Exiting: error connecting to Kibana: fail to get the Kibana version: HTTP GET request to http://localhost:5601/api/status fails: fail to execute the HTTP GET request: Get "http://localhost:5601/api/status": dial tcp 127.0.0.1:5601: connect: connection refused. Response:

```

---

<div class="post-metadata">

**Author:** ![it\_dev](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/it_dev/32/110019_2.png) [@it\_dev](https://discuss.elastic.co/u/it_dev)\
**Post date:** [August 30, 2022, 5:41am UTC](https://discuss.elastic.co/t/squid-module-does-not-parse-logs/313088/16 "2022-08-30T05:41:14Z")

</div>

There is no /data directory in /etc/filebeat

---

<div class="post-metadata">

**Author:** ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)\
**Post date:** [August 30, 2022, 5:51am UTC](https://discuss.elastic.co/t/squid-module-does-not-parse-logs/313088/17 "2022-08-30T05:51:05Z")

</div>

Read the Docs as I suggested

> **[Directory layout | Filebeat Reference \[8.3\] | Elastic](https://www.elastic.co/guide/en/beats/filebeat/8.3/directory-layout.html)**

> **data** The location for persistent data files. `/var/lib/filebeat`

Clean out that directory

---

<div class="post-metadata">

**Author:** ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)\
**Post date:** [August 30, 2022, 5:53am UTC](https://discuss.elastic.co/t/squid-module-does-not-parse-logs/313088/18 "2022-08-30T05:53:11Z")

</div>

> [@it\_dev](#):
>
> ```auto
> {"log.level":"error","@timestamp":"2022-08-30T07:39:37.955Z","log.origin":{"file.name":"instance/beat.go","file.line":1051},"message":"Exiting: error connecting to Kibana: fail to get the Kibana version: HTTP GET request to http://localhost:5601/api/status fails: fail to execute the HTTP GET request: Get \"http://localhost:5601/api/status\": dial tcp 127.0.0.1:5601: connect: connection refused. Response: ","service.name":"filebeat","ecs.version":"1.6.0"}
> Exiting: error connecting to Kibana: fail to get the Kibana version: HTTP GET request to http://localhost:5601/api/status fails: fail to execute the HTTP GET request: Get "http://localhost:5601/api/status": dial tcp 127.0.0.1:5601: connect: connection refused. Response:
> 
> ```

Also you did not set the Kibana configuration correct in your filebeat.yml

you need to set this

```auto
setup.kibana:

  # Kibana Host
  # Scheme and port can be left out and will be set to the default (http and 5601)
  # In case you specify and additional path, the scheme is required: http://localhost:5601/path
  # IPv6 addresses should always be defined as: https://[2001:db8::1]:5601
  host: "localhost:5601" <!----- SET THIS

```

Also where are the logs from when you actually run filebeat.

`journalctl -u filebeat.service`

---

<div class="post-metadata">

**Author:** ![it\_dev](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/it_dev/32/110019_2.png) [@it\_dev](https://discuss.elastic.co/u/it_dev)\
**Post date:** [August 30, 2022, 7:16am UTC](https://discuss.elastic.co/t/squid-module-does-not-parse-logs/313088/19 "2022-08-30T07:16:39Z")

</div>

Thank you very much, after cleaning that directory filebeat shipped squid logs with right parsing.

But:

1. Why lots of bunch of empty and unnecessary fields are also shipped? aws.cloud, azure.activitylogs etc?  
I want only squid logs to be shipped.
2. The real log size of squid ~25mb. But in Data Streams it shows 66mb.

What maybe the problem?  
Again, thank you for your help!

---

<div class="post-metadata">

**Author:** ![it\_dev](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/it_dev/32/110019_2.png) [@it\_dev](https://discuss.elastic.co/u/it_dev)\
**Post date:** [August 30, 2022, 7:19am UTC](https://discuss.elastic.co/t/squid-module-does-not-parse-logs/313088/20 "2022-08-30T07:19:25Z")

</div>

> Also you did not set the Kibana configuration correct in your filebeat.yml

Is it required?  
I want to create my own dashboards.

[Next page](https://discuss.elastic.co/t/squid-module-does-not-parse-logs/313088.md?page=2)