Filebeat setup for cakephp logs

sanjeev1895 · July 30, 2023, 3:27pm

Hi,

Can anyone advice how to configure the filebeat and logstash for cakephp logs.
I need to configure filebeat for following cakephp logs,

cake.log
error.log
Is do I need to enable any module in filebeat?
Please check the logs format below..
error.log:

2023-07-26 04:35:45 Error: [BadRequestException] Invalid Store
Request URL: /
Stack Trace:
#0 /var/www/build/storefront/storefront-516/app/Controller/AppController.php(76): AppController->checkStore()
#1 /var/www/build/storefront/storefront-516/app/Controller/SiteController.php(25): AppController->beforeFilter()
#2 /var/www/build/storefront/storefront-516/lib/Cake/Event/CakeEventManager.php(243): SiteController->beforeFilter()
#3 /var/www/build/storefront/storefront-516/lib/Cake/Controller/Controller.php(682): CakeEventManager->dispatch()
#4 /var/www/build/storefront/storefront-516/lib/Cake/Routing/Dispatcher.php(189): Controller->startupProcess()
#5 /var/www/build/storefront/storefront-516/lib/Cake/Routing/Dispatcher.php(167): Dispatcher->_invoke()
#6 /var/www/build/storefront/storefront-516/app/webroot/index.php(114): Dispatcher->dispatch()
#7 {main}

cakephp logs:

2023-07-26 16:33:39 Cake: Error:- Custom Forms - SMTP server did not accept the password. Array
(
    [file] => /var/www/build/storefront/storefront-522/app/Controller/FormsController.php
    [line] => 42
    [function] => save_form_responses
    [class] => CustomFormComponent
    [type] => ->
)

Thanks in advance.

stephenb · July 31, 2023, 2:42am

@sanjeev1895
What version are you on?
How much do you want to parse.

With just filebeat + multiline processing

- type: filestream

  # Unique ID among all inputs, an ID is required.
  id: cake-log
  tags: "cake-log"

  # Change to true to enable this input configuration.
  enabled: true

  # Paths that should be crawled and fetched. Glob based paths.
  paths:
    - /Users/sbrown/workspace/sample-data/discuss/multi-line/cake.log
    #- c:\programdata\elasticsearch\logs\*
  parsers:
  - multiline:
      type: pattern
      pattern: '^[0-9]{4}-[0-9]{2}-[0-9]{2}'
      negate: true
      match: after

This loaded the cake.log the error log is the same you could us a separate filestream and tag.

If you want to parse it further you can use and ingest pipeline OR you could use logstash there are pros and cons to each.

Ingest pipeline you do not need to logstash and you could probably parse.
Logstash has more powerful parsing but requires running another component.

Get the logs loaded with just filebeat and come back with more questions.

Here is my sample files these loaded fine with the above config just change the path.

cake-error.log

2023-07-26 04:35:45 Error: [BadRequestException] Invalid Store
Request URL: /
Stack Trace:
#0 /var/www/build/storefront/storefront-516/app/Controller/AppController.php(76): AppController->checkStore()
#1 /var/www/build/storefront/storefront-516/app/Controller/SiteController.php(25): AppController->beforeFilter()
#2 /var/www/build/storefront/storefront-516/lib/Cake/Event/CakeEventManager.php(243): SiteController->beforeFilter()
#3 /var/www/build/storefront/storefront-516/lib/Cake/Controller/Controller.php(682): CakeEventManager->dispatch()
#4 /var/www/build/storefront/storefront-516/lib/Cake/Routing/Dispatcher.php(189): Controller->startupProcess()
#5 /var/www/build/storefront/storefront-516/lib/Cake/Routing/Dispatcher.php(167): Dispatcher->_invoke()
#6 /var/www/build/storefront/storefront-516/app/webroot/index.php(114): Dispatcher->dispatch()
#7 {main}
2023-07-26 04:36:45 Error: [BadRequestException] Invalid Store
Request URL: /
Stack Trace:
#0 /var/www/build/storefront/storefront-516/app/Controller/AppController.php(76): AppController->checkStore()
#1 /var/www/build/storefront/storefront-516/app/Controller/SiteController.php(25): AppController->beforeFilter()
#2 /var/www/build/storefront/storefront-516/lib/Cake/Event/CakeEventManager.php(243): SiteController->beforeFilter()
#3 /var/www/build/storefront/storefront-516/lib/Cake/Controller/Controller.php(682): CakeEventManager->dispatch()
#4 /var/www/build/storefront/storefront-516/lib/Cake/Routing/Dispatcher.php(189): Controller->startupProcess()
#5 /var/www/build/storefront/storefront-516/lib/Cake/Routing/Dispatcher.php(167): Dispatcher->_invoke()
#6 /var/www/build/storefront/storefront-516/app/webroot/index.php(114): Dispatcher->dispatch()
#7 {main}
2023-07-26 04:37:45 Error: [BadRequestException] Invalid Store
Request URL: /
Stack Trace:
#0 /var/www/build/storefront/storefront-516/app/Controller/AppController.php(76): AppController->checkStore()
#1 /var/www/build/storefront/storefront-516/app/Controller/SiteController.php(25): AppController->beforeFilter()
#2 /var/www/build/storefront/storefront-516/lib/Cake/Event/CakeEventManager.php(243): SiteController->beforeFilter()
#3 /var/www/build/storefront/storefront-516/lib/Cake/Controller/Controller.php(682): CakeEventManager->dispatch()
#4 /var/www/build/storefront/storefront-516/lib/Cake/Routing/Dispatcher.php(189): Controller->startupProcess()
#5 /var/www/build/storefront/storefront-516/lib/Cake/Routing/Dispatcher.php(167): Dispatcher->_invoke()
#6 /var/www/build/storefront/storefront-516/app/webroot/index.php(114): Dispatcher->dispatch()
#7 {main}

cake/log

2023-07-26 16:33:39 Cake: Error:- Custom Forms - SMTP server did not accept the password. Array
(
    [file] => /var/www/build/storefront/storefront-522/app/Controller/FormsController.php
    [line] => 42
    [function] => save_form_responses
    [class] => CustomFormComponent
    [type] => ->
)
2023-07-26 16:34:39 Cake: Error:- Custom Forms - SMTP server did not accept the password. Array
(
    [file] => /var/www/build/storefront/storefront-522/app/Controller/FormsController.php
    [line] => 42
    [function] => save_form_responses
    [class] => CustomFormComponent
    [type] => ->
)
2023-07-26 16:35:39 Cake: Error:- Custom Forms - SMTP server did not accept the password. Array
(
    [file] => /var/www/build/storefront/storefront-522/app/Controller/FormsController.php
    [line] => 42
    [function] => save_form_responses
    [class] => CustomFormComponent
    [type] => ->
)

sanjeev1895 · August 4, 2023, 2:16pm

Hi @stephenb

Am I using the ELK stack 8.9 version. and I want to parse the cakephp logs using logstash grok pattern.
I need to send the logs from filebeat modules. can you please tell me how to configure this.

stephenb · August 4, 2023, 5:57pm

You are going to need to do a little reading and work

and here is a skeleton logstash.conf and some more direction,

sanjeev1895 · August 6, 2023, 1:37pm

@stephenb

I configured the apache server access and error logs. It's working fine as I expected. For this I enabled the filebeat apache module and configured the log file path.

But for cakephp, where I need to pass the log filepath?
I tried to pass the cakephp log file path in filebeat apache module and system module but it's not working. below is my cakephp configuration.

input {
  beats {
     port => 5044
 }
}

filter {
    if [log][file][path] == ["/var/log/cakephp/staging/pos/error.log"] {
      grok {
        match => { "message" => "%{DATA:time_stamp}\[%{DATA:message}(\r|\n)+(?m)%{GREEDYDATA:stack-trace}" }
 }
    }

else if [log][file][path] == ["/var/log/cakephp/staging/apipos/error.log"] {
      grok {
        match => { "message" => "%{DATA:time_stamp}\[%{DATA:message}(\r|\n)+(?m)%{GREEDYDATA:stack-trace}" }
   }
       }
}

output {
if [log][file][path] == ["/var/log/cakephp/staging/pos/error.log"] {
  elasticsearch {
    hosts => ["https://a.a.a.a:9200"]
    index => "orchid-staging-cakephp-%{+YYYY.MM.dd}"
    user => "elastic"
    password => "ppppp"
    ssl => true
    cacert => "/etc/logstash/http_ca.crt"
  }
}
else if [log][file][path] == ["/var/log/cakephp/staging/apipos/error.log"] {
elasticsearch {
    hosts => ["https://a.a.a.a:9200"]
    index => "orchid-staging-cakephp-%{+YYYY.MM.dd}"
    user => "elastic"
    password => "pppppppp"
    ssl => true
    cacert => "/etc/logstash/http_ca.crt"
  }
}
}

Filebeat: system module:
var.paths: ["/var/www/cakelog/staging/pos/error.log","/var/www/cakelog/staging/apipos/error.log"]

Can you please advice for this setup.

stephenb · August 6, 2023, 3:46pm

Hi @sanjeev1895

Glad you got the Apache logs working.

For the cake logs

Do not use the system module that's going to expect syslog.

I gave you a working example above using the filestream input. You put that in filebeat.yml` You need to include the multi-line parser as well.

sanjeev1895 · August 6, 2023, 4:52pm

Hi @stephenb

I tried your method but I can't get the logs.. below is my config..

filebeat.inputs:

# Each - is an input. Most options can be set at the input level, so
# you can use different inputs for various configurations.
# Below are the input specific configurations.

# filestream is an input for collecting log messages from files.
- type: filestream

  # Unique ID among all inputs, an ID is required.
  id: error-log
  tags: "error-log"

  # Change to true to enable this input configuration.
  enabled: false

  # Paths that should be crawled and fetched. Glob based paths.
  paths:
          #- /var/log/*.log
          - /var/log/cakephp/staging/pos/error.log
    #- c:\programdata\elasticsearch\logs\*
    parsers:
    - multiline:
       type: pattern
       pattern: '^[0-9]{4}-[0-9]{2}-[0-9]{2}'
       negate: true
       match: after

And the output configuration is,

output.logstash:
  # The Logstash hosts
    hosts: ["a.a.a.a:5044"]

Also as per the above comment I disabled the system module. Can you please tell me what I missed here. Thanks

stephenb · August 6, 2023, 5:17pm

That's perhaps because in your logstash output you have filters so those logs may never be reaching elasticsearch. You have to look at it and as a whole.

I would put another output to elasticsearch with no filter and a test indices and see if they arrive there and then you can figure out how to filter and sort them.

My guess is they're getting filtered out with your if and else in the log stash output section

Just put an else with something like this So whatever does not meet your filters goes to that then look in there

elasticsearch {
    hosts => ["https://a.a.a.a:9200"]
    index => "test cakephp-%{+YYYY.MM.dd}"
    user => "elastic"
    password => "pppppppp"
    ssl => true
    cacert => "/etc/logstash/http_ca.crt"
  }

stephenb · August 6, 2023, 5:25pm

@sanjeev1895

OHHH Darn!!! Make sure that filebeat is input is enabled... Your config above still shows it disabled so it won't read anything

You will notice my sample above. It is enabled to true

sanjeev1895 · August 7, 2023, 4:37pm

@stephenb
Yes, I corrected that. But Won't work. I tried what you have mentioned above methods. Below is my entire configuration.. can you please check and advise me what I'm wrong here...
filebeat.yml configuration:

filebeat.inputs:

# Each - is an input. Most options can be set at the input level, so
# you can use different inputs for various configurations.
# Below are the input specific configurations.

# filestream is an input for collecting log messages from files.
- type: filestream

  # Unique ID among all inputs, an ID is required.
  id: my-filestream-id
  #  tags: "error-log"

  # Change to true to enable this input configuration.
  enabled: true

  # Paths that should be crawled and fetched. Glob based paths.
  paths:
    - /var/log/cakelog/staging/pos/error.log

  parsers:
    - multiline:
      type: pattern
      pattern: '^[0-9]{4}-[0-9]{2}-[0-9]{2}'
      negate: true
      match: after

# ========= Kibana ====
setup.kibana:
        host: "a.a.a.a:5601"

# ---------- Logstash Output -------
output.logstash:
  # The Logstash hosts
   hosts: ["a.a.a.a:5044"]

logstash conf file:

input {
  beats {
    port => 5044
  }
}

output {
  elasticsearch {
    hosts => ["https://a.a.a.a:9200"]
    index => "%{[@metadata][beat]}-%{[@metadata][version]}"
    action => "create"
    user => "elastic"
    password => "ppppppp"
    ssl => true
    cacert => "/etc/logstash/http_ca.crt"
  }
}

I want to parse this cakephp log file:

2023-08-04 01:43:07 Error: [PDOException] SQLSTATE[42000]: Syntax error or access violation: 1064 You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near 'AND Contact.contact_type = 1 AND (first_name ='' and last_name ='')' at line 6
Request URL: /ammo_read/listOrdersRetrieve
Stack Trace:
#0 /var/www/build-live/apipos/apipos-109/lib/Cake/Model/Datasource/DboSource.php(502): PDOStatement->execute()
#1 /var/www/build-live/apipos/apipos-109/lib/Cake/Model/Datasource/DboSource.php(468): DboSource->_execute()
#2 /var/www/build-live/apipos/apipos-109/lib/Cake/Model/Datasource/DboSource.php(715): DboSource->execute()
#3 /var/www/build-live/apipos/apipos-109/lib/Cake/Model/Datasource/DboSource.php(641): DboSource->fetchAll()
#4 /var/www/build-live/apipos/apipos-109/lib/Cake/Model/Model.php(3489): DboSource->query()
#5 /var/www/build-live/apipos/apipos-109/app/Model/Contact.php(831): Model->query()
#6 /var/www/build-live/apipos/apipos-109/app/Controller/AmmoReadyController.php(1304): Contact->checkDuplicate1()
#7 [internal function]: AmmoReadyController->listOrdersRetrieve()
#8 /var/www/build-live/apipos/apipos-109/lib/Cake/Controller/Controller.php(499): ReflectionMethod->invokeArgs()
#9 /var/www/build-live/apipos/apipos-109/lib/Cake/Routing/Dispatcher.php(193): Controller->invokeAction()
#10 /var/www/build-live/apipos/apipos-109/lib/Cake/Routing/Dispatcher.php(167): Dispatcher->_invoke()
#11 /var/www/build-live/apipos/apipos-109/app/webroot/index.php(110): Dispatcher->dispatch()
#12 {main}

stephenb · August 8, 2023, 4:21am

I just loaded the file at the bottom with this config

You should try the same...

Do you have more than one error set in the file or just 1?

Are there new entries in the log or are you trying to reload the same file (which will not work)
if you want to do that you have you cleanup the data registry

You can start filebeat with to see what is being sent

filebeat -e -d "*"

# filestream is an input for collecting log messages from files.
- type: filestream

  # Unique ID among all inputs, an ID is required.
  id: cake-log
  tags: "cake-error-log"

  # Change to true to enable this input configuration.
  enabled: true

  # Paths that should be crawled and fetched. Glob based paths.
  paths:
    - /Users/sbrown/workspace/sample-data/discuss/multi-line/cake-error.log
    #- c:\programdata\elasticsearch\logs\*
  parsers:
  - multiline:
      type: pattern
      pattern: '^[0-9]{4}-[0-9]{2}-[0-9]{2}'
      negate: true
      match: after

I just loaded this file

2023-07-26 04:35:45 Error: [BadRequestException] Invalid Store
Request URL: /
Stack Trace:
#0 /var/www/build/storefront/storefront-516/app/Controller/AppController.php(76): AppController->checkStore()
#1 /var/www/build/storefront/storefront-516/app/Controller/SiteController.php(25): AppController->beforeFilter()
#2 /var/www/build/storefront/storefront-516/lib/Cake/Event/CakeEventManager.php(243): SiteController->beforeFilter()
#3 /var/www/build/storefront/storefront-516/lib/Cake/Controller/Controller.php(682): CakeEventManager->dispatch()
#4 /var/www/build/storefront/storefront-516/lib/Cake/Routing/Dispatcher.php(189): Controller->startupProcess()
#5 /var/www/build/storefront/storefront-516/lib/Cake/Routing/Dispatcher.php(167): Dispatcher->_invoke()
#6 /var/www/build/storefront/storefront-516/app/webroot/index.php(114): Dispatcher->dispatch()
#7 {main}
2023-07-26 04:36:45 Error: [BadRequestException] Invalid Store
Request URL: /
Stack Trace:
#0 /var/www/build/storefront/storefront-516/app/Controller/AppController.php(76): AppController->checkStore()
#1 /var/www/build/storefront/storefront-516/app/Controller/SiteController.php(25): AppController->beforeFilter()
#2 /var/www/build/storefront/storefront-516/lib/Cake/Event/CakeEventManager.php(243): SiteController->beforeFilter()
#3 /var/www/build/storefront/storefront-516/lib/Cake/Controller/Controller.php(682): CakeEventManager->dispatch()
#4 /var/www/build/storefront/storefront-516/lib/Cake/Routing/Dispatcher.php(189): Controller->startupProcess()
#5 /var/www/build/storefront/storefront-516/lib/Cake/Routing/Dispatcher.php(167): Dispatcher->_invoke()
#6 /var/www/build/storefront/storefront-516/app/webroot/index.php(114): Dispatcher->dispatch()
#7 {main}
2023-07-26 04:37:45 Error: [BadRequestException] Invalid Store
Request URL: /
Stack Trace:
#0 /var/www/build/storefront/storefront-516/app/Controller/AppController.php(76): AppController->checkStore()
#1 /var/www/build/storefront/storefront-516/app/Controller/SiteController.php(25): AppController->beforeFilter()
#2 /var/www/build/storefront/storefront-516/lib/Cake/Event/CakeEventManager.php(243): SiteController->beforeFilter()
#3 /var/www/build/storefront/storefront-516/lib/Cake/Controller/Controller.php(682): CakeEventManager->dispatch()
#4 /var/www/build/storefront/storefront-516/lib/Cake/Routing/Dispatcher.php(189): Controller->startupProcess()
#5 /var/www/build/storefront/storefront-516/lib/Cake/Routing/Dispatcher.php(167): Dispatcher->_invoke()
#6 /var/www/build/storefront/storefront-516/app/webroot/index.php(114): Dispatcher->dispatch()
#7 {main}
2023-08-04 01:43:07 Error: [PDOException] SQLSTATE[42000]: Syntax error or access violation: 1064 You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near 'AND Contact.contact_type = 1 AND (first_name ='' and last_name ='')' at line 6
Request URL: /ammo_read/listOrdersRetrieve
Stack Trace:
#0 /var/www/build-live/apipos/apipos-109/lib/Cake/Model/Datasource/DboSource.php(502): PDOStatement->execute()
#1 /var/www/build-live/apipos/apipos-109/lib/Cake/Model/Datasource/DboSource.php(468): DboSource->_execute()
#2 /var/www/build-live/apipos/apipos-109/lib/Cake/Model/Datasource/DboSource.php(715): DboSource->execute()
#3 /var/www/build-live/apipos/apipos-109/lib/Cake/Model/Datasource/DboSource.php(641): DboSource->fetchAll()
#4 /var/www/build-live/apipos/apipos-109/lib/Cake/Model/Model.php(3489): DboSource->query()
#5 /var/www/build-live/apipos/apipos-109/app/Model/Contact.php(831): Model->query()
#6 /var/www/build-live/apipos/apipos-109/app/Controller/AmmoReadyController.php(1304): Contact->checkDuplicate1()
#7 [internal function]: AmmoReadyController->listOrdersRetrieve()
#8 /var/www/build-live/apipos/apipos-109/lib/Cake/Controller/Controller.php(499): ReflectionMethod->invokeArgs()
#9 /var/www/build-live/apipos/apipos-109/lib/Cake/Routing/Dispatcher.php(193): Controller->invokeAction()
#10 /var/www/build-live/apipos/apipos-109/lib/Cake/Routing/Dispatcher.php(167): Dispatcher->_invoke()
#11 /var/www/build-live/apipos/apipos-109/app/webroot/index.php(110): Dispatcher->dispatch()
#12 {main}

GET filebeat-8.8.0/_search

{
  "took": 1,
  "timed_out": false,
  "_shards": {
    "total": 1,
    "successful": 1,
    "skipped": 0,
    "failed": 0
  },
  "hits": {
    "total": {
      "value": 8,
      "relation": "eq"
    },
    "max_score": 1,
    "hits": [
      {
        "_index": ".ds-filebeat-8.8.0-2023.08.08-000001",
        "_id": "-9te04kBB_lD3cFyDIcI",
        "_score": 1,
        "_source": {
          "@timestamp": "2023-08-08T04:18:22.328Z",
          "input": {
            "type": "filestream"
          },
          "host": {
            "name": "hyperion.lan"
          },
          "agent": {
            "name": "hyperion.lan",
            "type": "filebeat",
            "version": "8.8.0",
            "ephemeral_id": "0fbe201e-58af-4405-8071-666a64b1acab",
            "id": "769d262d-ef89-45be-a2d7-ccbbedd9fc19"
          },
          "ecs": {
            "version": "8.0.0"
          },
          "log": {
            "file": {
              "path": "/Users/sbrown/workspace/sample-data/discuss/multi-line/cake-error.log"
            },
            "flags": [
              "multiline"
            ],
            "offset": 855
          },
          "message": """2023-07-26 04:35:45 Error: [BadRequestException] Invalid Store
Request URL: /
Stack Trace:
#0 /var/www/build/storefront/storefront-516/app/Controller/AppController.php(76): AppController->checkStore()
#1 /var/www/build/storefront/storefront-516/app/Controller/SiteController.php(25): AppController->beforeFilter()
#2 /var/www/build/storefront/storefront-516/lib/Cake/Event/CakeEventManager.php(243): SiteController->beforeFilter()
#3 /var/www/build/storefront/storefront-516/lib/Cake/Controller/Controller.php(682): CakeEventManager->dispatch()
#4 /var/www/build/storefront/storefront-516/lib/Cake/Routing/Dispatcher.php(189): Controller->startupProcess()
#5 /var/www/build/storefront/storefront-516/lib/Cake/Routing/Dispatcher.php(167): Dispatcher->_invoke()
#6 /var/www/build/storefront/storefront-516/app/webroot/index.php(114): Dispatcher->dispatch()
#7 {main}""",
          "tags": [
            "cake-error-log"
          ]
        }
      },
.......

sanjeev1895 · August 8, 2023, 4:30am

@stephenb
Thank you so much for the reply . I have more than 1 error set. Whenever I get the error logs in this file I need to send it to kibana data view like how do we configure the apache logs with apache module.

Yesterday I also tried to load this command filebeat -e -d /etc/filebeat/filebeat_multiline.yml. and I'm getting the exact same output as you mentioned above. but my motive is to send this output to index and it should be viewed in the kibana.

Thanks.

stephenb · August 8, 2023, 5:14am

Nope syntax error

-c to specify the config file so it's not reading that file It's just reading the default file.

-d is debug and since you didn't specify a valid, it's just doing normal

filebeat -e -c /etc/filebeat/filebeat_multiline.yml

sanjeev1895 · August 8, 2023, 12:53pm

@stephenb

I tried this command with above config filebeat -e -c /etc/filebeat/filebeat.yml it worked for me. here in filebeat.yml file I mentioned the output as in the console like below.. so I getting the output in console.
========output================

output.console:
               pretty: true

But how do i configure the the output to logstash index pattern. so that I can create the data view and I can monitor the daily logs.

stephenb · August 8, 2023, 1:41pm

Since you want to do

filebeat>logstash>Elasticsearch

You need to set up a datastream

Then you configure file beat to point the logstash

And logstash to write to the datastream.

That is the correct method.

You can set the ILM policy to do daily if you want.

If you have more questions on that, you should open specific topic on it

sanjeev1895 · August 8, 2023, 5:25pm

@stephenb

Can please tell how did you send this cake logs to kibana dashboard. As you mentioned in the very 1st screenshot.

stephenb · August 8, 2023, 5:41pm

I used my config everuthing else defaults
I ran
filebeat setup -e
which created the filebeat assets... which creates a data view.

then I Ran
filebeat -e
with the output to elasticsearch directly ... NOT trying to name a special index just using defaults

Then I Just when to Kibana -> Discover

Here is my complete filebeat.yml



filebeat.inputs:

# Each - is an input. Most options can be set at the input level, so
# you can use different inputs for various configurations.
# Below are the input specific configurations.

# filestream is an input for collecting log messages from files.
- type: filestream

  # Unique ID among all inputs, an ID is required.
  id: cake-log
  tags: "cake-error-log"

  # Change to true to enable this input configuration.
  enabled: true

  # Paths that should be crawled and fetched. Glob based paths.
  paths:
    - /Users/sbrown/workspace/sample-data/discuss/multi-line/cake-error.log
    #- c:\programdata\elasticsearch\logs\*
  parsers:
  - multiline:
      type: pattern
      pattern: '^[0-9]{4}-[0-9]{2}-[0-9]{2}'
      negate: true
      match: after


# ============================== Filebeat modules ==============================

filebeat.config.modules:
  # Glob pattern for configuration loading
  path: ${path.config}/modules.d/*.yml

  # Set to true to enable config reloading
  reload.enabled: false

  # Period on which files under path should be checked for changes
  #reload.period: 10s

# ======================= Elasticsearch template setting =======================

setup.template.settings:
  index.number_of_shards: 1
  #index.codec: best_compression
  #_source.enabled: false



setup.kibana:

# ================================== Outputs ===================================

# Configure what output to use when sending the data collected by the beat.

# ---------------------------- Elasticsearch Output ----------------------------
output.elasticsearch:
  # Array of hosts to connect to.
  hosts: ["http://localhost:9200"]

  # Protocol - either `http` (default) or `https`.
  #protocol: "https"

  # Authentication credentials - either API key or username/password.
  # #api_key: "id:api_key"
  # username: "elastic"
  # password: "blueboa"
  # ssl.verification_mode: "none"

sanjeev1895 · August 8, 2023, 5:45pm

@stephenb
Okay. Thank you so much.

sanjeev1895 · August 9, 2023, 12:51pm

@stephenb

While I'm loading this command filebeat setup -e in remote server where I configured the filebeat, I'm getting this error,


"message":"error connecting to Elasticsearch at https://173.31.59.112:9200: Get \"https://173.31.59.112:9200\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"error","@timestamp":"2023-08-09T12:44:23.011Z","log.origin":{"file.name":"instance/beat.go","file.line":1274},"message":"Exiting: couldn't connect to any of the configured Elasticsearch hosts. Errors: [error connecting to Elasticsearch at https://173.31.59.112:9200: Get \"https://173.31.59.112:9200\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)]","service.name":"filebeat","ecs.version":"1.6.0"}
Exiting: couldn't connect to any of the configured Elasticsearch hosts. Errors: [error connecting to Elasticsearch at https://173.31.59.112:9200: Get "https://173.31.59.112:9200": context deadline exceeded (Client.Timeout exceeded while awaiting headers)]

For this I check the my security group everything was fine and I also disabled the system firewall (ufw). can you please tell me what I need to change for this?

stephenb · August 9, 2023, 2:04pm

Not Sure.... Please run

filebeat test output

And please share your output section of your filebeat.yml

Probably SSL

Please can also run this from the filebeat box and show output

curl -k -v -u elastic https://173.31.59.112:9200