# How hard is it to add a custom module?

**URL:** <https://discuss.elastic.co/t/how-hard-is-it-to-add-a-custom-module/234046>\
**Category:** Beats\
**Tags:** filebeat\
**Created:** [May 24, 2020, 2:30pm UTC](https://discuss.elastic.co/t/how-hard-is-it-to-add-a-custom-module/234046 "2020-05-24T14:30:52Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![Jonny\_McCullagh](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jonny_mccullagh/32/68830_2.png) [@Jonny\_McCullagh](https://discuss.elastic.co/u/Jonny_McCullagh)\
**Post date:** [May 24, 2020, 2:30pm UTC](https://discuss.elastic.co/t/how-hard-is-it-to-add-a-custom-module/234046/1 "2020-05-24T14:30:52Z")

</div>

I've been battling with this for days, I can't believe it can be this hard to process a custom log format.  
My log lines mainly look like :  
`[2020-05-24 13:40:06,414] {{jobs.py:1725}} WARNING - No viable dags retrieved from /usr/local/airflow/dags/some_script.py`  
But they occasionally have python exceptions which go on to multiple lines.  
I've created a custom module under /usr/share/filebeat/module/airflow

```auto
├── dags
│ ├── config
│ │ └── main.yml
│ ├── ingest
│ │ └── pipeline.json
│ └── manifest.yml
├── fields.go
├── _meta
│ ├── config.yml
│ ├── docs.asciidoc
│ └── fields.yml
└── module.yml

```

and enabled this under  
/etc/filebeat/modules.d/airflow.yml

```auto
- module: airflow
  dags:
    enabled: true
    var.paths:
      - "/path/to/logs/*.log"

```

My main.yml tries to account for the multiline python errors:

```auto
type: log
paths:
{{ range $i, $path := .paths }}
 - {{$path}}
{{ end }}
exclude_files: [".gz$"]
multiline:
  pattern: '^\['
  negate: true
  match: after

```

And this is my pipeline.yml

```auto
{
  "description": "Pipeline for Airflow DAG logs",
  "processors": [{
    "grok": {
      "field": "message",
      "patterns":[
        "\\[%{TIMESTAMP:timestamp}\\] {{%{DATA:script_name}:%{NUMBER:dag_id}}} %{LOGLEVEL:level} - %{GREEDYMULTILINE:airflow_message}"
      ],
      "ignore_missing": true,
      "pattern_definitions": {
        "TIMESTAMP": "%{YEAR}-%{MONTHNUM}-%{MONTHDAY} %{HOUR}:%{MINUTE}:%{SECOND}"
      }
    }
  },
  {
    "date": {
      "field": "timestamp",
      "target_field": "@timestamp",
      "formats": ["yyyy-MM-dd HH:mm:ss"],
      "ignore_failure": true
    }
  }
  ],
  "on_failure" : [{
    "set" : {
      "field" : "error.message",
      "value" : "{{ _ingest.on_failure_message }}"
    }
  }]
}

```

When I restart filebeat with debugging I can see:

```auto
2020-05-24T13:40:10Z DBG Publish event: {
  "@timestamp": "2020-05-24T13:40:10.680Z",
  "@metadata": {
    "beat": "filebeat",
    "type": "doc",
    "version": "6.0.1",
    "pipeline": "filebeat-6.0.1-airflow-dags-pipeline"
  },
  "source": "/usr/local/airflow/dags/some_script.py.log",
  "offset": 4355437,
  "message": "[2020-05-24 13:40:06,414] {{jobs.py:1725}} WARNING - No viable dags retrieved from /usr/local/airflow/dags/some_script.py",
  "fileset": {
    "module": "airflow",
    "name": "dags"
  },
  "prospector": {
    "type": "log"
  },
  "beat": {
    "name": "airflow-server-development-useast2-10-100-2-158",
    "hostname": "airflow-server-development-useast2-10-100-2-158",
    "version": "6.0.1"
  }
}

```

I don't know if that is supposed to indicate success but I do not see anything in Kibana.  
If I remove the multiline config in the module config/main.yml in Kibana I only get entries with error.message saying: Provided Grok expressions do not match field value

Help appreciated as I am on the verge of dumping Filebeat.

---

<div class="post-metadata">

**Author:** ![Andre\_Letterer](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/andre_letterer/32/42248_2.png) [@Andre\_Letterer](https://discuss.elastic.co/u/Andre_Letterer)\
**Post date:** [June 6, 2020, 1:20am UTC](https://discuss.elastic.co/t/how-hard-is-it-to-add-a-custom-module/234046/2 "2020-06-06T01:20:44Z")

</div>

I would say you are nearly there:  
`manifest.yml` missing  
`airflow.yml` ok  
`main.yml` ok  
`pipeline.json` not ok

Let me add the missing `manifest.yml`:

```auto
module_version: 1.0

var:
  - name: paths
    default:
      - /var/log/airflow/airflow.log*
    os.darwin:
      - /usr/local/airflow/var/log/airflow/airflow.log*
    os.windows:
      - c:/programdata/airflow/var/log/airflow/airflow.log*

ingest_pipeline: ingest/pipeline.json
input: config/main.yml 

```

You are getting this GROK error because `GREEDYMULTILINE` is missing:  
Please correct it like this:

```auto
{
  "description": "Pipeline for Airflow DAG logs",
  "processors": [{
    "grok": {
      "field": "message",
      "patterns":[
        "\\[%{TIMESTAMP:timestamp}\\] {{%{DATA:script_name}:%{NUMBER:dag_id}}} %{LOGLEVEL:level} - %{GREEDYMULTILINE:airflow_message}"
      ],
      "ignore_missing": true,
      "pattern_definitions": {
        "TIMESTAMP": "%{YEAR}-%{MONTHNUM}-%{MONTHDAY} %{HOUR}:%{MINUTE}:%{SECOND}",
        "GREEDYMULTILINE" : "(.|\n)*"
      }
    }
  },
  {
    "date": {
      "field": "timestamp",
      "target_field": "@timestamp",
      "formats": ["yyyy-MM-dd HH:mm:ss"],
      "ignore_failure": true
    }
  }
  ],
  "on_failure" : [{
    "set" : {
      "field" : "error.message",
      "value" : "{{ _ingest.on_failure_message }}"
    }
  }]
}

```

I made it working with that one.

---

<div class="post-metadata">

**Author:** ![Jonny\_McCullagh](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jonny_mccullagh/32/68830_2.png) [@Jonny\_McCullagh](https://discuss.elastic.co/u/Jonny_McCullagh)\
**Post date:** [June 8, 2020, 11:52am UTC](https://discuss.elastic.co/t/how-hard-is-it-to-add-a-custom-module/234046/3 "2020-06-08T11:52:13Z")

</div>

Thanks Andre,. I had the manifest but that GREEDYMULTILINE pattern definition got it working.  
I've put it publicly on gitlab if anyone else needs it in future:

> **[Jonny McCullagh / filebeat-airflow](https://gitlab.com/jonnymccullagh/filebeat-airflow)**
>
> A filebeat module for processing airflow DAG logs

  
Cheers,  
jonny

---

<div class="post-metadata">

**Author:** ![Andre\_Letterer](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/andre_letterer/32/42248_2.png) [@Andre\_Letterer](https://discuss.elastic.co/u/Andre_Letterer)\
**Post date:** [June 8, 2020, 6:12pm UTC](https://discuss.elastic.co/t/how-hard-is-it-to-add-a-custom-module/234046/4 "2020-06-08T18:12:54Z")

</div>

Ok, perfect. Nice to hear you got it working.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [July 6, 2020, 6:13pm UTC](https://discuss.elastic.co/t/how-hard-is-it-to-add-a-custom-module/234046/5 "2020-07-06T18:13:05Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.