# Filebeat logstash module not working with json processor

**URL:** <https://discuss.elastic.co/t/filebeat-logstash-module-not-working-with-json-processor/213728>\
**Category:** Beats\
**Tags:** beats-module, filebeat\
**Created:** [January 3, 2020, 4:39pm UTC](https://discuss.elastic.co/t/filebeat-logstash-module-not-working-with-json-processor/213728 "2020-01-03T16:39:13Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![lorenzopolidori](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/lorenzopolidori/32/60253_2.png) [@lorenzopolidori](https://discuss.elastic.co/u/lorenzopolidori)\
**Post date:** [January 3, 2020, 4:39pm UTC](https://discuss.elastic.co/t/filebeat-logstash-module-not-working-with-json-processor/213728/1 "2020-01-03T16:39:13Z")

</div>

Hi,

I am trying to send logs with Filebeat 7.5.1 from an EC2 Ubuntu instance to cloud Elasticsearch.

The log file is `/var/log/logback/oauth-service.log`, in the following logstash format:

```
{"@timestamp":"2020-01-02T16:05:16.692+01:00","@version":"1","message":"Running with Spring Boot v2.2.2.RELEASE, Spring v5.2.2.RELEASE","logger_name":"com.moovimento.oauth.Application","thread_name":"main","level":"DEBUG","level_value":10000}
{"@timestamp":"2020-01-02T16:05:16.693+01:00","@version":"1","message":"The following profiles are active: dev","logger_name":"com.moovimento.oauth.Application","thread_name":"main","level":"INFO","level_value":20000}
{"@timestamp":"2020-01-02T16:05:17.400+01:00","@version":"1","message":"Bootstrapping Spring Data MongoDB repositories in DEFAULT mode.","logger_name":"org.springframework.data.repository.config.RepositoryConfigurationDelegate","thread_name":"main","level":"INFO","level_value":20000}

```

My `/etc/filebeat/filebeat.yml` is as follows:

```
filebeat.inputs:
- type: log
  enabled: true
  paths:
    - /var/log/*.log
  fields:
    level: debug

filebeat.config.modules:
  path: ${path.config}/modules.d/*.yml
  reload.enabled: false

setup.template.settings:
  index.number_of_shards: 1

cloud.id: "<name>:<key>"
cloud.auth: "elastic:<password>"

processors:
  - add_host_metadata: ~
  - add_cloud_metadata: ~
  - add_docker_metadata: ~
  - add_kubernetes_metadata: ~

logging.level: debug

```

My `/etc/filebeat/modules.d/logstash.yml` is as follows

```
- module: logstash
  log:
    enabled: true
    var.paths: ["/var/log/logback/*"]
    var.format: json
  slowlog:
    enabled: false

```

I setup the pipeline with

```
filebeat modules enable logstash
filebeat setup -e
service filebeat restart

```

However, for some reason I can's see any logs on Kibana.

If I omit `var.format: json` in `logstash.yml`, I can see the logs on Kibana but not properly formatted, since the grok processor expects plain format, i.e. the message field contains the whole escaped json logs.

`logstash` is actually not running on the machine, my application is directly logging into `/var/log/logback/oauth-service.log` using `logback`.

Note: I can ingest nginx access logs and see them on kibana with no problems using the nginx Filebeat module.

---

<div class="post-metadata">

**Author:** ![faec](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/faec/32/46988_2.png) [@faec](https://discuss.elastic.co/u/faec)\
**Post date:** [January 8, 2020, 8:43pm UTC](https://discuss.elastic.co/t/filebeat-logstash-module-not-working-with-json-processor/213728/2 "2020-01-08T20:43:18Z")

</div>

Hmm, it is strange that the format line alone would make the difference. Can you check the logs for Filebeat itself? Since you are writing "logstash-format" logs rather than using logstash itself, my first suspicion is that the logstash module is making some assumption that fails on your data (e.g. a missing field, a field of the wrong type, etc.) which is preventing ingestion. It's hard to tell what it might be from the log itself, though, so checking the Filebeat logs to look for failures is probably the next step.

Alternatively, you can still ingest json without using the logstash module -- if you use a plain `log` input with json settings that should bypass any logstash-specific assumptions that are breaking things. See the [log input configuration docs](https://www.elastic.co/guide/en/beats/filebeat/current/filebeat-input-log.html) for details. Whether this works for you depends on why you needed the logstash module specifically, though.

---

<div class="post-metadata">

**Author:** ![lorenzopolidori](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/lorenzopolidori/32/60253_2.png) [@lorenzopolidori](https://discuss.elastic.co/u/lorenzopolidori)\
**Post date:** [January 8, 2020, 9:54pm UTC](https://discuss.elastic.co/t/filebeat-logstash-module-not-working-with-json-processor/213728/3 "2020-01-08T21:54:39Z")

</div>

Hi @faec, thanks for your suggestion. Your second option is exactly what I ended up doing:

```
- type: log
    enabled: true

    paths:
        - /var/log/logback/*.log

    fields:
        application:
            name: app-name 

    fields_under_root: true

    json:
        keys_under_root: true
        overwrite_keys: true
        add_error_key: true

```

This is working as expected.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [February 5, 2020, 9:54pm UTC](https://discuss.elastic.co/t/filebeat-logstash-module-not-working-with-json-processor/213728/4 "2020-02-05T21:54:39Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.