# Detecting JSON in Ingest pipeline

**URL:** <https://discuss.elastic.co/t/detecting-json-in-ingest-pipeline/151464>\
**Category:** Elasticsearch\
**Created:** [October 8, 2018, 1:42pm UTC](https://discuss.elastic.co/t/detecting-json-in-ingest-pipeline/151464 "2018-10-08T13:42:45Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![Merclangrat](https://avatars.discourse-cdn.com/v4/letter/m/a3d4f5/32.png) [@Merclangrat](https://discuss.elastic.co/u/Merclangrat)\
**Post date:** [October 8, 2018, 1:42pm UTC](https://discuss.elastic.co/t/detecting-json-in-ingest-pipeline/151464/1 "2018-10-08T13:42:46Z")

</div>

Hi all!

I have the following pipeline:  
{  
"description" : "Ingest pipeline",  
"processors" : [  
{  
"grok": {  
"field": "message",  
"patterns": ["%{TIMESTAMP\_ISO8601:logtime} %{WORD:loglevel} - %{GREEDYDATA:logdata}"]  
}  
},  
{  
"json": {  
"field": "logdata",  
"add\_to\_root": true  
}  
},  
{  
"remove": {  
"field": ["message","logdata"]  
}  
}  
]  
}'

The log line looks like this (it's got from our app, then processed by Filebeat):  
2018-10-08 13:39:36,247 INFO - {"a": "b"....}

But, sometimes our app throws loglines which aren't JSON:  
2018-10-08 08:54:18,592 INFO - user 10470684 cannot assume zuid 10470684

In this case pipeline fails, but if this log line doesn't appear in EFK, this is not the problem. The problem is that **ingesting is completely stopped, and I need to restart Filebeat.**

Now I skip that lines in Filebeat config, but I can't figure out if I add all problematic lines to its config. I would like to:

- just skip line if it's not JSON
- or (better but not mandatory), add it as-is (like "message": "my logline"), if it's not JSON

---

<div class="post-metadata">

**Author:** ![jakelandis](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jakelandis/32/36163_2.png) [@jakelandis](https://discuss.elastic.co/u/jakelandis)\
**Post date:** [October 9, 2018, 8:52am UTC](https://discuss.elastic.co/t/detecting-json-in-ingest-pipeline/151464/2 "2018-10-09T08:52:19Z")

</div>

You should be able to use the on\_failure handler to accomplish this. [https://www.elastic.co/guide/en/elasticsearch/reference/current/handling-failure-in-pipelines.html](https://www.elastic.co/guide/en/elasticsearch/reference/current/handling-failure-in-pipelines.html)

For example:

```auto
PUT _ingest/pipeline/test1
{
  "description": "Ingest pipeline",
  "processors": [
    {
      "grok": {
        "field": "message",
        "patterns": [
          "%{TIMESTAMP_ISO8601:logtime} %{WORD:loglevel} - %{GREEDYDATA:logdata}"
        ]
      }
    },
    {
      "json": {
        "field": "logdata",
        "add_to_root": true,
        "on_failure": [
          {
            "grok": {
              "field": "message",
              "patterns": [
                "%{TIMESTAMP_ISO8601:logtime} %{WORD:loglevel} - %{GREEDYDATA:log_line}"
              ]
            }
          }
        ]
      }
    },
     {
      "remove": {
        "field": [
          "message",
          "logdata"
        ]
      }
    }
  ]
}

POST test/_doc/1?pipeline=test1
{
  "message": """2018-10-08 13:39:36,247 INFO - {"a": "b"}"""
}

POST test/_doc/2?pipeline=test1
{
  "message": "2018-10-08 08:54:18,592 INFO - user 10470684 cannot assume zuid 10470684"
}

GET test/_doc/1
GET test/_doc/2

```

Doc 1 get parsed as JSON and  
Doc 2 handles the JSON failure and results in

```auto
{
  "_index": "test",
  "_type": "_doc",
  "_id": "2",
  "_version": 5,
  "found": true,
  "_source": {
    "loglevel": "INFO",
    "log_line": "user 10470684 cannot assume zuid 10470684",
    "logtime": "2018-10-08 08:54:18,592"
  }
}

```

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [November 6, 2018, 8:52am UTC](https://discuss.elastic.co/t/detecting-json-in-ingest-pipeline/151464/3 "2018-11-06T08:52:21Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.