# Parsed JSON and Grokked Strings in the same pipeline

**URL:** <https://discuss.elastic.co/t/parsed-json-and-grokked-strings-in-the-same-pipeline/146211>\
**Category:** Logstash\
**Created:** [August 27, 2018, 4:11pm UTC](https://discuss.elastic.co/t/parsed-json-and-grokked-strings-in-the-same-pipeline/146211 "2018-08-27T16:11:43Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![isaacsanders](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/isaacsanders/32/34805_2.png) [@isaacsanders](https://discuss.elastic.co/u/isaacsanders)\
**Post date:** [August 27, 2018, 4:11pm UTC](https://discuss.elastic.co/t/parsed-json-and-grokked-strings-in-the-same-pipeline/146211/1 "2018-08-27T16:11:43Z")

</div>

The flow of data in my system is described below:

1. Files on my filesystem, some log, some jsonlog
2. Filebeats  
a. Human readable log lines  
b. JSON log lines
3. Logstash  
a. grok

Something is coercing the JSON object from my jsonlog lines into a format that looks like a Ruby hash.

I would like to be able to handle the different data paths dynamically. I don't think I need multiple pipelines or filebeat configurations, but I could be wrong.

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [August 27, 2018, 8:01pm UTC](https://discuss.elastic.co/t/parsed-json-and-grokked-strings-in-the-same-pipeline/146211/2 "2018-08-27T20:01:05Z")

</div>

> Something is coercing the JSON object from my jsonlog lines into a format that looks like a Ruby hash.

How so? Where are you seeing this?

---

<div class="post-metadata">

**Author:** ![isaacsanders](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/isaacsanders/32/34805_2.png) [@isaacsanders](https://discuss.elastic.co/u/isaacsanders)\
**Post date:** [August 27, 2018, 9:32pm UTC](https://discuss.elastic.co/t/parsed-json-and-grokked-strings-in-the-same-pipeline/146211/3 "2018-08-27T21:32:16Z")

</div>

I was seeing it in the output generated by Logstash.

I also managed to fix the problem. I don't really understand how I did it, I've just found the jumble of incantations that outputs things the way I want them to be.

I'm not really sure this actually works either, but somehow it has managed to work...

This is my filebeat configuration:

```auto
filebeat:
  inputs:
  - type: log
    recursive_glob:
      enabled: true
    paths: /var/log/**/*.log
  - type: log
    recursive_glob:
      enabled: true
    paths: /var/log/**/*.jsonlog
    processors:
    - rename: # this must be broken or incorrect, because I both don't want it to happen and it isn't happening...
        fields:
          from: "message"
          to: "message_json"

logging:
  files:
    name: filebeat.log

output:
  logstash:
    hosts:
    - # hostname

```

This is my logstash configuration:

```auto
input {
  beats {
    id => "input.beats.dev"
    port => 5044
    add_field => { environment => "dev" }
  }
}

filter {
  mutate {
    rename => { "source" => "source_path" }
    copy => { "message" => "unparsed" }
  }
  grok {
    match => {
      "message" => [
        # A list of grok patterns
      ]
    }
    overwrite => ["message"]
  }
  if "_grokparsefailure" in [tags] {
    json {
      source => "unparsed"
      skip_on_invalid_json => true
    }
  }
  json {
    source => "metrics_json"
    remove_field => ["metrics_json"]
  }
  date {
    match => ["timestamp", "ISO8601", "yyyy-MM-dd HH:mm:ss"]
    remove_field => ["timestamp"]
  }
  date {
    match => ["timestamp", "ISO8601", "yyyy-MM-dd HH:mm:ss"]
    remove_field => ["timestamp"]
  }
  mutate {
    uppercase => ["level"]
    remove_field => ["unparsed"]
  }
}

output {
  stdout { codec => json_lines }
}

```

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [September 24, 2018, 9:33pm UTC](https://discuss.elastic.co/t/parsed-json-and-grokked-strings-in-the-same-pipeline/146211/4 "2018-09-24T21:33:47Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.