# Stripping text from a JSON file

**URL:** <https://discuss.elastic.co/t/stripping-text-from-a-json-file/79428>\
**Category:** Logstash\
**Created:** [March 21, 2017, 2:11pm UTC](https://discuss.elastic.co/t/stripping-text-from-a-json-file/79428 "2017-03-21T14:11:02Z")\
**Posts on this page:** 7\
**Page:** 1

<div class="post-metadata">

**Author:** ![Toontje](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/toontje/32/16100_2.png) [@Toontje](https://discuss.elastic.co/u/Toontje)\
**Post date:** [March 21, 2017, 2:11pm UTC](https://discuss.elastic.co/t/stripping-text-from-a-json-file/79428/1 "2017-03-21T14:11:03Z")

</div>

Hi all!

I have a valid JSON structure that is prepended with a fixed string and then a number. Something like:

```
RRR This is the leading string created at 01-01-2017 at 12:34:56:
{ 
     valid JSON 
}

```

So there is a fixes string "RRR This is the ...." and a variable part, the date and time. Ah, and BTW, the string is always the same length.  
How do i use FileBeats or LogStash to strip off that leading string so i only get valid JSON in ES?

This is what i tried so far:

```
   input {
    beats {
        port => "5043"
    }
}

filter {
    mutate {
        gsub => ["message", "^(.*){", ""]
    }

    json {
        source => "data"
    }

    date {
        match => ["receivedTime", "UNIX"]
        target => "@timestamp"
    }
}

output {
  elasticsearch {
  hosts => ["localhost:9200"]
  index => filebeat
  }
}

```

and i keep getting the following message:

```
2017/03/21 18:26:01.644365 json.go:34: ERR Error decoding JSON: invalid character 'R' looking for beginning of value
2017/03/21 18:26:01.644428 json.go:34: ERR Error decoding JSON: json: cannot unmarshal number into Go value of type map[string]interface {}

```

Thanks,

Ton.

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [March 22, 2017, 6:22am UTC](https://discuss.elastic.co/t/stripping-text-from-a-json-file/79428/2 "2017-03-22T06:22:27Z")

</div>

Do not configure Filebeat to treat the input as JSON, because it isn't JSON. Do make sure to configure the multiline feature so the lines are joined into a single logical message.

I don't see how the `source` option for your json filter could make sense. Where does the `data` field come from?

---

<div class="post-metadata">

**Author:** ![Toontje](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/toontje/32/16100_2.png) [@Toontje](https://discuss.elastic.co/u/Toontje)\
**Post date:** [March 22, 2017, 9:13am UTC](https://discuss.elastic.co/t/stripping-text-from-a-json-file/79428/3 "2017-03-22T09:13:20Z")

</div>

Ok, removed the JSON from FileBeat, "data" is the top field in my JSON document. If all my log entries are on the same line, do i need the multiline?

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [March 22, 2017, 11:22am UTC](https://discuss.elastic.co/t/stripping-text-from-a-json-file/79428/4 "2017-03-22T11:22:32Z")

</div>

> If all my log entries are on the same line, do i need the multiline?

No.

---

<div class="post-metadata">

**Author:** ![Toontje](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/toontje/32/16100_2.png) [@Toontje](https://discuss.elastic.co/u/Toontje)\
**Post date:** [March 22, 2017, 6:34pm UTC](https://discuss.elastic.co/t/stripping-text-from-a-json-file/79428/5 "2017-03-22T18:34:25Z")

</div>

Remember i'm still a newbie. After fighting a lot with regex i just solved it like this:

```
filter {
    grok {
        patterns_dir => ["./patterns"]
        match => { "message" => "%{DATA:todelete}: %{GREEDYDATA:message}" }
        remove_field => "todelete"
        overwrite => ["message"]
    }
   json {
       source => "message"
   }
   date {
       match => ["receivedTime", "UNIX_MS"]
   }
}

```

This works as i want it to work. Now i face the "Objects in arrays are not well supported" issue, but i will open another topic for this.

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [March 22, 2017, 8:08pm UTC](https://discuss.elastic.co/t/stripping-text-from-a-json-file/79428/6 "2017-03-22T20:08:26Z")

</div>

Replace `%{DATA:todelete}` with `%{DATA}` and you won't need `remove_field`. You can also use a mutate filter's gsub option to trim the `message` field in-place.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [April 19, 2017, 8:08pm UTC](https://discuss.elastic.co/t/stripping-text-from-a-json-file/79428/7 "2017-04-19T20:08:28Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.