# Logstash : Extract values from a String field in Json

**URL:** <https://discuss.elastic.co/t/logstash-extract-values-from-a-string-field-in-json/197349>\
**Category:** Logstash\
**Created:** [August 29, 2019, 1:22pm UTC](https://discuss.elastic.co/t/logstash-extract-values-from-a-string-field-in-json/197349 "2019-08-29T13:22:45Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![chimbu3306](https://avatars.discourse-cdn.com/v4/letter/c/c67d28/32.png) [@chimbu3306](https://discuss.elastic.co/u/chimbu3306)\
**Post date:** [August 29, 2019, 1:22pm UTC](https://discuss.elastic.co/t/logstash-extract-values-from-a-string-field-in-json/197349/1 "2019-08-29T13:22:45Z")

</div>

I have the below Json submitted to my logstash from kinesis where the message string in the json contains all the required values which I need to store it as a separate field in elasticsearch . Currently its stored as a single message string field, I tried using the mutate and json filter but unable to get the results

```
{
"@timestamp": "2019-08-29T13:02:47.468Z",
"id": "34947135803057280532301987232169408483366183409373609984",
"message": "{\"@timestamp\": \"2019-08-29 13:02:47.467\", \"priority\": \"INFO\", \"application\": \"authentication\", \"class\": \"com.ecomm.environment.PropertiesEnvironmentConfig\", \"file\": \"PropertiesEnvironmentConfig.java:287\", \"requestId\": \"\", \"correlationId\": \"\", \"clientCorrelationId\": \"\", \"breadcrumb\": \"\", \"principalId\": \"\", \"customerId\": \"\", \"clientUserAgent\": \"\", \"payload\": {\"logType\":\"EVENT\",\"message\":\"Loaded environment properties\",\"details\":{\"environmentType\":\"sandbox\",\"filename\":\"env.properties\"}} }\n",
"@version": "1",
"messageType": "DATA_MESSAGE",
}

```

I am looking for something like this,

```
{
"@timestamp": "2019-08-29T13:02:47.468Z",
"id": "34947135803057280532301987232169408483366183409373609984",
"payload": { 
    "logType":"EVENT",
    "message":"Loaded environment properties"
},
"environmentType":"sandbox",
"filename":"env.properties",
"priority": "INFO", 
"application": "authentication", 
"class": "com.ecomm.environment.PropertiesEnvironmentConfig", 
"file": "PropertiesEnvironmentConfig.java:287", 
"requestId": "", 
"correlationId": "", 
"clientCorrelationId": "", 
"breadcrumb": "", 
"principalId": "", 
"customerId": "", 
"clientUserAgent": "", 
"@version": "1",
"messageType": "DATA_MESSAGE"
}
```

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [August 29, 2019, 1:35pm UTC](https://discuss.elastic.co/t/logstash-extract-values-from-a-string-field-in-json/197349/2 "2019-08-29T13:35:44Z")

</div>

With a json filter like this

```
json { source => "message" remove_field => ["message"] }

```

I get

```
         "breadcrumb" => "",
            "payload" => {
    "details" => {
               "filename" => "env.properties",
        "environmentType" => "sandbox"
    },
    "logType" => "EVENT",
    "message" => "Loaded environment properties"
},
               "file" => "PropertiesEnvironmentConfig.java:287",
          "requestId" => "",
        "application" => "authentication",
           "priority" => "INFO",
              "class" => "com.ecomm.environment.PropertiesEnvironmentConfig",
        "principalId" => "",
         "customerId" => "",
               "tags" => [
    [0] "_timestampparsefailure"
],
        "_@timestamp" => "2019-08-29 13:02:47.467",
         "@timestamp" => 2019-08-29T13:34:45.040Z,
"clientCorrelationId" => "",
    "clientUserAgent" => "",
      "correlationId" => ""

```

What issues do you have with that?

---

<div class="post-metadata">

**Author:** ![chimbu3306](https://avatars.discourse-cdn.com/v4/letter/c/c67d28/32.png) [@chimbu3306](https://discuss.elastic.co/u/chimbu3306)\
**Post date:** [August 29, 2019, 3:40pm UTC](https://discuss.elastic.co/t/logstash-extract-values-from-a-string-field-in-json/197349/3 "2019-08-29T15:40:22Z")

</div>

I have jsons from kinesis contains just a string in the message field and seeing loads of error in the logs, is there way i can apply the json filter only to valid messages?

```
{
"@timestamp": "2019-08-15:22:03,203Z",
"id": "34947135803057280532301987232169408483366183409373609984",
"message": "END RequestId: 0ef9fae7-6d12-4f35-a861-2ff7fb00bb1d\n",
"@version": "1",
 "messageType": "DATA_MESSAGE",
 }

[2019-08-29T15:22:03,203][WARN][logstash.filters.json] Error parsing json 
{:source=>"message", :raw=>"END RequestId: 0d749912-3dc0-4a52-96ce-5ab94f7de77f\n", 
:exception=>#<LogStash::Json::ParserError: Unrecognized token 'END': was expecting ('true', 
'false' or 'null')
at [Source: (byte[])"END RequestId: 0d749912-3dc0-4a52-96ce-5ab94f7de77f
"; line: 1, column: 5]>}
```

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [August 29, 2019, 4:15pm UTC](https://discuss.elastic.co/t/logstash-extract-values-from-a-string-field-in-json/197349/4 "2019-08-29T16:15:32Z")

</div>

> [@chimbu3306](#):
>
> is there way i can apply the json filter only to valid messages?

Use the [skip\_on\_invalid\_json](https://www.elastic.co/guide/en/logstash/current/plugins-filters-json.html#plugins-filters-json-skip_on_invalid_json) option.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [September 26, 2019, 4:15pm UTC](https://discuss.elastic.co/t/logstash-extract-values-from-a-string-field-in-json/197349/5 "2019-09-26T16:15:36Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.