# Remove plain text message in Logstash file input

**URL:** https://discuss.elastic.co/t/remove-plain-text-message-in-logstash-file-input/325269
**Category:** Logstash
**Created:** [February 10, 2023, 1:24pm UTC](https://discuss.elastic.co/t/remove-plain-text-message-in-logstash-file-input/325269 "2023-02-10T13:24:16Z")
**Posts on this page:** 3
**Page:** 1

<div class="post-metadata">

### Author: ![djrshn2346](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/djrshn2346/32/108594_2.png) [@djrshn2346](https://discuss.elastic.co/u/djrshn2346)
#### Post date: [February 10, 2023, 1:24pm UTC](https://discuss.elastic.co/t/remove-plain-text-message-in-logstash-file-input/325269/1 "2023-02-10T13:24:16Z")

</div>

I am adding filter to remove a plain text as it causes error while JSON parsing.

```auto
file {
        id => "my_lt_log"
        path => "/logs/logtransformer.log"
        type => "log"
        start_position => "beginning"
      }
if [type] == "log" {
        date {
            match => ["timestamp", "yyyy-MM-dd'T'HH:mm:ss.SSSZ"]
            target => "timestamp"
          }
          if [message] =~ /.*JAVA_TOOL_OPTIONS.*/{
            drop { }
            }

        json {
            skip_on_invalid_json => true
            source => "message"
            target => "json"
            remove_field=>["message", "kubernetes"]
            }

```

I will not get any log in Elasticsearch which is fine, but the input file /logs/logtransformer.log shows one error logs:

```auto
{"version": "1.1.0", "timestamp": "2023-02-10T13:14:48.467Z", "severity": "warning", "service_id": "eric-log-transformer", "metadata" : {"namespace": "zyadros", "pod_name": "eric-log-transformer-56b7fdbc6f-4hn45", "node_name": "node-10-63-142-147", "pod_uid": "bfe6ffa6-397f-4785-b91e-54c2b6c37cb5", "container_name": "logtransformer"}, "message": "Error parsing json {:source=>'message', :raw=>'Picked up JAVA_TOOL_OPTIONS: -XX:InitialRAMPercentage=50.000000 -XX:MaxRAMPercentage=75.000000', :exception=>#<LogStash::Json::ParserError: Unrecognized token 'Picked': was expecting (JSON String, Number, Array, Object or token 'null', 'true' or 'false')
 at [Source: (byte[])'Picked up JAVA_TOOL_OPTIONS: -XX:InitialRAMPercentage=50.000000 -XX:MaxRAMPercentage=75.000000'; line: 1, column: 8]>}"}
{"version": "1.1.0", "timestamp": "2023-02-10T13:14:49.382Z", "severity": "warning", "service_id": "eric-log-transformer", "metadata" : {"namespace": "zyadros", "pod_name": "eric-log-transformer-56b7fdbc6f-4hn45", "node_name": "node-10-63-142-147", "pod_uid": "bfe6ffa6-397f-4785-b91e-54c2b6c37cb5", "container_name": "logtransformer"}, "message": "Could not index event to OpenSearch. {:status=>400, :action=>['index', {:_id=>nil, :_index=>'adp-app-logs-2023.02.10', :routing=>nil}, {'@version'=>'1', 'timestamp'=>'%{[json][timestamp]}', 'service_id'=>'%{[json][service_id]}', 'tags'=>['_jsonparsefailure'], 'version'=>'%{[json][version]}', 'message'=>['Picked up JAVA_TOOL_OPTIONS: -XX:InitialRAMPercentage=50.000000 -XX:MaxRAMPercentage=75.000000', '%{[json][message]}'], 'severity'=>'%{[json][severity]}', 'metadata'=>{'container_name'=>'%{[json][metadata][container_name]}', 'node_name'=>'node-10-63-142-147', 'namespace'=>'zyadros', 'pod_name'=>'eric-log-transformer-56b7fdbc6f-4hn45', 'pod_uid'=>'bfe6ffa6-397f-4785-b91e-54c2b6c37cb5'}, 'filename'=>'/logs/metrics.log', '@timestamp'=>2023-02-10T13:14:47.800974Z, 'logplane'=>'adp-app-logs'}], :response=>{'index'=>{'_index'=>'adp-app-logs-2023.02.10', '_type'=>'_doc', '_id'=>'3Q52O4YBCoUR1nQu1R4i', 'status'=>400, 'error'=>{'type'=>'mapper_parsing_exception', 'reason'=>'failed to parse field [timestamp] of type [date] in document with id '3Q52O4YBCoUR1nQu1R4i'. Preview of field's value: '%{[json][timestamp]}'', 'caused_by'=>{'type'=>'illegal_argument_exception', 'reason'=>'failed to parse date field [%{[json][timestamp]}] with format [strict_date_optional_time||epoch_millis]', 'caused_by'=>{'type'=>'date_time_parse_exception', 'reason'=>'date_time_parse_exception: Failed to parse with all enclosed parsers'}}}}}}"}

```

And this actual message without filter is:

```auto
Picked up JAVA_TOOL_OPTIONS: -XX:InitialRAMPercentage=50.000000 -XX:MaxRAMPercentage=75.000000

```

---

<div class="post-metadata">

### Author: ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)
#### Post date: [February 10, 2023, 1:24pm UTC](https://discuss.elastic.co/t/remove-plain-text-message-in-logstash-file-input/325269/2 "2023-02-10T13:24:17Z")

</div>

OpenSearch/OpenDistro are AWS run products and differ from the original Elasticsearch and Kibana products that Elastic builds and maintains. You may need to contact them directly for further assistance.

(This is an automated response from your friendly Elastic bot. Please report this post if you have any suggestions or concerns :elasticheart: )

---

<div class="post-metadata">

### Author: ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)
#### Post date: [March 10, 2023, 1:24pm UTC](https://discuss.elastic.co/t/remove-plain-text-message-in-logstash-file-input/325269/3 "2023-03-10T13:24:50Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.