File input plugin is treating the line as plain string eventhough the input is json

Elastic Stack Logstash
1

File input plugin is treating the line as plain string eventhough the input is json.
Output in Opensearch is :

{
        "_index" : "sample-logs-2023.04.24",
        "_type" : "_doc",
        "_id" : "ui9PsocBICgSyzdw_QSk",
        "_score" : 1.92991,
        "_source" : {
          "@timestamp" : "2023-04-24T08:09:58.358583Z",
          "@version" : "1",
          "message" : "{\"version\":\"1.0.0\",\"severity\":\"info\",\"timestamp\":\"2023-04-24T08:08:53.151Z\",\"service_id\":\"sample\",\"message\":\"sample\",\"metadata\":{\"container_name\":\"sample\",\"namespace\":\"sample\",\"node_name\":\"node-10-63-142-146\",\"pod_name\":\"sample\",\"pod_uid\":\"e861f42f-7899-4c93-b4be-26aa06a46168\"}}",
          "filename" : "/logs/sample.log",
          "logplane" : "sample-logs"
        }
      }

The complete log is coming inside message field.
Input is having:

file {
        id => "lt_log_file"
        path => "/logs/ltproxy.log"
        type => "lt_log"
        start_position => "beginning"
      }

Filter:

if [type] == "lt_log" {
        if [message] =~ /.*JAVA_TOOL_OPTIONS.*/{
          drop { }
          }

Also even in elasticsearch , we are seeing the same behavior.

2

OpenSearch/OpenDistro are AWS run products and differ from the original Elasticsearch and Kibana products that Elastic builds and maintains. You may need to contact them directly for further assistance.

(This is an automated response from your friendly Elastic bot. Please report this post if you have any suggestions or concerns :elasticheart: )

3

It is not clear what is the issue as you didn't share your entire pipeline.

Are you parsing the message with a json filter?

4

no i am not using any kind of json filter.

5

Well, you need to if you want to have the fields of your individual json document as fields in your elasticsearch document.

Try to add a json filter in your pipeline.

json {
    source => "message"
}
6

I tried this earlier but it appended multiple backslashes in the output logs.

7

Impossible to know what you mean without you sharing it.

You output it to the console? If so, this is expected as the double quotes would be escaped.

8

output:

metadata'=>{'container_name'=>'%{[json][metadata][container_name]}'}, 'version'=>'%{[json][version]}', '@timestamp'=>2023-04-24T11:46:09.796967Z, 'message'=>['{\'version\': \'1.1.0\', \'timestamp\': \'2023-04-24T11:46:09.008Z\', \'severity\': \'warning\', \'service_id\': \'eric-log-transformer\', \'metadata\' : {\'namespace\': \'staging2\', \'pod_name\': \'eric-log-transformer-86b7674897-p66sc\', \'node_name\': \'ip-172-31-194-204.eu-north-1.compute.internal\', \'container_name\': \'logtransformer\'}, \'message\': \'Could not index event to OpenSearch. {:status=>400, :action=>['index', {:_id=>nil, :_index=>'adp-app-logs-2023.04.24', :routing=>nil}, {'logplane'=>'adp-app-logs', 'filename'=>'/logs/logtransformer.log', '@version'=>'1', 'timestamp'=>'%{[json][timestamp]}', 'metadata'=>{'container_name'=>'%{[json][metadata][container_name]}'}, 'version'=>'%{[json][version]}', '@timestamp'=>2023-04-24T11:46:08.790699Z, 'message'=>['{\\'version\\': \\'1.1.0\\', \\'timestamp\\': \\'2023-04-24T11:46:08.275Z\\', \\'severity\\': \\'warning\\', \\'service_id\\': \\'eric-log-transformer\\', \\'metadata\\' : {\\'namespace\\': \\'staging2\\', \\'pod_name\\': \\'eric-log-transformer-86b7674897-p66sc\\', \\'node_name\\': \\'ip-172-31-194-204.eu-north-1.compute.internal\\', \\'container_name\\': \\'logtransformer\\'}, \\'message\\': \\'Could not index event to OpenSearch. {:status=>400, :action=>['index', {:_id=>nil, :_index=>'adp-app-logs-2023.04.24', :routing=>nil}, {'logplane'=>'adp-app-logs', 'filename'=>'/logs/logtransformer.log', '@version'=>'1', 'timestamp'=>'%{[json][timestamp]}', 'metadata'=>{'container_name'=>'%{[json][metadata][container_name]}'}, 'version'=>'%{[json][version]}', '@timestamp'=>2023-04-24T11:46:07.778117Z, 'message'=>['{\\\\'version\\\\': \\\\'1.1.0\\\\', \\\\'timestamp\\\\': \\\\'2023-04-24T11:46:07.178Z\\\\', \\\\'severity\\\\': \\\\'warning\\\\', \\\\'service_id\\\\': \\\\'eric-log-transformer\\\\', \\\\'metadata\\\\' : {\\\\'namespace\\\\': \\\\'staging2\\\\', \\\\'pod_name\\\\': \\\\'eric-log-transformer-86b7674897-p66sc\\\\', \\\\'node_name\\\\': \\\\'ip-172-31-194-204.eu-north-1.compute.internal\\\\', \\\\'container_name\\\\': \\\\'logtransformer\\\\'}, \\\\'message\\\\': \\\\'Could not index event to OpenSearch. {:status=>400, :action=>['index', {:_id=>nil, :_index=>'adp-app-logs-2023.04.24', :routing=>nil}, {'logplane'=>'adp-app-logs', 'filename'=>'/logs/logtransformer.log', '@version'=>'1', 'timestamp'=>'%{[json][timestamp]}', 'metadata'=>{'container_name'=>'%{[json][metadata][container_name]}'}, 'version'=>'%{[json][version]}', '@timestamp'=>2023-04-24T11:46:06.697581Z, 'message'=>['{\\\\\\\\'version\\\\\\\\': \\\\\\\\'1.1.0\\\\\\\\', \\\\\\\\'timestamp\\\\\\\\': \\\\\\\\'2023-04-24T11:46:05.973Z\\\\\\\\', \\\\\\\\'severity\\\\\\\\': \\\\\\\\'warning\\\\\\\\', \\\\\\\\'service_id\\\\\\\\': \\\\\\\\'eric-log-transformer\\\\\\\\', \\\\\\\\'metadata\\\\\\\\' : {\\\\\\\\'namespace\\\\\\\\': \\\\\\\\'staging2\\\\\\\\', \\\\\\\\'pod_name\\\\\\\\': \\\\\\\\'eric-log-transformer-86b7674897-p66sc\\\\\\\\', \\\\\\\\'node_name\\\\\\\\': \\\\\\\\'ip-172-31-194-204.eu-north-1.compute.internal\\\\\\\\', \\\\\\\\'container_name\\\\\\\\': \\\\\\\\'logtransformer\\\\\\\\'}, \\\\\\\\'message\\\\\\\\': \\\\\\\\'Could not index event to OpenSearch. {:status=>400, :action=>['index', {:_id=>nil, :_index=>'adp-app-logs-2023.04.24', :routing=>nil}, {'logplane'=>'adp-app-logs', 'filename'=>'/logs/logtransformer.log', '@version'=>'1', 'timestamp'=>'%{[json][timestamp]}', 'metadata'=>{'container_name'=>'%{[json][metadata][container_name]}'}, 'version'=>'%{[json][version]}', '@timestamp'=>2023-04-24T11:46:05.691505Z, 'message'=>['{\\\\\\\\\\\\\\\\'version\\\\\\\\\\\\\\\\': \\\\\\\\\\\\\\\\'1.1.0\\\\\\\\\\\\\\\\', \\\\\\\\\\\\\\\\'timestamp\\\\\\\\\\\\\\\\': \\\\\\\\\\\\\\\\'2023-04-24T11:46:05.175Z\\\\\\\\\\\\\\\\', \\\\\\\\\\\\\\\\'severity\\\\\\\\\\\\\\\\': \\\\\\\\\\\\\\\\'warning\\\\\\\\\\\\\\\\', \\\\\\\\\\\\\\\\'service_id\\\\\\\\\\\\\\\\': \\\\\\\\\\\\\\\\'eric-log-transformer\\\\\\\\\\\\\\\\', \\\\\\\\\\\\\\\\'metadata\\\\\\\\\\\\\\\\' : {\\\\\\\\\\\\\\\\'namespace\\\\\\\\\\\\\\\\': \\\\\\\\\\\\\\\\'staging2\\\\\\\\\\\\\\\\', \\\\\\\\\\\\\\\\'pod_name\\\\\\\\\\\\\\\\': \\\\\\\\\\\\\\\\'eric-log-transformer-86b7674897-p66sc\\\\\\\\\\\\\\\\', \\\\\\\\\\\\\\\\'node_name\\\\\\\\\\\\\\\\': \\\\\\\\\\\\\\\\'ip-172-31-194-204.eu-north-1.compute.internal\\\\\\\\\\\\\\\\', \\\\\\\\\\\\\\\\'container_name\\\\\\\\\\\\\\\\': \\\\\\\\\\\\\\\\'logtransformer\\\\\\\\\\\\\\\\'}, \\\\\\\\\\\\\\\\'message\\\\\\\\\\\\\\\\': \\\\\\\\\\\\\\\\'Could not index event to OpenSearch. {:status=>400, :action=>['index', {:_id=>nil, :_index=>'adp-app-logs-2023.04.24', :routing=>nil}, {'logplane'=>'adp-app-logs', 'filename'=>'/logs/logtransformer.log', '@version'=>'1', 'timestamp'=>'%{[json][timestamp]}', 'metadata'=>{'container_name'=>'%{[json][metadata][container_name]}'}, 'version'=>'%{[json][version]}', '@timestamp'=>2023-04-24T11:46:04.686430Z, 'message'=>['{\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\'version\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\': \\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\'1.1.0\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\', \\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\'timestamp\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\': \\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\'2023-04-24T11:46:04.185Z\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\', \\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\'severity\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\': \\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\'warning\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\', \\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\'service_id\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\': \\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\'eric-log-transformer\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\', \\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\'metadata\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\' : {\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\'namespace\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\': \\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\'staging2\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\', \\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\'pod_name\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\': \\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\'eric-log-transformer-86b7674897-p66sc\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\', \\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\'node_name\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\': \\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\'ip-172-31-194-204.eu-north-1.compute.internal\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\', \\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\'container_name\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\': \\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\'logtransformer\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\'}, \\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\'message\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\': \\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\'Could not index event to OpenSearch.
9

You have some indexing error, but Opensearch is not supported here, you need to check with the Opensearch community.

Could not index event to OpenSearch.

10

After applying json filter only I am getting this Could Not index error, but apart from that I am not getting this error.

11

Yeah, as I said in the previous answer, to have the individual fields in your document you need to parse your message with the json filter.

If after parsing it you are having errors to index the data, you need to investigate it, but as said before, Opensearch is not supported here, you need to check on an opensearch community.

12

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.