"
},
"original" => "test sending to logstash remotely"
},
"process" => {
"thread" => {
"id" => 64024,
"name" => "MainThread"
},
"name" => "MainProcess",
"pid" => 17356
},
"@timestamp" => 2023-05-18T00:14:00.701Z,
"@version" => "1",
"type" => "python-tcp"
}
```
---
### Author:  [@bsauvage1](https://discuss.elastic.co/u/bsauvage1)
#### Post date: [May 22, 2023, 12:48pm UTC](https://discuss.elastic.co/t/unable-to-parse-message/333635/4 "2023-05-22T12:48:06Z")
Hello. Sorry to ask back on this. Is this is an issue worth pursuing and raising a bug report about, ithe ecs python plugin, or logstash itself ?
Thanks!
---
### Author:  [@leandrojmp](https://discuss.elastic.co/u/leandrojmp)
#### Post date: [May 22, 2023, 12:50pm UTC](https://discuss.elastic.co/t/unable-to-parse-message/333635/5 "2023-05-22T12:50:18Z")
> [@bsauvage1](#):
>
> Hello. Sorry to ask back on this. Is this is an issue worth pursuing and raising a bug report about, ithe ecs python plugin, or logstash itself ?
It is not clear what is your current issue since you didn't share any log error from Logstash.
You need to share logs from Logstash.
---
### Author:  [@bsauvage1](https://discuss.elastic.co/u/bsauvage1)
#### Post date: [May 22, 2023, 1:22pm UTC](https://discuss.elastic.co/t/unable-to-parse-message/333635/6 "2023-05-22T13:22:56Z")
Apologies if it wasn't clear/visible properly, I believe it was the first line in the code section I had sent - please let me know if this is the below you are after:
[2023-05-18T00:14:11,299][WARN][logstash.outputs.elasticsearch][main][6d2fa9664f0cd654c856a248033a8772afbcd07dc5cc5a974c91b14e3aed3707] Could not index event to Elasticsearch. status: 400, action: ["create", {:\_id=\>nil, :\_index=\>"logs-generic-default", :routing=\>nil}, {"message"=\>"test sending to logstash remotely", "ecs"=\>{"version"=\>"1.6.0"}, "log.level"=\>"info", "log"=\>{"logger"=\>" **main**", "origin"=\>{"file"=\>{"line"=\>2931, "name"=\>"snippets.py"}, "function"=\>""}, "original"=\>"test sending to logstash remotely"}, "process"=\>{"thread"=\>{"id"=\>64024, "name"=\>"MainThread"}, "name"=\>"MainProcess", "pid"=\>17356}, "@timestamp"=\>2023-05-18T00:14:00.701Z, "@version"=\>"1", "type"=\>"python-tcp", "data\_stream"=\>{"type"=\>"logs", "dataset"=\>"generic", "namespace"=\>"default"}}], response: {"create"=\>{"\_index"=\>".ds-logs-generic-default-2023.05.15-000001", "\_id"=\>"Gso1LIgBR4WsEcuaAddT", "status"=\>400, "error"=\>{"type"=\>"mapper\_parsing\_exception", "reason"=\>"failed to parse field [process] of type [long] in document with id 'Gso1LIgBR4WsEcuaAddT'. Preview of field's value: '{name=MainProcess, pid=17356, thread={name=MainThread, id=64024}}'", "caused\_by"=\>{"type"=\>"illegal\_argument\_exception", "reason"=\>"Cannot parse object as number"}}}}
And the json input was:
```auto
{
"message" => "test sending to logstash remotely",
"ecs" => {
"version" => "1.6.0"
},
"log.level" => "info",
"log" => {
"logger" => " __main__",
"origin" => {
"file" => {
"line" => 2931,
"name" => "snippets.py"
},
"function" => ""
},
"original" => "test sending to logstash remotely"
},
"process" => {
"thread" => {
"id" => 64024,
"name" => "MainThread"
},
"name" => "MainProcess",
"pid" => 17356
},
"@timestamp" => 2023-05-18T00:14:00.701Z,
"@version" => "1",
"type" => "python-tcp"
}
```
---
### Author:  [@Badger](https://discuss.elastic.co/u/Badger)
#### Post date: [May 22, 2023, 5:33pm UTC](https://discuss.elastic.co/t/unable-to-parse-message/333635/7 "2023-05-22T17:33:01Z")
> [@bsauvage1](#):
>
> "type"=\>"mapper\_parsing\_exception", "reason"=\>"failed to parse field [process] of type [long]
In your event the [process] field is an object containing [process][thread], [process][name] etc. However, elasticsearch expects it to be of type long. That could be because there is a [mapping](https://www.elastic.co/guide/en/elasticsearch/reference/current/mapping.html), either explicitly set, or via a [template](https://www.elastic.co/guide/en/elasticsearch/reference/current/index-templates.html), or through [dynamic mapping](https://www.elastic.co/guide/en/elasticsearch/reference/current/dynamic-field-mapping.html) of a previous event.
It is not a bug.
---
### Author:  [@bsauvage1](https://discuss.elastic.co/u/bsauvage1)
#### Post date: [May 23, 2023, 1:45am UTC](https://discuss.elastic.co/t/unable-to-parse-message/333635/8 "2023-05-23T01:45:39Z")
I am using the ECS template 1.6.0 (using the ecs-logging python library), which specifies [process] as an object with sub-objects.
> **[Introduction | ECS Logging Reference | Elastic](https://www.elastic.co/guide/en/ecs-logging/overview/master/intro.html)**
I am running stack 8.6.2, and as per the doc the pipeline should use by default ecs implied schema v8. Regardless, I added `pipeline.ecs_compatibility: v1` to logstash.yml. Unfortunately it gives the same error as before.
Should I specify the use of the ecs schema differently?
---
### Author:  [@Badger](https://discuss.elastic.co/u/Badger)
#### Post date: [May 23, 2023, 2:46am UTC](https://discuss.elastic.co/t/unable-to-parse-message/333635/9 "2023-05-23T02:46:42Z")
No. I understand that ecs-v1/elasticsearch-8x.json defines [process] as an object containing a bunch of fields (args, arg\_count, [etc.](https://www.elastic.co/guide/en/ecs/8.7/ecs-process.html)) but that does not set the mapping in elasticsearch. To send ECS compatible events to elasticsearch you need the index to have an ECS compatible mapping, and yours doesn't.
Use the [Mapping API](https://www.elastic.co/guide/en/elasticsearch/reference/current/indices-get-mapping.html) and I am confident that it will show you that.
So the question is, how did the index get that mapping? I don't run elasticsearch so I cannot test it, but my reading of the code of the output plugin is that it will attempt to set an ECS compatible template on the index, which, provided none of the fields already exist, would result in a compatible mapping.
Working back from that it suggests that something has already created the [process] field with a mapping type of long (possibly from an event where [process] is just the PID).
Can you switch to another (non currently existing) index name and see if the problem persists?
---
### Author:  [@bsauvage1](https://discuss.elastic.co/u/bsauvage1)
#### Post date: [May 23, 2023, 3:41am UTC](https://discuss.elastic.co/t/unable-to-parse-message/333635/10 "2023-05-23T03:41:55Z")
When I check my existing index (using the Kibana webUI) I see that indeed it maps [process] as a long.
I tried adding `ecs_compatibility => "v1"`in main.conf inside output {elasticsearch {...}} but still the same errors. I notice in the Stack Management GUI that indices are till the same.
Is there an easy way to delete and rebuild the index so that it natively accepts ECS fields definitions, without resorting to some heavy mapping code ?
I looked at the page below and tried to create a new template for index pattern `logs-*-*`, then I filter component templates by ecs and still find quite a few - the only one that seems to have mappings for [process] tree is `.alerts-ecs-mappings` but I'm really unsure what to do here.
> **[Index management in Kibana | Elasticsearch Guide \[8.7\] | Elastic](https://www.elastic.co/guide/en/elasticsearch/reference/current/index-mgmt.html)**
---
### Author:  [@system](https://discuss.elastic.co/u/system)
#### Post date: [June 20, 2023, 3:42am UTC](https://discuss.elastic.co/t/unable-to-parse-message/333635/11 "2023-06-20T03:42:40Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.