Hello,
I have a weird issue using the Ingestion Pipeline. When I parse with a single Grok processor, I get all the fields parsed properly (including timestam). Howver, if I had a subsequent "date" processor, I get "[timestamp] not present as part of path [timestamp]".
Here is a test pipleline, tested against docker.elastic.co/elasticsearch/elasticsearch:5.3.3 ; which reproduces what we experience in our 5.3.3 (non-docker) cluster.
This simulattion works perfectly:
curl -u elastic:changeme 'localhost:9200/_ingest/pipeline/_simulate?pretty&verbose' -d'
{
"pipeline": {
"description": "Grok ingestion pipeline nginx logs",
"version": 0,
"processors": [
{
"grok": {
"field": "message",
"trace_match": true,
"patterns": [
"%{COMBINEDAPACHELOG}%{GREEDYDATA:additional_fields}"
]
}
}
]
},
"docs": [
{
"_index": "index",
"_type": "nginx",
"_id": "id",
"_source": {
"message": "1.2.3.4 - - [28/Mar/2018:18:21:44 +0200] \"GET / HTTP/1.1\" 302 213 \"-\" \"Mozilla\""
}
}
]
}'
and yields:
"processor_results" : [
{
"doc" : {
"_id" : "id",
"_type" : "nginx",
"_index" : "index",
"_source" : {
"ident" : "-",
"verb" : "GET",
"additional_fields" : "",
"message" : "1.2.3.4 - - [28/Mar/2018:18:21:44 +0200] \"GET / HTTP/1.1\" 302 213 \"-\" \"Mozilla\"",
"response" : "302",
"httpversion" : "1.1",
"timestamp" : "28/Mar/2018:18:21:44 +0200"
...
ow if I add a date processor, it fails complaining about timestamp not being present (despite being properly extracted above):
curl -u elastic:changeme 'localhost:9200/_ingest/pipeline/_simulate?pretty&verbose' -d'
{
"pipeline": {
"description": "Grok ingestion pipeline nginx logs",
"version": 0,
"processors": [
{
"grok": {
"field": "message",
"trace_match": true,
"patterns": [
"%{COMBINEDAPACHELOG}%{GREEDYDATA:additional_fields}"
]
},
"date": {
"field": "timestamp",
"formats": [
"dd/MMM/yyyy:HH:mm:ss Z"
],
"timezone": "Europe/Paris"
}
}
]
},
"docs": [
{
"_index": "index",
"_type": "nginx",
"_id": "id",
"_source": {
"message": "1.2.3.4 - - [28/Mar/2018:18:21:44 +0200] \"GET / HTTP/1.1\" 302 213 \"-\" \"Mozilla\""
}
}
]
}'
The response is:
{
"docs" : [
{
"processor_results" : [
{
"error" : {
"root_cause" : [
{
"type" : "illegal_argument_exception",
"reason" : "field [timestamp] not present as part of path [timestamp]"
}
],
"type" : "illegal_argument_exception",
"reason" : "field [timestamp] not present as part of path [timestamp]"
}
}
]
}
]
}
Any clue about what is going on ?
Thanks
M