# \[SOLVED\] Pipeline: field \[timestamp\] not present as part of path \[timestamp\]

**URL:** <https://discuss.elastic.co/t/solved-pipeline-field-timestamp-not-present-as-part-of-path-timestamp/126695>\
**Category:** Elasticsearch\
**Created:** [April 4, 2018, 8:31am UTC](https://discuss.elastic.co/t/solved-pipeline-field-timestamp-not-present-as-part-of-path-timestamp/126695 "2018-04-04T08:31:42Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![leucos](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/leucos/32/7302_2.png) [@leucos](https://discuss.elastic.co/u/leucos)\
**Post date:** [April 4, 2018, 8:31am UTC](https://discuss.elastic.co/t/solved-pipeline-field-timestamp-not-present-as-part-of-path-timestamp/126695/1 "2018-04-04T08:31:43Z")

</div>

Hello,

I have a weird issue using the Ingestion Pipeline. When I parse with a single Grok processor, I get all the fields parsed properly (including timestam). Howver, if I had a subsequent "date" processor, I get "[timestamp] not present as part of path [timestamp]".

Here is a test pipleline, tested against [docker.elastic.co/elasticsearch/elasticsearch:5.3.3](http://docker.elastic.co/elasticsearch/elasticsearch:5.3.3) ; which reproduces what we experience in our 5.3.3 (non-docker) cluster.

This simulattion works perfectly:

```
curl -u elastic:changeme 'localhost:9200/_ingest/pipeline/_simulate?pretty&verbose' -d'
{
    "pipeline": {
        "description": "Grok ingestion pipeline nginx logs",
        "version": 0,
        "processors": [
            {
                "grok": {
                    "field": "message",
                    "trace_match": true,
                    "patterns": [
                        "%{COMBINEDAPACHELOG}%{GREEDYDATA:additional_fields}"
                    ]
                }
            }
        ]
    },
    "docs": [
        {
            "_index": "index",
            "_type": "nginx",
            "_id": "id",
            "_source": {
                "message": "1.2.3.4 - - [28/Mar/2018:18:21:44 +0200] \"GET / HTTP/1.1\" 302 213 \"-\" \"Mozilla\""
            }
        }
    ]
}'

```

and yields:

```
"processor_results" : [
{
    "doc" : {
    "_id" : "id",
    "_type" : "nginx",
    "_index" : "index",
    "_source" : {
        "ident" : "-",
        "verb" : "GET",
        "additional_fields" : "",
        "message" : "1.2.3.4 - - [28/Mar/2018:18:21:44 +0200] \"GET / HTTP/1.1\" 302 213 \"-\" \"Mozilla\"",
        "response" : "302",
        "httpversion" : "1.1",
        "timestamp" : "28/Mar/2018:18:21:44 +0200"
...

```

ow if I add a date processor, it fails complaining about timestamp not being present (despite being properly extracted above):

```
curl -u elastic:changeme 'localhost:9200/_ingest/pipeline/_simulate?pretty&verbose' -d'
{
    "pipeline": {
        "description": "Grok ingestion pipeline nginx logs",
        "version": 0,
        "processors": [
            {
                "grok": {
                    "field": "message",
                    "trace_match": true,
                    "patterns": [
                        "%{COMBINEDAPACHELOG}%{GREEDYDATA:additional_fields}"
                    ]
                },
                "date": {
                    "field": "timestamp",
                    "formats": [
                        "dd/MMM/yyyy:HH:mm:ss Z"
                    ],
                    "timezone": "Europe/Paris"
                }
            }
        ]
    },
    "docs": [
        {
            "_index": "index",
            "_type": "nginx",
            "_id": "id",
            "_source": {
                "message": "1.2.3.4 - - [28/Mar/2018:18:21:44 +0200] \"GET / HTTP/1.1\" 302 213 \"-\" \"Mozilla\""
            }
        }
    ]
}'

```

The response is:

```
{
"docs" : [
    {
    "processor_results" : [
        {
        "error" : {
            "root_cause" : [
            {
                "type" : "illegal_argument_exception",
                "reason" : "field [timestamp] not present as part of path [timestamp]"
            }
            ],
            "type" : "illegal_argument_exception",
            "reason" : "field [timestamp] not present as part of path [timestamp]"
        }
        }
    ]
    }
]
}

```

Any clue about what is going on ?

Thanks

M

---

<div class="post-metadata">

**Author:** ![leucos](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/leucos/32/7302_2.png) [@leucos](https://discuss.elastic.co/u/leucos)\
**Post date:** [April 4, 2018, 11:53am UTC](https://discuss.elastic.co/t/solved-pipeline-field-timestamp-not-present-as-part-of-path-timestamp/126695/2 "2018-04-04T11:53:56Z")

</div>

Ok I figured it out. Each processor is a JSON object and should be enclose by staches ({}).  
Thus, this contruct is wrong!

```
processors": [
        {
            "grok": {
                "field": "message",
                "trace_match": true,
                "patterns": [
                    "%{COMBINEDAPACHELOG}%{GREEDYDATA:additional_fields}"
                ]
            },
            "date": {

```

because date and grok are in the same object. It should look like:

```
{ 
    "grok": ....
},{
    "date": ....
}

```

So the complete (working) simulate example in shown in the original question should look like this:

```
curl -u elastic:changeme -XPOST 'localhost:9200/_ingest/pipeline/_simulate' -H"Content-Type: application/json" -d'
{
    "pipeline": {
        "description": "Grok ingestion pipeline nginx logs",
        "version": 0,
        "processors": [
            {
                "grok": {
                    "field": "message",
                    "trace_match": true,
                    "patterns": [
                        "%{COMBINEDAPACHELOG}%{GREEDYDATA:additional_fields}"
                    ]
                }
            },{
                "date" : {
                    "field" : "timestamp",
                    "formats" : [
                      "dd/MMM/yyyy:HH:mm:ss Z"
                    ],
                    "timezone" : "Europe/Paris"
                  }
                }
            
        ]
    },
    "docs": [
        {
            "_index": "index",
            "_type": "nginx",
            "_id": "id",
            "_source": {
                "message": "1.2.3.4 - - [28/Mar/2018:18:21:44 +0200] \"GET / HTTP/1.1\" 302 213 \"-\" \"Mozilla\""
            }
        }
    ]
}'
```

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [May 2, 2018, 11:53am UTC](https://discuss.elastic.co/t/solved-pipeline-field-timestamp-not-present-as-part-of-path-timestamp/126695/3 "2018-05-02T11:53:58Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.