# Streamed Logs not loaded into Kibana

**URL:** <https://discuss.elastic.co/t/streamed-logs-not-loaded-into-kibana/194052>\
**Category:** Kibana\
**Created:** [August 6, 2019, 4:09pm UTC](https://discuss.elastic.co/t/streamed-logs-not-loaded-into-kibana/194052 "2019-08-06T16:09:05Z")\
**Posts on this page:** 17\
**Page:** 1

<div class="post-metadata">

**Author:** ![ben.sharp](https://avatars.discourse-cdn.com/v4/letter/b/f08c70/32.png) [@ben.sharp](https://discuss.elastic.co/u/ben.sharp)\
**Post date:** [August 6, 2019, 4:09pm UTC](https://discuss.elastic.co/t/streamed-logs-not-loaded-into-kibana/194052/1 "2019-08-06T16:09:05Z")

</div>

Hi there,

I am attempting to stream logs from AWS Lambda into Elasticsearch/Kibana.

I am doing so via a POST to the Elasticsearch endpoint, + the index and document type (e.g. POST [https://example-es-domain.com/alblogs/alb-access-logs/](https://example-es-domain.com/alblogs/alb-access-logs/)).

I have verified that the lambda is sending out the request by sending to requestbin, and have verified that the information contained in each doc is correct.

However I can't see any of the information sent to Elasticsearch being loaded into Kibana, and was wondering if I was missing something.

The full flow for the operation is:

ALB -\> S3 -\> AWS Lambda -\> Elasticsearch

---

<div class="post-metadata">

**Author:** ![christophilus](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christophilus/32/42991_2.png) [@christophilus](https://discuss.elastic.co/u/christophilus)\
**Post date:** [August 6, 2019, 4:49pm UTC](https://discuss.elastic.co/t/streamed-logs-not-loaded-into-kibana/194052/2 "2019-08-06T16:49:25Z")

</div>

There are a lot of possible things that could be going wrong, but it seems most likely to me to be something in your network configuration isn't allowing the request through to Elasticsearch. I'd start by looking at your router configuration.

---

<div class="post-metadata">

**Author:** ![ben.sharp](https://avatars.discourse-cdn.com/v4/letter/b/f08c70/32.png) [@ben.sharp](https://discuss.elastic.co/u/ben.sharp)\
**Post date:** [August 6, 2019, 4:52pm UTC](https://discuss.elastic.co/t/streamed-logs-not-loaded-into-kibana/194052/3 "2019-08-06T16:52:53Z")

</div>

Hi Chris,

Thanks for getting back to me.

I'm using the integrated AWS service, and after doing some troubleshooting set the access to open, so as far as I know there shouldn't be any issues communicating with Elasticsearch.

---

<div class="post-metadata">

**Author:** ![wylie](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/wylie/32/81794_2.png) [@wylie](https://discuss.elastic.co/u/wylie)\
**Post date:** [August 6, 2019, 6:19pm UTC](https://discuss.elastic.co/t/streamed-logs-not-loaded-into-kibana/194052/4 "2019-08-06T18:19:25Z")

</div>

Have you verified that the documents are in your Elasticsearch index? You could use the CAT endpoints: [https://www.elastic.co/guide/en/elasticsearch/reference/current/cat-indices.html](https://www.elastic.co/guide/en/elasticsearch/reference/current/cat-indices.html)

Have you created an Index Pattern mapping in Kibana?

---

<div class="post-metadata">

**Author:** ![ben.sharp](https://avatars.discourse-cdn.com/v4/letter/b/f08c70/32.png) [@ben.sharp](https://discuss.elastic.co/u/ben.sharp)\
**Post date:** [August 7, 2019, 8:46am UTC](https://discuss.elastic.co/t/streamed-logs-not-loaded-into-kibana/194052/5 "2019-08-07T08:46:42Z")

</div>

Hi Wylie,

The index I am attempting to create doesn't exist in Elasticsearch, so I'm assuming the documents aren't being ingested by Elasticsearch. As such I can't create an Index Pattern.

---

<div class="post-metadata">

**Author:** ![ben.sharp](https://avatars.discourse-cdn.com/v4/letter/b/f08c70/32.png) [@ben.sharp](https://discuss.elastic.co/u/ben.sharp)\
**Post date:** [August 7, 2019, 8:50am UTC](https://discuss.elastic.co/t/streamed-logs-not-loaded-into-kibana/194052/6 "2019-08-07T08:50:33Z")

</div>

I've just tried a manual POST via Postman to the url, and have now gotten this error:

```
{
    "error": {
        "root_cause": [
            {
                "type": "mapper_parsing_exception",
                "reason": "failed to parse field [timestamp] of type [date] in document with id 'TOBDa2wBWM5KlmuOLUeH'"
            }
        ],
        "type": "mapper_parsing_exception",
        "reason": "failed to parse field [timestamp] of type [date] in document with id 'TOBDa2wBWM5KlmuOLUeH'",
        "caused_by": {
            "type": "illegal_argument_exception",
            "reason": "Invalid format: \"2019-08-06T15:54:46.701974Z\" is malformed at \".701974Z\""
        }
    },
    "status": 400
}

```

So this looks to be the reason why the documents aren't being accepted into Elasticsearch.

The timestamp field being passed in the request is

> "timestamp": "2019-08-06T15:54:46.701974Z",

Could you help me to understand what's wrong with this format? I believe "YYYY-MM-DDTHH:mm:ss.ZZ" format should be supported?

I tried creating the index and mapping the format by running the following before inserting the documents as well, but am still getting the same error:

```
PUT alb-access-logs-
{
    "mappings": {
      "alb-access-logs": {
        "properties": {
          "timestamp": {
            "type": "date",
            "format": "yyyy-MM-dd'T'HH:mm:ss.ZZ"
          }
      }
    }
  }
}

```

---

<div class="post-metadata">

**Author:** ![wylie](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/wylie/32/81794_2.png) [@wylie](https://discuss.elastic.co/u/wylie)\
**Post date:** [August 7, 2019, 5:14pm UTC](https://discuss.elastic.co/t/streamed-logs-not-loaded-into-kibana/194052/8 "2019-08-07T17:14:26Z")

</div>

Your dates look like they are in the `date_nanos` format, not in `date` format: [https://www.elastic.co/guide/en/elasticsearch/reference/7.x/date\_nanos.html](https://www.elastic.co/guide/en/elasticsearch/reference/7.x/date_nanos.html)

You have two options:

- Change your input to be in date format, which is fully supported in Kibana
- Use date\_nanos which is partially supported in Kibana 7.3 and forward [due to differences between Javascript and Java number implementations](https://github.com/elastic/kibana/issues/40183)

I believe date\_nanos is supported in Discover and Visualize, but other products will round to the nearest millisecond on display.

---

<div class="post-metadata">

**Author:** ![ben.sharp](https://avatars.discourse-cdn.com/v4/letter/b/f08c70/32.png) [@ben.sharp](https://discuss.elastic.co/u/ben.sharp)\
**Post date:** [August 8, 2019, 10:09am UTC](https://discuss.elastic.co/t/streamed-logs-not-loaded-into-kibana/194052/9 "2019-08-08T10:09:07Z")

</div>

Thanks Wylie.

I'd like to keep the data I'm trying to load in the same if possible, so I've tried to set up an index with the mapping timestamp to type date\_nanos:

```
PUT nanos-alb-access-logs-
{
    "mappings": {
      "alb-access-logs": {
        "properties": {
          "timestamp": {
            "type": "date_nanos",
            "format": "yyyy-MM-dd'T'HH:mm:ss.ZZ"
          }
      }
    }
  }
}

```

However I'm getting an error saying that I need a handler on the field timestamp? Could you help me to format my console command correctly?

```
{
  "error": {
    "root_cause": [
      {
        "type": "mapper_parsing_exception",
        "reason": "No handler for type [date_nanos] declared on field [timestamp]"
      }
    ],
    "type": "mapper_parsing_exception",
    "reason": "Failed to parse mapping [alb-access-logs]: No handler for type [date_nanos] declared on field [timestamp]",
    "caused_by": {
      "type": "mapper_parsing_exception",
      "reason": "No handler for type [date_nanos] declared on field [timestamp]"
    }
  },
  "status": 400
}
```

---

<div class="post-metadata">

**Author:** ![wylie](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/wylie/32/81794_2.png) [@wylie](https://discuss.elastic.co/u/wylie)\
**Post date:** [August 8, 2019, 3:17pm UTC](https://discuss.elastic.co/t/streamed-logs-not-loaded-into-kibana/194052/10 "2019-08-08T15:17:51Z")

</div>

Are you running Elasticsearch 7.0 or greater?

---

<div class="post-metadata">

**Author:** ![ben.sharp](https://avatars.discourse-cdn.com/v4/letter/b/f08c70/32.png) [@ben.sharp](https://discuss.elastic.co/u/ben.sharp)\
**Post date:** [August 9, 2019, 8:38am UTC](https://discuss.elastic.co/t/streamed-logs-not-loaded-into-kibana/194052/11 "2019-08-09T08:38:23Z")

</div>

I have just checked and I'm using 6.7, so I guess I will need to look at changing the data I'm importing.

Thanks for all your help.

---

<div class="post-metadata">

**Author:** ![matw](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/matw/32/13913_2.png) [@matw](https://discuss.elastic.co/u/matw)\
**Post date:** [August 9, 2019, 3:40pm UTC](https://discuss.elastic.co/t/streamed-logs-not-loaded-into-kibana/194052/12 "2019-08-09T15:40:40Z")

</div>

you could try the following

```
PUT alb-access-logs
{
     "mappings": {
        "properties": {
            "timestamp": {
              "type": "date"
           }
      }
   }
}

```

Test with demo data

```
 POST alb-access-logs/_doc
 {
    "timestamp": "2019-08-06T15:54:46.701974Z"
 }

```

You will loose the microseconds for sorting (milliseconds are kept), but since \_source equals your input, the textual information is kept.

```
GET alb-access-logs/_search
{
  "sort" : [
      {"timestamp" : {"order" : "asc", "mode" : "avg"}}
   ]
}

```

I'm in a hurry, just tested with 7.3, but should work with 6.7 also

---

<div class="post-metadata">

**Author:** ![ben.sharp](https://avatars.discourse-cdn.com/v4/letter/b/f08c70/32.png) [@ben.sharp](https://discuss.elastic.co/u/ben.sharp)\
**Post date:** [August 9, 2019, 4:06pm UTC](https://discuss.elastic.co/t/streamed-logs-not-loaded-into-kibana/194052/13 "2019-08-09T16:06:14Z")

</div>

> You will loose the microseconds for sorting

That's not a concern for me, so if I can get this working with just the "date" type format that would be a good outcome for me.

I just gave this a go but unfortunately got the following error:

```
{
  "error": {
    "root_cause": [
      {
        "type": "mapper_parsing_exception",
        "reason": "Root mapping definition has unsupported parameters: [timestamp : {type=date}]"
      }
    ],
    "type": "mapper_parsing_exception",
    "reason": "Failed to parse mapping [properties]: Root mapping definition has unsupported parameters: [timestamp : {type=date}]",
    "caused_by": {
      "type": "mapper_parsing_exception",
      "reason": "Root mapping definition has unsupported parameters: [timestamp : {type=date}]"
    }
  },
  "status": 400
}

```

---

<div class="post-metadata">

**Author:** ![matw](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/matw/32/13913_2.png) [@matw](https://discuss.elastic.co/u/matw)\
**Post date:** [August 9, 2019, 5:18pm UTC](https://discuss.elastic.co/t/streamed-logs-not-loaded-into-kibana/194052/14 "2019-08-09T17:18:47Z")

</div>

ah, request needs to be adapted for ES 6.7! will come back to you

---

<div class="post-metadata">

**Author:** ![ben.sharp](https://avatars.discourse-cdn.com/v4/letter/b/f08c70/32.png) [@ben.sharp](https://discuss.elastic.co/u/ben.sharp)\
**Post date:** [August 12, 2019, 10:27am UTC](https://discuss.elastic.co/t/streamed-logs-not-loaded-into-kibana/194052/15 "2019-08-12T10:27:35Z")

</div>

> [@matw](#):
>
> ah, request needs to be adapted for ES 6.7! will come back to you

Thanks!

---

<div class="post-metadata">

**Author:** ![matw](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/matw/32/13913_2.png) [@matw](https://discuss.elastic.co/u/matw)\
**Post date:** [August 12, 2019, 2:51pm UTC](https://discuss.elastic.co/t/streamed-logs-not-loaded-into-kibana/194052/16 "2019-08-12T14:51:28Z")

</div>

Please try the following, tested locally in 6.8, I've used you index + type naming, which is in your case identical.

PUT a pattern for you index

```
PUT alb-access-logs
{
  "mappings": {
    "alb-access-logs": {
      "properties": {
        "timestamp": {
          "type": "date"
        }
      }
    }
  }
}

```

POST an example

```
POST alb-access-logs/alb-access-logs
 {
    "timestamp": "2019-08-06T15:54:46.701974Z"
 }

```

Let's check the results:

```
 GET alb-access-logs/_search 
{
  "sort" : [
      {"timestamp" : {"order" : "asc", "mode" : "avg"}}
   ]
}

```

this should work

---

<div class="post-metadata">

**Author:** ![ben.sharp](https://avatars.discourse-cdn.com/v4/letter/b/f08c70/32.png) [@ben.sharp](https://discuss.elastic.co/u/ben.sharp)\
**Post date:** [August 12, 2019, 5:29pm UTC](https://discuss.elastic.co/t/streamed-logs-not-loaded-into-kibana/194052/17 "2019-08-12T17:29:56Z")

</div>

This is now working, thank you!

I am going to test with Lambda and will update with the results there.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [September 9, 2019, 5:30pm UTC](https://discuss.elastic.co/t/streamed-logs-not-loaded-into-kibana/194052/18 "2019-09-09T17:30:10Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.