# Script for generating index template and ascii doc

**URL:** <https://discuss.elastic.co/t/script-for-generating-index-template-and-ascii-doc/33756>\
**Category:** Beats\
**Tags:** beats-development\
**Created:** [November 4, 2015, 3:19pm UTC](https://discuss.elastic.co/t/script-for-generating-index-template-and-ascii-doc/33756 "2015-11-04T15:19:52Z")\
**Posts on this page:** 11\
**Page:** 1

<div class="post-metadata">

**Author:** ![mrkschan](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mrkschan/32/5386_2.png) [@mrkschan](https://discuss.elastic.co/u/mrkschan)\
**Post date:** [November 4, 2015, 3:19pm UTC](https://discuss.elastic.co/t/script-for-generating-index-template-and-ascii-doc/33756/1 "2015-11-04T15:19:52Z")

</div>

There are scripts living in topbeat that generates index template and ascii doc. Though, they are not supporting nested data type yet. Is nested data type a good practice to use in ES? If yes, shall they be supported in those script?

I patched the scripts in nginxbeat [1](https://github.com/mrkschan/nginxbeat/commit/18e504a379834c025871d1962ea8592055f0fa70#diff-7bdf48ad8fa7a8a672923017f9213fe6), [2](https://github.com/mrkschan/nginxbeat/commit/3890104b5b9c0fa377e2df32c5e8dd3ac9e029f3#diff-ebf812a25c8b2aa92008bb4df178f929), shall topbeat apply the same set of changes even though it does not require nested data type? I think that can help other beat developers to get started more easily.

---

<div class="post-metadata">

**Author:** ![monica](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/monica/32/3696_2.png) [@monica](https://discuss.elastic.co/u/monica)\
**Post date:** [November 4, 2015, 5:05pm UTC](https://discuss.elastic.co/t/script-for-generating-index-template-and-ascii-doc/33756/2 "2015-11-04T17:05:42Z")

</div>

From what I know, Kibana doesn't support "nested" query. There is an open PR for that: [https://github.com/elastic/kibana/pull/4806](https://github.com/elastic/kibana/pull/4806). As we would like to use Kibana, nested data type was not an option for the Beats.

I see that the generated template for nginxbeat is using nested data type. I am curious to know if you managed to find a way to visualize the data with Kibana or do you use something else for visualization?

In the future, I am thinking to have a common script that generates the template for each Beat, with extended functionality. as I am finding annoying to copy and adjust the script when a new Beat is created. I think it's a good idea if this script supports nested as well 😄

---

<div class="post-metadata">

**Author:** ![steffens](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/steffens/32/79630_2.png) [@steffens](https://discuss.elastic.co/u/steffens)\
**Post date:** [November 5, 2015, 2:06am UTC](https://discuss.elastic.co/t/script-for-generating-index-template-and-ascii-doc/33756/3 "2015-11-05T02:06:56Z")

</div>

I think it is aggregations in kibana not working with nested data, but filtering should work. Right?

---

<div class="post-metadata">

**Author:** ![mrkschan](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mrkschan/32/5386_2.png) [@mrkschan](https://discuss.elastic.co/u/mrkschan)\
**Post date:** [November 5, 2015, 4:26am UTC](https://discuss.elastic.co/t/script-for-generating-index-template-and-ascii-doc/33756/4 "2015-11-05T04:26:29Z")

</div>

I didn't try visualize the nested data type yet 😛 sorry i cannot give any input here. Though, nginx status report needs that or otherwise i have to use dynamic data type (which is discouraged, i believe).

---

<div class="post-metadata">

**Author:** ![ruflin](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ruflin/32/3116_2.png) [@ruflin](https://discuss.elastic.co/u/ruflin)\
**Post date:** [November 5, 2015, 6:52am UTC](https://discuss.elastic.co/t/script-for-generating-index-template-and-ascii-doc/33756/5 "2015-11-05T06:52:06Z")

</div>

@steffens Correct. Nested aggregations don't work. Filtering / querying is not a problem.

---

<div class="post-metadata">

**Author:** ![mrkschan](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mrkschan/32/5386_2.png) [@mrkschan](https://discuss.elastic.co/u/mrkschan)\
**Post date:** [November 5, 2015, 3:12pm UTC](https://discuss.elastic.co/t/script-for-generating-index-template-and-ascii-doc/33756/6 "2015-11-05T15:12:08Z")

</div>

So, I think I better emit multiple events of a type (e.g. upstream) rather than using nested data type to hold them?

---

<div class="post-metadata">

**Author:** ![ruflin](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ruflin/32/3116_2.png) [@ruflin](https://discuss.elastic.co/u/ruflin)\
**Post date:** [November 6, 2015, 11:19am UTC](https://discuss.elastic.co/t/script-for-generating-index-template-and-ascii-doc/33756/7 "2015-11-06T11:19:36Z")

</div>

@mrkschan can you make an example?

---

<div class="post-metadata">

**Author:** ![mrkschan](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mrkschan/32/5386_2.png) [@mrkschan](https://discuss.elastic.co/u/mrkschan)\
**Post date:** [November 6, 2015, 4:23pm UTC](https://discuss.elastic.co/t/script-for-generating-index-template-and-ascii-doc/33756/8 "2015-11-06T16:23:54Z")

</div>

See below the original dynamic data type and a transformed nested data type in an array.

```auto
# Original dynamic data type
        "server_zones": {
          "hg.nginx.org": {
            "discarded": 1732,
            "processing": 0,
            "received": 9404252,
            "requests": 33082,
            "responses": {
                "1xx": 0,
                "2xx": 29893,
                "3xx": 857,
                "4xx": 453,
                "5xx": 147,
                "total": 31350
            },
            "sent": 943707757
          },
          "trac.nginx.org": {
            "discarded": 396,
            "processing": 1,
            "received": 16824065,
            "requests": 47339,
            "responses": {
                "1xx": 0,
                "2xx": 22694,
                "3xx": 21060,
                "4xx": 3061,
                "5xx": 127,
                "total": 46942
            },
            "sent": 771492649
          },
          "lxr.nginx.org": {
            "discarded": 112,
            "processing": 0,
            "received": 888931,
            "requests": 3684,
            "responses": {
                "1xx": 0,
                "2xx": 3361,
                "3xx": 125,
                "4xx": 80,
                "5xx": 6,
                "total": 3572
            },
            "sent": 82231555
          }
        },

# Transformed nested data type
        "server_zones": [{
            "name": "hg.nginx.org",
            "discarded": 1732,
            "processing": 0,
            "received": 9404252,
            "requests": 33082,
            "responses": {
                "1xx": 0,
                "2xx": 29893,
                "3xx": 857,
                "4xx": 453,
                "5xx": 147,
                "total": 31350
            },
            "sent": 943707757
        }, {
            "name": "trac.nginx.org",
            "discarded": 396,
            "processing": 1,
            "received": 16824065,
            "requests": 47339,
            "responses": {
                "1xx": 0,
                "2xx": 22694,
                "3xx": 21060,
                "4xx": 3061,
                "5xx": 127,
                "total": 46942
            },
            "sent": 771492649
        }, {
            "name": "lxr.nginx.org",
            "discarded": 112,
            "processing": 0,
            "received": 888931,
            "requests": 3684,
            "responses": {
                "1xx": 0,
                "2xx": 3361,
                "3xx": 125,
                "4xx": 80,
                "5xx": 6,
                "total": 3572
            },
            "sent": 82231555
        }],

```

---

<div class="post-metadata">

**Author:** ![ruflin](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ruflin/32/3116_2.png) [@ruflin](https://discuss.elastic.co/u/ruflin)\
**Post date:** [November 9, 2015, 8:26am UTC](https://discuss.elastic.co/t/script-for-generating-index-template-and-ascii-doc/33756/9 "2015-11-09T08:26:17Z")

</div>

@mrkschan In your case I would probably go for option 2 as this would allow for more fexibility as the events are sent independent. Like you could send data for a busy host more frequent etc.

---

<div class="post-metadata">

**Author:** ![tudor](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/tudor/32/3753_2.png) [@tudor](https://discuss.elastic.co/u/tudor)\
**Post date:** [November 9, 2015, 9:33am UTC](https://discuss.elastic.co/t/script-for-generating-index-template-and-ascii-doc/33756/10 "2015-11-09T09:33:25Z")

</div>

Right, I think it's better to avoid arrays and create multiple smaller documents. We did the same for Topbeat.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [July 5, 2017, 9:58pm UTC](https://discuss.elastic.co/t/script-for-generating-index-template-and-ascii-doc/33756/11 "2017-07-05T21:58:21Z")

</div>


