# What the correct way to use dynamic object **URL:** \ **Category:** Elasticsearch\ **Created:** [April 26, 2017, 10:36pm UTC](https://discuss.elastic.co/t/what-the-correct-way-to-use-dynamic-object/83777 "2017-04-26T22:36:13Z")\ **Posts on this page:** 9\ **Page:** 1
**Author:** ![laspal](https://avatars.discourse-cdn.com/v4/letter/l/c6cbf5/32.png) [@laspal](https://discuss.elastic.co/u/laspal)\ **Post date:** [April 26, 2017, 10:36pm UTC](https://discuss.elastic.co/t/what-the-correct-way-to-use-dynamic-object/83777/1 "2017-04-26T22:36:13Z")
Hi, My index looks like ``` indexDoc = { "mappings": { "test": { "properties": { "timestamp": { "type": "date", "doc_values": True, "format": "strict_date_optional_time||epoch_millis||dd/MMM/YYYY :HH:mm:ss Z" }, "created_on": { "type": "date", "index": "not_analyzed", "doc_values": True, "format": "date_optional_time" }, "logmsg": { "type": "object", "dynamic": True }, "logtype": { "type": "string", "index": "not_analyzed", "doc_values": True }, } }, "_default_": { "_all": { "enabled": True, "norms": { "enabled": False } }, "properties": { "timestamp": { "type": "date", "doc_values": True, "format": "strict_date_optional_time||epoch_millis||dd/MMM/YYYY:HH:mm:ss Z" } }, "dynamic_templates": [{ "string_fields": { "match": "*", "match_mapping_type": "string", "mapping": { "type": "string", "norms": { "enabled": False }, "fielddata": { "format": "disabled" }, "fields": { "raw": { "type": "string", "index": "not_analyzed", "doc_values": True } } } } }, { "other_fields": { "match": "*", "match_mapping_type": "*", "mapping": { "doc_values": True } } }] } }, "settings": { "refresh_interval": "5s", "number_of_shards": 5, "number_of_replicas": 2 } ``` } I have elastricsearch and kibana only. I am not using logstash. Basically I need to capture full log file in logmsg. In python I can only use 32000 char in the string, so I am breaking my logfile msg into chunks of 32000. my final list looks like ``` output = [{'created_on': 'sometime', 'logtype': 'sometype', logmsg: { '_msg1: 'aaa', _msg2:'bbb', _msg3': 'ccc'}}] ``` I am using bulk api to store the values. Is this the correct way of doing it, or can I achieve this result in some other way. Problem is in kibana discover tab, in selected field I get infinite field. logmsg.\_msg1, logmsg.\_msg2, logmsg.\_msg3--------- ---
**Author:** ![msimos](https://avatars.discourse-cdn.com/v4/letter/m/bb73d2/32.png) [@msimos](https://discuss.elastic.co/u/msimos)\ **Post date:** [April 27, 2017, 12:32am UTC](https://discuss.elastic.co/t/what-the-correct-way-to-use-dynamic-object/83777/2 "2017-04-27T00:32:46Z")
Hi, What result are you trying to achieve? You can either index the field as one long message field. Or potentially break up the message into separate fields depending on what your log message looks like. ---
**Author:** ![laspal](https://avatars.discourse-cdn.com/v4/letter/l/c6cbf5/32.png) [@laspal](https://discuss.elastic.co/u/laspal)\ **Post date:** [April 27, 2017, 2:02am UTC](https://discuss.elastic.co/t/what-the-correct-way-to-use-dynamic-object/83777/3 "2017-04-27T02:02:23Z")
WIll like to index the field as one long message filed. ---
**Author:** ![laspal](https://avatars.discourse-cdn.com/v4/letter/l/c6cbf5/32.png) [@laspal](https://discuss.elastic.co/u/laspal)\ **Post date:** [April 27, 2017, 2:19am UTC](https://discuss.elastic.co/t/what-the-correct-way-to-use-dynamic-object/83777/4 "2017-04-27T02:19:39Z")
How can I have one filed as logmsg instead of haveing logmsg.\_msg\_0, logmsg.\_msg\_1 as multiple field. ---
**Author:** ![laspal](https://avatars.discourse-cdn.com/v4/letter/l/c6cbf5/32.png) [@laspal](https://discuss.elastic.co/u/laspal)\ **Post date:** [April 27, 2017, 3:25pm UTC](https://discuss.elastic.co/t/what-the-correct-way-to-use-dynamic-object/83777/5 "2017-04-27T15:25:08Z")
Please help. I guess I am creating my index wrong. Basically I need to capture full log file as a field, no need to analyze that field. Normally my log file contains some 80k lines. as a result I was breaking the line as creating dynamic field. As a result kibana was timing out in fetching values more then 1 day. How do i create my index correctly. I am using python and doing bulk operation,. ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [April 27, 2017, 3:40pm UTC](https://discuss.elastic.co/t/what-the-correct-way-to-use-dynamic-object/83777/6 "2017-04-27T15:40:46Z")
Why do you need to have the entire log file as a single event? What are you trying to achieve? The usual way to index log files is to do so line by line as separate documents. If there are lines that belong together, these can be grouped together as a single event through multiline processing. ---
**Author:** ![laspal](https://avatars.discourse-cdn.com/v4/letter/l/c6cbf5/32.png) [@laspal](https://discuss.elastic.co/u/laspal)\ **Post date:** [April 27, 2017, 4:50pm UTC](https://discuss.elastic.co/t/what-the-correct-way-to-use-dynamic-object/83777/7 "2017-04-27T16:50:40Z")
I was doing that before, but team want to check full logs as a single event. ---
**Author:** ![msimos](https://avatars.discourse-cdn.com/v4/letter/m/bb73d2/32.png) [@msimos](https://discuss.elastic.co/u/msimos)\ **Post date:** [April 27, 2017, 6:14pm UTC](https://discuss.elastic.co/t/what-the-correct-way-to-use-dynamic-object/83777/8 "2017-04-27T18:14:52Z")
Hi, Logstash already does this for example with the message field. Also Elasticsearch has the \_all field which is effectively a concatenation of all your fields. I am not sure what the issue is with python, I would check your mapping since you should be able to index a single value more then 32k, here's indexing a single field of 50,000: ```auto $ cat bulk.py #!/usr/bin/python from elasticsearch import Elasticsearch value = "" for i in range (0, 50000): value += "a" client = Elasticsearch(hosts=['http://elastic:changeme@localhost:9200']) body = '{ "index" : { "_index" : "test", "_type" : "type1", "_id" : "1" } }\n' body += '{ "field1" : "'+value+'" }\n' response = client.bulk(body=body) print response $ python bulk.py {u'items': [{u'index': {u'status': 200, u'_type': u'type1', u'_shards': {u'successful': 2, u'failed': 0, u'total': 2}, u'_index': u'test', u'_version': 7, u'created': False, u'result': u'updated', u'_id': u'1'}}], u'errors': False, u'took': 22} ``` ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [May 25, 2017, 6:15pm UTC](https://discuss.elastic.co/t/what-the-correct-way-to-use-dynamic-object/83777/9 "2017-05-25T18:15:38Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.