# Output to indices based on field in message **URL:** \ **Category:** Logstash\ **Created:** [April 26, 2017, 2:39pm UTC](https://discuss.elastic.co/t/output-to-indices-based-on-field-in-message/83719 "2017-04-26T14:39:08Z")\ **Posts on this page:** 6\ **Page:** 1
**Author:** ![LmYjQ](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/lmyjq/32/97480_2.png) [@LmYjQ](https://discuss.elastic.co/u/LmYjQ)\ **Post date:** [April 26, 2017, 2:39pm UTC](https://discuss.elastic.co/t/output-to-indices-based-on-field-in-message/83719/1 "2017-04-26T14:39:08Z")
I have some logs have same filenames like access\_20170427.log, then content in it is like [{user\_id:123, act:play, timestamp:2017-04-23 08:00:00},{user\_id:456, act:play, timestamp:2017-04-23 08:00:00}] I want to put them into different indices in elasticsearch by the value of user\_id. The index names will be like user\_123, user\_456, user\_789. There are two questions for me: 1. how can I put records into the right index based on the value of user\_id? I guess add a field and use if clause should work, but I really not familiar with the syntax. 2. the set of my user\_id is large as million, is it a bad design to have too many indices. index by act maybe better? index names: play, like, share, fav？ Thank you. ---
**Author:** ![theuntergeek](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/theuntergeek/32/44961_2.png) [@theuntergeek](https://discuss.elastic.co/u/theuntergeek)\ **Post date:** [April 26, 2017, 2:58pm UTC](https://discuss.elastic.co/t/output-to-indices-based-on-field-in-message/83719/2 "2017-04-26T14:58:31Z")
> [@LmYjQ](#): > > how can I put records into the right index based on the value of user\_id? I guess add a field and use if clause should work, but I really not familiar with the syntax. ```auto index => "%{user_id}-%{+YYYY.MM.dd}" ``` This would make daily indices named after `user_id`. More info on [sprintf format in the documentation](https://www.elastic.co/guide/en/logstash/current/event-dependent-configuration.html#sprintf). > [@LmYjQ](#): > > the set of my user\_id is large as million, is it a bad design to have too many indices. index by act maybe better? index names: play, like, share, fav？ After explaining how to do it, now I will tell you to _not_ do something that will create millions of unique indices with few records in them. That's a fast track to watching your Elasticsearch cluster fail. If the records are all similar, and there is no security requirement to isolate them, why separate them at all? Why not just have daily or weekly indices with _all_ of the similar records in them? You can use the new \_rollover API (also available in Curator) to roll over indices when they reach a certain age and/or have a set number of documents in them. ---
**Author:** ![LmYjQ](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/lmyjq/32/97480_2.png) [@LmYjQ](https://discuss.elastic.co/u/LmYjQ)\ **Post date:** [April 27, 2017, 3:20am UTC](https://discuss.elastic.co/t/output-to-indices-based-on-field-in-message/83719/3 "2017-04-27T03:20:30Z")
Thank you Aaron. For my first question, before this > [@theuntergeek](#): > > index =\> "%{user\_id}-%{+YYYY.MM.dd}" all the log content are in a message, I'm not sure how to "parse" them by now. I think I could use the filter:grok to do something like this, with one of the 120 grok-patterns. filter { grok { match =\> { "message" =\> "%{USERNAME:user\_id}" } add\_field =\> { "user\_id" =\> "user\_%{user\_id}" } } } But in fact, there are **several fields have similar pattern**. {user\_id1:123456, user\_id2:456123, video\_id:123456, act:play} which means user1 plays user2's video. Now my input is json from redis like this, > 2017-04-23T16:20:31+08:00 cv.product.access.mobile {"race":"album","video\_id":43633036,"ip":"117.177.78.48","cdn":"[cdn-web-qn.colorv.cn](http://cdn-web-qn.colorv.cn)","act":"update","ad\_type":"AdExchange","agent":"Mozilla/5.0 (Linux; Android 5.1.1; vivo X6SPlus D Build/LMY47V; wv) AppleWebKit/537.36 (KHTML, like Gecko) Version/4.0 Chrome/53.0.2785.49 Mobile MQQBrowser/6.2 TBS/043128 Safari/537.36 MicroMessenger/6.5.7.1041 NetType/WIFI Language/zh\_CN","author\_zone":0,"author\_registered\_at":"2015-08-02 13:06:05","post\_id":0,"published\_at":"","referer":"","reference\_id":0,"sessid":"7e620e610447414bafe5091470f8b0b2","duration":144,"author\_is\_priest":0,"download\_type":"myapp","author\_udid":"d850a4a042d6382","status\_404":"","author\_version":"and-3.6.13-gdt","mold\_id":10006,"url":"[http://video.colorv.cn/play/43633036?from=timeline&isappinstalled=0&from=share","author\_os":"and","page\_kind":"mini","request\_id":"ff6925b4c86b4d34be534a6609edfa2d","referrer\_id":"","author\_id":3934438,"play\_time":60,"method":"GET","published](http://video.colorv.cn/play/43633036?from=timeline&isappinstalled=0&from=share%22,%22author_os%22:%22and%22,%22page_kind%22:%22mini%22,%22request_id%22:%22ff6925b4c86b4d34be534a6609edfa2d%22,%22referrer_id%22:%22%22,%22author_id%22:3934438,%22play_time%22:60,%22method%22:%22GET%22,%22published)":0} How can I distinguish them and extract each key into a field? And for my second question, 1. We will have two applications: one for BI in kibana, the other is construct a recommender system. 2. Our log data for each user will be large, [quote="theuntergeek, post:2, topic:83719"] millions of unique indices with few records in them [/quote]I mean, every click and request in our app(making and sharing short video). So our plan is to create index for each user and video, use timestamp field as filter when quering in BI or collect training sample in recommender system. Your mean this plan will performance bad,but if we index by date, we will have many users, videos, acts in a single "date-index", is it hard for ES to find a record we need? So what is the trade-off here, how to decide the index structure according to our problem? ---
**Author:** ![theuntergeek](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/theuntergeek/32/44961_2.png) [@theuntergeek](https://discuss.elastic.co/u/theuntergeek)\ **Post date:** [April 27, 2017, 4:10am UTC](https://discuss.elastic.co/t/output-to-indices-based-on-field-in-message/83719/4 "2017-04-27T04:10:26Z")
> [@LmYjQ](#): > > How can I distinguish them and extract each key into a field? This should be in its own topic. There are a lot of grok help threads that may already answer this question for you, in fact. > [@LmYjQ](#): > > So our plan is to create index for each user and video, use timestamp field as filter when quering in BI or collect training sample in recommender system. As stated, this is a **Bad Idea™** if you have millions of unique records. That's just too many indices to be able to manage. > [@LmYjQ](#): > > but if we index by date, we will have many users, videos, acts in a single "date-index", is it hard for ES to find a record we need? No, it isn't. Elasticsearch can find it _very_ easily, and within milliseconds. > [@LmYjQ](#): > > So what is the trade-off here, The only real trade-off here is one that _must_ be made. You simply cannot architect your system the way you were originally planning and have it be functional. > [@LmYjQ](#): > > how to decide the index structure according to our problem? That's the million dollar question, now, isn't it? This is also not related to the original topic, so I suggest you start a thread in the Elasticsearch forum area here. There have been many who have already received good answers about mapping their data properly (which is what you just asked for here). ---
**Author:** ![LmYjQ](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/lmyjq/32/97480_2.png) [@LmYjQ](https://discuss.elastic.co/u/LmYjQ)\ **Post date:** [April 27, 2017, 5:34am UTC](https://discuss.elastic.co/t/output-to-indices-based-on-field-in-message/83719/6 "2017-04-27T05:34:54Z")
That's really helpful, thanks a lot! ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [May 25, 2017, 5:48am UTC](https://discuss.elastic.co/t/output-to-indices-based-on-field-in-message/83719/7 "2017-05-25T05:48:26Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.