# Flume message indexing and search on attributes **URL:** \ **Category:** Elasticsearch\ **Created:** [December 2, 2011, 4:42pm UTC](https://discuss.elastic.co/t/flume-message-indexing-and-search-on-attributes/6054 "2011-12-02T16:42:25Z")\ **Posts on this page:** 4\ **Page:** 1
**Author:** ![Srinivasan\_Subramani](https://avatars.discourse-cdn.com/v4/letter/s/5e9695/32.png) [@Srinivasan\_Subramani](https://discuss.elastic.co/u/Srinivasan_Subramani)\ **Post date:** [December 2, 2011, 4:42pm UTC](https://discuss.elastic.co/t/flume-message-indexing-and-search-on-attributes/6054/1 "2011-12-02T16:42:25Z")
We started playing around with Elastic Search for logs collected via Flume from our servers. ES is awesome and we plan to go productive with the solution wherein Flume is directed to pump input to ES. Question: we want the ability to search via ES on Flume event attributes like priority, custom meta tags and more. Additionally we want to go ascending or descending based on time stamps or the nano seconds that flume provides. Given that Flume dumps Json we believe it should be good for ES. In order to achieve search on attributes and ordering, do we need any special mapping ? Can some one on the list please enlighten on how to setup ES config so we can achieve the above. Thanks in advance. Cheers Srini ---
**Author:** ![Clinton\_Gormley](https://avatars.discourse-cdn.com/v4/letter/c/50afbb/32.png) [@Clinton\_Gormley](https://discuss.elastic.co/u/Clinton_Gormley)\ **Post date:** [December 3, 2011, 10:10am UTC](https://discuss.elastic.co/t/flume-message-indexing-and-search-on-attributes/6054/2 "2011-12-03T10:10:31Z")
Hi Srini > Question: we want the ability to search via ES on Flume event > attributes like priority, custom meta tags and more. Additionally we > want to go ascending or descending based on time stamps or the nano > seconds that flume provides. > > Given that Flume dumps Json we believe it should be good for ES. In > order to achieve search on attributes and ordering, do we need any > special mapping ? Can some one on the list please enlighten on how to > setup ES config so we can achieve the above. ES does its best to guess what type of data each field contains (the first time it sees the new field), eg for a doc with: { "title": "Foo", "count": 5, "live": true, "date": "2011-12-03 12:00:00" } ...ES would correctly identify: - title: full text string - count: long - live: boolean - date: datetime However, with: { "count": "10", "status": "ACTIVE", "tags": ["foo","bar-baz"] } ... it would identify all of these as full-text strings, which probably isn't what you want. "count" should be a number, "status" and "tags" should be type "string", but with {"index": "not\_analyzed"} so that you can search for the exact term "STATUS" and not have it match "Status", and searching for "bar" shouldn't match "foo-bar". To avoid these errors, you should predefine your mappings. ES makes it easy to try things out by just inserting docs. You can use the 'get mapping' API to see how ES has mapped each field. You can use this mapping info to build your own correct mapping, which you can specify when you create the index. clint ---
**Author:** ![Srinivasan\_Subramani](https://avatars.discourse-cdn.com/v4/letter/s/5e9695/32.png) [@Srinivasan\_Subramani](https://discuss.elastic.co/u/Srinivasan_Subramani)\ **Post date:** [December 3, 2011, 10:17am UTC](https://discuss.elastic.co/t/flume-message-indexing-and-search-on-attributes/6054/3 "2011-12-03T10:17:59Z")
Hi Clint Thanks a lot for that. I will check this out and mail back in case I need further hints. Cheers Srini Sent from my BlackBerry® smartphone -----Original Message----- From: Clinton Gormley [clint@traveljury.com](mailto:clint@traveljury.com) Sender: [elasticsearch@googlegroups.com](mailto:elasticsearch@googlegroups.com) Date: Sat, 03 Dec 2011 11:10:31 To: [elasticsearch@googlegroups.com](mailto:elasticsearch@googlegroups.com) Reply-To: [elasticsearch@googlegroups.com](mailto:elasticsearch@googlegroups.com) Subject: Re: Flume message indexing and search on attributes Hi Srini > Question: we want the ability to search via ES on Flume event > attributes like priority, custom meta tags and more. Additionally we > want to go ascending or descending based on time stamps or the nano > seconds that flume provides. > > Given that Flume dumps Json we believe it should be good for ES. In > order to achieve search on attributes and ordering, do we need any > special mapping ? Can some one on the list please enlighten on how to > setup ES config so we can achieve the above. ES does its best to guess what type of data each field contains (the first time it sees the new field), eg for a doc with: { "title": "Foo", "count": 5, "live": true, "date": "2011-12-03 12:00:00" } ...ES would correctly identify: - title: full text string - count: long - live: boolean - date: datetime However, with: { "count": "10", "status": "ACTIVE", "tags": ["foo","bar-baz"] } ... it would identify all of these as full-text strings, which probably isn't what you want. "count" should be a number, "status" and "tags" should be type "string", but with {"index": "not\_analyzed"} so that you can search for the exact term "STATUS" and not have it match "Status", and searching for "bar" shouldn't match "foo-bar". To avoid these errors, you should predefine your mappings. ES makes it easy to try things out by just inserting docs. You can use the 'get mapping' API to see how ES has mapped each field. You can use this mapping info to build your own correct mapping, which you can specify when you create the index. clint ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 3:46am UTC](https://discuss.elastic.co/t/flume-message-indexing-and-search-on-attributes/6054/4 "2017-07-06T03:46:33Z")