# Error in elk **URL:** \ **Category:** Elasticsearch\ **Created:** [July 13, 2020, 11:30am UTC](https://discuss.elastic.co/t/error-in-elk/240956 "2020-07-13T11:30:43Z")\ **Posts on this page:** 13\ **Page:** 1
**Author:** ![Deepika\_Rawat](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/deepika_rawat/32/71899_2.png) [@Deepika\_Rawat](https://discuss.elastic.co/u/Deepika_Rawat)\ **Post date:** [July 13, 2020, 11:30am UTC](https://discuss.elastic.co/t/error-in-elk/240956/1 "2020-07-13T11:30:43Z")
I tried to import using kv filter my logs started importing like this.. indent preformatted text by 4 spaces "authserver" =\> "a\_India RADIUS", "proto" =\> "6", "devname" =\> "FW\_1", "10:56:12\tdate" =\> "2020-06-22\tlocal7\tnotice\t\ttime=10:56:11", "host" =\> "kali", "dstintf" =\> "wan1", "path" =\> "/root/Cybrotech-/log00", "subtype" =\> "webfilter", "srcintf" =\> "ssl.root", "method" =\> "domain", "eventtype" =\> "ftgd\_allow", "hostname" =\> "[webmail.accessarellc.net](http://webmail.accessarellc.net)", "cat" =\> "33", "srcintfrole" =\> "undefined", "dstip" =\> "20.73.98.154", "type" =\> "utm", "sessionid" =\> "677535", "dstintfrole" =\> "wan", "srcport" =\> "6095", "url" =\> "/", "profile" =\> "monitor-all", "srcip" =\> "10.212.134.190", "logid" =\> "07013312", "policyid" =\> "17", "eventtime" =\> "12803571", "direction" =\> "outgoing", "level" =\> "notice", "@version" =\> "1", "reqtype" =\> "direct", "catdesc" =\> ""Health", "action" =\> "passthrough", "vd" =\> "root", "dstport" =\> "443", "service" =\> "HTTPS", "@timestamp" =\> 2020-07-13T09:10:47.811Z, "sentbyte" =\> "192", "devid" =\> "FG0TK19907000", "group" =\> "SSLVPN\_Group", "msg" =\> "URL belongs to an allowed category in policy", "user" =\> "\ASINGH", "rcvdbyte" =\> "0" } after some time i got this error on screen ```auto indent preformatted text by 4 spaces ``` "\_type"=\>"doc", "\_id"=\>"LzhxR3MBoH6QvDEw21Sy", "status"=\>400, "error"=\>{"type"=\>"illegal\_argument\_exception", "reason"=\>"Limit of total fields [1000] in index [log00\_210270] has been exceeded"}}}} ---
**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\ **Post date:** [July 13, 2020, 9:02pm UTC](https://discuss.elastic.co/t/error-in-elk/240956/2 "2020-07-13T21:02:12Z")
Take a look at [https://www.elastic.co/guide/en/elasticsearch/reference/current/mapping.html#mapping-limit-settings](https://www.elastic.co/guide/en/elasticsearch/reference/current/mapping.html#mapping-limit-settings) ---
**Author:** ![Deepika\_Rawat](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/deepika_rawat/32/71899_2.png) [@Deepika\_Rawat](https://discuss.elastic.co/u/Deepika_Rawat)\ **Post date:** [July 14, 2020, 10:35am UTC](https://discuss.elastic.co/t/error-in-elk/240956/3 "2020-07-14T10:35:15Z")
@warkolm okay so that means it is not possible to import data with that much indexes into elastic search.. ---
**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\ **Post date:** [July 14, 2020, 10:36am UTC](https://discuss.elastic.co/t/error-in-elk/240956/4 "2020-07-14T10:36:23Z")
It's possible. This topic might help you understand a bit more on what's happening - [Approaches to deal with "Limit of total fields [1000] in index has been exceeded"](https://discuss.elastic.co/t/approaches-to-deal-with-limit-of-total-fields-1000-in-index-has-been-exceeded/241039)? ---
**Author:** ![Deepika\_Rawat](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/deepika_rawat/32/71899_2.png) [@Deepika\_Rawat](https://discuss.elastic.co/u/Deepika_Rawat)\ **Post date:** [July 14, 2020, 10:54am UTC](https://discuss.elastic.co/t/error-in-elk/240956/5 "2020-07-14T10:54:32Z")
@warkolm okay thank you...so i am trying to increase my index mapping through index setting indent preformatted text by 4 spaces ```auto PUT my_index/_settings { "index.mapping.total_fields.limit": 2000 } ``` as i don't get how to use dynamic mapping and flattened data structure because i have a configuration file with only kv filter in filter section ---
**Author:** ![Deepika\_Rawat](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/deepika_rawat/32/71899_2.png) [@Deepika\_Rawat](https://discuss.elastic.co/u/Deepika_Rawat)\ **Post date:** [July 14, 2020, 11:04am UTC](https://discuss.elastic.co/t/error-in-elk/240956/6 "2020-07-14T11:04:54Z")
@warkolm...this is the filter section of my configuration file for the above logs indent preformatted text by 4 spaces filter { kv { remove\_field =\> ["message"] } } I am new to elastic search i have no idea where to define flattened data type...if you can help through this it would be grateful...as i had tried all the options and searched a lot about this error but i can't find anything ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [July 14, 2020, 12:34pm UTC](https://discuss.elastic.co/t/error-in-elk/240956/7 "2020-07-14T12:34:31Z")
It seems all of your message is not in KV format. This gives you a field name that contains a time (4th from the top), which rapidly increases the number of fields. ---
**Author:** ![Deepika\_Rawat](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/deepika_rawat/32/71899_2.png) [@Deepika\_Rawat](https://discuss.elastic.co/u/Deepika_Rawat)\ **Post date:** [July 14, 2020, 1:21pm UTC](https://discuss.elastic.co/t/error-in-elk/240956/9 "2020-07-14T13:21:11Z")
@Christian_Dahlqvist i have my log like this 172.16.3.254 Jun 22 11:00:40 date=2020-06-22 local7 notice time=11:00:39 devname="LR\_FW\_1" devid="FG20TK19907000" logid="00000013" type="traffic" subtype="forward" level="notice" vd="root" eventtime=1592803839 srcip=10.22.134.155 srcport=5596 srcintf="ssl.root" srcintfrole="undefined" dstip=10.10.11.20 dstport=53 dstintf="AHCP-MIBLR\_ACT" dstintfrole="undefined" poluuid="f73d12a2-622c-51ea-c684-4ec26e252a5c" sessionid=39680340 proto=17 action="accept" user="MIA\abhishy" group="SSLVPN\_Group" authserver="Mia\_Ina RADIUS" policyid=18 policytype="policy" service="DNS" dstcountry="Reserved" srccountry="Reserved" trandisp="noop" duration=180 sentbyte=60 rcvdbyte=153 sentpkt=1 rcvdpkt=1 vpn="AHCP-MLR\_ACT" vpntype="ipsec-static" appcat="unscanned" ---
**Author:** ![Deepika\_Rawat](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/deepika_rawat/32/71899_2.png) [@Deepika\_Rawat](https://discuss.elastic.co/u/Deepika_Rawat)\ **Post date:** [July 14, 2020, 1:22pm UTC](https://discuss.elastic.co/t/error-in-elk/240956/10 "2020-07-14T13:22:29Z")
will you able to help me how to write a filter section for this log...i have a 7 GB something file @warkolm @Christian_Dahlqvist ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [July 14, 2020, 1:27pm UTC](https://discuss.elastic.co/t/error-in-elk/240956/11 "2020-07-14T13:27:04Z")
Did you look at the tutorial I linked to in your other thread? ---
**Author:** ![Deepika\_Rawat](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/deepika_rawat/32/71899_2.png) [@Deepika\_Rawat](https://discuss.elastic.co/u/Deepika_Rawat)\ **Post date:** [July 14, 2020, 1:32pm UTC](https://discuss.elastic.co/t/error-in-elk/240956/12 "2020-07-14T13:32:30Z")
@Christian_Dahlqvist yes i tried to change my conf file and added grok filter but it is not working with kv filter ---
**Author:** ![Deepika\_Rawat](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/deepika_rawat/32/71899_2.png) [@Deepika\_Rawat](https://discuss.elastic.co/u/Deepika_Rawat)\ **Post date:** [July 15, 2020, 6:34am UTC](https://discuss.elastic.co/t/error-in-elk/240956/13 "2020-07-15T06:34:45Z")
@Christian_Dahlqvist ...I changed my filter section by this now indent preformatted text by 4 spaces filter { grok { match =\> { "message" =\> "%{IP:client}%{SYSLOGBASE}+%{GREEDYDATA:msgbody}"} } grok { match =\> { "msgbody" =\> ["%{GREEDYDATA:KV}\s"] } } kv { source =\> "msgbody" field\_split =\> " " value\_split =\> "=" remove\_char\_key =\> "\<\>," remove\_char\_value =\> "\<\>," trim\_key =\> "\<\>," trim\_value =\> "\<\>," include\_brackets =\> false remove\_field =\> ["message"] } I hope now it is matching to the logs..just check once ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [August 12, 2020, 6:34am UTC](https://discuss.elastic.co/t/error-in-elk/240956/14 "2020-08-12T06:34:47Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.