# I want to import a 3GB system file to ELK in one go **URL:** \ **Category:** Elasticsearch\ **Created:** [July 9, 2020, 4:53pm UTC](https://discuss.elastic.co/t/i-want-to-import-a-3gb-system-file-to-elk-in-one-go/240572 "2020-07-09T16:53:45Z")\ **Posts on this page:** 20\ **Page:** 1
**Author:** ![Deepika\_Rawat](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/deepika_rawat/32/71899_2.png) [@Deepika\_Rawat](https://discuss.elastic.co/u/Deepika_Rawat)\ **Post date:** [July 9, 2020, 4:53pm UTC](https://discuss.elastic.co/t/i-want-to-import-a-3gb-system-file-to-elk-in-one-go/240572/1 "2020-07-09T16:53:45Z")
But it is showing Disk error.I installed graylog now but it is also not working .Will anyone be able to tell me how to read a 3 GB file in ELK or any other free open source software just like ELK if you will be able to tell.kindly suggest on urgent basis ---
**Author:** ![Deepika\_Rawat](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/deepika_rawat/32/71899_2.png) [@Deepika\_Rawat](https://discuss.elastic.co/u/Deepika_Rawat)\ **Post date:** [July 11, 2020, 6:30am UTC](https://discuss.elastic.co/t/i-want-to-import-a-3gb-system-file-to-elk-in-one-go/240572/2 "2020-07-11T06:30:08Z")
If anyone know any answer related to this kindly comment ---
**Author:** ![Vinayak\_Sapre](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/vinayak_sapre/32/45939_2.png) [@Vinayak\_Sapre](https://discuss.elastic.co/u/Vinayak_Sapre)\ **Post date:** [July 11, 2020, 6:50am UTC](https://discuss.elastic.co/t/i-want-to-import-a-3gb-system-file-to-elk-in-one-go/240572/3 "2020-07-11T06:50:59Z")
> [@Deepika\_Rawat](#): > > Will anyone be able to tell me how to read a 3 GB file Can you provide more details on the file format? ---
**Author:** ![Deepika\_Rawat](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/deepika_rawat/32/71899_2.png) [@Deepika\_Rawat](https://discuss.elastic.co/u/Deepika_Rawat)\ **Post date:** [July 11, 2020, 8:12am UTC](https://discuss.elastic.co/t/i-want-to-import-a-3gb-system-file-to-elk-in-one-go/240572/4 "2020-07-11T08:12:40Z")
Hey thnx for the reply I have a system log file of 3GB with data like this ``` 172.16.2.22 Jun 22 10:49:18 date=2020-06-22 local7 notice time=10:49:17 devname="FW_1" devid="FG2TK19907000" logid="000020" type="traffic" subtype="forward" level="notice" vd="root" eventtime=1592803157 srcip=172.16.16.21 srcport=57945 srcintf="VLAN-2" srcintfrole="lan" dstip=52.179.24.121 dstport=443 dstintf="wan2" dstintfrole="wan" poluuid="7b6d31e94-51ea-a540-5761e9fa764f" sessionid=38967804 proto=6 action="accept" policyid=10 policytype="policy" service="HTTPS" dstcountry="United States" srccountry="Reserved" trandisp="snat" transip=49.206.32.147 transport=57945 appid=41469 app="Microsoft.Portal" appcat="Collaboration" apprisk="elevated" applist="default" duration=104665 sentbyte=271556 rcvdbyte=381246 sentpkt=3499 rcvdpkt=1767 sentdelta=310 rcvddelta=432 ``` I tried to import it through ELK but it is giving me disk errors after some time..If not elk then can u suggest me any other free and open source software that can read and apply filers to 3GB file ---
**Author:** ![Vinayak\_Sapre](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/vinayak_sapre/32/45939_2.png) [@Vinayak\_Sapre](https://discuss.elastic.co/u/Vinayak_Sapre)\ **Post date:** [July 11, 2020, 9:02am UTC](https://discuss.elastic.co/t/i-want-to-import-a-3gb-system-file-to-elk-in-one-go/240572/5 "2020-07-11T09:02:23Z")
@Deepika_Rawat What's the exception? Is it on logstash side or ES side. ---
**Author:** ![Deepika\_Rawat](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/deepika_rawat/32/71899_2.png) [@Deepika\_Rawat](https://discuss.elastic.co/u/Deepika_Rawat)\ **Post date:** [July 11, 2020, 9:04am UTC](https://discuss.elastic.co/t/i-want-to-import-a-3gb-system-file-to-elk-in-one-go/240572/6 "2020-07-11T09:04:27Z")
On ES side the error was.... could not index to elastic search but when i split the file into 15MB each it does not show errors ---
**Author:** ![Vinayak\_Sapre](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/vinayak_sapre/32/45939_2.png) [@Vinayak\_Sapre](https://discuss.elastic.co/u/Vinayak_Sapre)\ **Post date:** [July 11, 2020, 9:14am UTC](https://discuss.elastic.co/t/i-want-to-import-a-3gb-system-file-to-elk-in-one-go/240572/7 "2020-07-11T09:14:45Z")
If you have access to ES can u post stacktraces from ES log. It could be batch size and number of concurrent batches. But hard to figure out from just one liner. Also msg in the last post does not say anything about disk. How did u conclude it's a disk error. Any additional info on your pipeline / hardware setup will be useful to debug. ---
**Author:** ![Deepika\_Rawat](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/deepika_rawat/32/71899_2.png) [@Deepika\_Rawat](https://discuss.elastic.co/u/Deepika_Rawat)\ **Post date:** [July 11, 2020, 9:22am UTC](https://discuss.elastic.co/t/i-want-to-import-a-3gb-system-file-to-elk-in-one-go/240572/8 "2020-07-11T09:22:44Z")
okay i will send the errors when i will execute again but it was cluster block exception error something....but just tell me is it posiible to import a 3GB file to elk on one go. ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [July 11, 2020, 9:58am UTC](https://discuss.elastic.co/t/i-want-to-import-a-3gb-system-file-to-elk-in-one-go/240572/9 "2020-07-11T09:58:41Z")
I would recommend using Logstash to read it line by line as it allows to to define filter that to parse the information in the log records. ---
**Author:** ![Deepika\_Rawat](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/deepika_rawat/32/71899_2.png) [@Deepika\_Rawat](https://discuss.elastic.co/u/Deepika_Rawat)\ **Post date:** [July 11, 2020, 10:15am UTC](https://discuss.elastic.co/t/i-want-to-import-a-3gb-system-file-to-elk-in-one-go/240572/10 "2020-07-11T10:15:09Z")
I tried using logstash but it is giving errors after some time of error code 403 ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [July 11, 2020, 10:26am UTC](https://discuss.elastic.co/t/i-want-to-import-a-3gb-system-file-to-elk-in-one-go/240572/11 "2020-07-11T10:26:43Z")
Then try to resolve that problem. Sharing the logs, error message and configuration would allow us to help. ---
**Author:** ![Deepika\_Rawat](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/deepika_rawat/32/71899_2.png) [@Deepika\_Rawat](https://discuss.elastic.co/u/Deepika_Rawat)\ **Post date:** [July 11, 2020, 10:39am UTC](https://discuss.elastic.co/t/i-want-to-import-a-3gb-system-file-to-elk-in-one-go/240572/12 "2020-07-11T10:39:18Z")
okay just tell me what is the maximum amount of size of file that can be imported to ELK on one go ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [July 11, 2020, 10:40am UTC](https://discuss.elastic.co/t/i-want-to-import-a-3gb-system-file-to-elk-in-one-go/240572/13 "2020-07-11T10:40:12Z")
You insert line by line into Elasticsearch, not full files. ---
**Author:** ![Deepika\_Rawat](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/deepika_rawat/32/71899_2.png) [@Deepika\_Rawat](https://discuss.elastic.co/u/Deepika_Rawat)\ **Post date:** [July 11, 2020, 10:42am UTC](https://discuss.elastic.co/t/i-want-to-import-a-3gb-system-file-to-elk-in-one-go/240572/14 "2020-07-11T10:42:18Z")
[I want to import a 3GB system file to ELK in one go](https://discuss.elastic.co/t/i-want-to-import-a-3gb-system-file-to-elk-in-one-go/240572/4) ...these are the logs and configuration file i have used KV filter so they are showing errors ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [July 11, 2020, 10:44am UTC](https://discuss.elastic.co/t/i-want-to-import-a-3gb-system-file-to-elk-in-one-go/240572/15 "2020-07-11T10:44:14Z")
I can see a sample log line but no config. or error messages ---
**Author:** ![Deepika\_Rawat](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/deepika_rawat/32/71899_2.png) [@Deepika\_Rawat](https://discuss.elastic.co/u/Deepika_Rawat)\ **Post date:** [July 11, 2020, 10:47am UTC](https://discuss.elastic.co/t/i-want-to-import-a-3gb-system-file-to-elk-in-one-go/240572/16 "2020-07-11T10:47:14Z")
``` indent preformatted text by 4 spacesinput ``` input { file { path =\>"/root/og00" start\_position =\> "beginning" sincedb\_path =\> "/dev/null" } } filter { kv { } } output { elasticsearch { hosts =\> "[http://localhost:9200](http://localhost:9200)" index =\> "log00\_210270" } stdout { } } ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [July 11, 2020, 10:58am UTC](https://discuss.elastic.co/t/i-want-to-import-a-3gb-system-file-to-elk-in-one-go/240572/17 "2020-07-11T10:58:01Z")
As the first few fields are not in Kv format you can not just use a KV filter. Instead you need to parse out the fields that are not in KV format first and store the long KV string in a separate field. You can then apply the KV filter to this field. Have a look at [this blog post](https://www.elastic.co/blog/a-practical-introduction-to-logstash) for a guide on how to parse data and work with Logstash. ---
**Author:** ![Deepika\_Rawat](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/deepika_rawat/32/71899_2.png) [@Deepika\_Rawat](https://discuss.elastic.co/u/Deepika_Rawat)\ **Post date:** [July 11, 2020, 11:39am UTC](https://discuss.elastic.co/t/i-want-to-import-a-3gb-system-file-to-elk-in-one-go/240572/18 "2020-07-11T11:39:57Z")
hey thankx for the reply...i am new to elk will you be able to write a filter section ...actually i don't know how to start...i will be grateful ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [July 11, 2020, 12:37pm UTC](https://discuss.elastic.co/t/i-want-to-import-a-3gb-system-file-to-elk-in-one-go/240572/19 "2020-07-11T12:37:20Z")
Have a look at the tutorial I linked to. I think that is a good point to start. ---
**Author:** ![Deepika\_Rawat](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/deepika_rawat/32/71899_2.png) [@Deepika\_Rawat](https://discuss.elastic.co/u/Deepika_Rawat)\ **Post date:** [July 13, 2020, 7:51am UTC](https://discuss.elastic.co/t/i-want-to-import-a-3gb-system-file-to-elk-in-one-go/240572/20 "2020-07-13T07:51:09Z")
> [@I want to import a 3GB system file to ELK in one go](https://discuss.elastic.co/t/i-want-to-import-a-3gb-system-file-to-elk-in-one-go/240572/17): > > As the first few fields are not in Kv format you can not just use a KV filter. Instead you need to parse out the fields that are not in KV format first and store the long KV string in a separate field. You can then apply the KV filter to this field. Have a look at [this blog post](https://www.elastic.co/blog/a-practical-introduction-to-logstash) for a guide on how to parse data and work with Logstash. what do you mean a long KV string in one field in this line...i don't want to display the first few fields [Next page](https://discuss.elastic.co/t/i-want-to-import-a-3gb-system-file-to-elk-in-one-go/240572.md?page=2)