# Regarding creating index pattern for .txt log files **URL:** \ **Category:** Logstash\ **Created:** [April 9, 2019, 8:09am UTC](https://discuss.elastic.co/t/regarding-creating-index-pattern-for-txt-log-files/175979 "2019-04-09T08:09:17Z")\ **Posts on this page:** 13\ **Page:** 1
**Author:** ![Kalyan\_MB](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kalyan_mb/32/42974_2.png) [@Kalyan\_MB](https://discuss.elastic.co/u/Kalyan_MB)\ **Post date:** [April 9, 2019, 8:09am UTC](https://discuss.elastic.co/t/regarding-creating-index-pattern-for-txt-log-files/175979/1 "2019-04-09T08:09:17Z")
Hello, i have below requirement to achieve can anyone help me with this? 1.Need to scan performance logs from a directory(client), and parse them to ELK stack(server) 1. create index patterns using lostash.conf file for the same 2. use this index pattern and create graphs. 3. once these graphs are saved, is there any way to update to the same index name when new log files are scanned and index pattern created with same name? **My log files look like below.** * * * ``` total used free shared buff/cache available ``` ## Mem: 128732 41030 44215 40285 43486 46630 Swap: 12287 0 12287 ## **filebeat.yml file** filebeat.prospectors: - input\_type: log paths: # - /home/vankata/190\_APS\_QUALIFICATION/kalyan\_elk\_logs/free\_data\*.txt - /home/vankata/190\_APS\_QUALIFICATION/Audproc\_cscf11/free\_data\*.txt encoding: utf-8 fields\_under\_root: true document\_type: log fields: service\_name: cfx app\_name: cfx\_perf\_logs * * * ## **logstash.config** input { beats { port =\> 5044 } } filter { dissect { mapping =\> {'message' =\> '%{mem\_type} %{total\_mem} %{used\_mem} %{free\_mem} %{shared\_mem} %{cache\_mem} %{availablemem}'} } } output { if [service\_name] == "cfx" { elasticsearch { #path =\> "/var/log/sdl\_logs/%{vnf\_id}/%{vm\_type}/%{instance\_id}/%{service\_name}/%{app\_name}_%{+yyyy-MM-dd-HH}.log" #codec =\> line { format =\> "%{message}" } #gzip =\> true hosts =\> ["[http://x.x.x.x:9200](http://x.x.x.x:9200)"] index =\> "perform1" } } if [service\_name] == "telemetry-agent" { file{ path =\> "/var/log/sdl\_logs/%{vnf\_id}/%{vm\_type}/%{instance\_id}/%{service\_name}/%{app\_name}_%{+yyyy-MM-dd-HH}.log" codec =\> line { format =\> "%{message}" } #gzip =\> true } } } * * * ---
**Author:** ![BKG](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/bkg/32/43591_2.png) [@BKG](https://discuss.elastic.co/u/BKG)\ **Post date:** [April 9, 2019, 8:54am UTC](https://discuss.elastic.co/t/regarding-creating-index-pattern-for-txt-log-files/175979/2 "2019-04-09T08:54:35Z")
Its a very big question. If I was you, I would check every step individually. You can load filebeat from the command line with `-e -d "*"` flag to show the results of harvesting the file to ensure it's configuration is working perfectly. Then you can run Logstash from command line which will let you see the incoming logs from filebeat and whether they fields are mapped correctly and whether Logstash can communicate with elasticsearch. Good luck! ---
**Author:** ![Kalyan\_MB](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kalyan_mb/32/42974_2.png) [@Kalyan\_MB](https://discuss.elastic.co/u/Kalyan_MB)\ **Post date:** [April 9, 2019, 9:02am UTC](https://discuss.elastic.co/t/regarding-creating-index-pattern-for-txt-log-files/175979/3 "2019-04-09T09:02:42Z")
@BKG i understand its big question!, thanks for your time, here is the output of binary runs. | 2019-04-09T13:15:53.952+0530 | DEBUG | [prospector] | log/prospector.go:362 | Check file for harvesting: /home/vankata/190\_APS\_QUALIFICATION/Audproc\_cscf11/free\_data\_11\_09\_03\_40.txt | | --- | --- | --- | --- | --- | | 2019-04-09T13:15:53.952+0530 | DEBUG | [prospector] | log/prospector.go:448 | Update existing file for harvesting: /home/vankata/190\_APS\_QUALIFICATION/Audproc\_cscf11/free\_data\_11\_09\_03\_40.txt, offset: 204 | | 2019-04-09T13:15:53.952+0530 | DEBUG | [prospector] | log/prospector.go:502 | File didn't change: /home/vankata/190\_APS\_QUALIFICATION/Audproc\_cscf11/free\_data\_11\_09\_03\_40.txt | from above what i can make out is, since file content is not changed harvesting is done ? output { if [service\_name] == "cfx" { elasticsearch { #path =\> "/var/log/sdl\_logs/%{vnf\_id}/%{vm\_type}/%{instance\_id}/%{service\_name}/%{app\_name}\_%{+yyyy-MM-dd-HH}.log" #codec =\> line { format =\> "%{message}" } #gzip =\> true hosts =\> ["[http://x.x.x.x:9200](http://x.x.x.x:9200)"] index =\> "perform1" in the logstash i am giving output to kibana? so i wont be able to see the formatted output file in ELK setup, correct me if i am wrong. thanks in advance ---
**Author:** ![BKG](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/bkg/32/43591_2.png) [@BKG](https://discuss.elastic.co/u/BKG)\ **Post date:** [April 9, 2019, 9:25am UTC](https://discuss.elastic.co/t/regarding-creating-index-pattern-for-txt-log-files/175979/4 "2019-04-09T09:25:32Z")
Is the file getting written to often enough to test? You could set up a script to add to the file for testing purposes. Something like: `while true; echo "my,test,csv,data,goes,here" >> /home/vankata/190_APS_QUALIFICATION/Audproc_cscf11/free_data_11_09_03_40.txt; sleep 5; done;` ---
**Author:** ![Kalyan\_MB](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kalyan_mb/32/42974_2.png) [@Kalyan\_MB](https://discuss.elastic.co/u/Kalyan_MB)\ **Post date:** [April 9, 2019, 9:34am UTC](https://discuss.elastic.co/t/regarding-creating-index-pattern-for-txt-log-files/175979/5 "2019-04-09T09:34:04Z")
right now the directory where i am working has fixed log files, i will change few files contents and try.. ---
**Author:** ![Kalyan\_MB](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kalyan_mb/32/42974_2.png) [@Kalyan\_MB](https://discuss.elastic.co/u/Kalyan_MB)\ **Post date:** [April 9, 2019, 12:36pm UTC](https://discuss.elastic.co/t/regarding-creating-index-pattern-for-txt-log-files/175979/6 "2019-04-09T12:36:14Z")
Hi @BKG, below is the first line of log input. ``` total used free shared buff/cache available Mem: 128732 44366 32347 40869 46018 42704 ``` used filter in logstash: filter { dissect { mapping =\> {'message' =\> '%{mem\_type} %{total\_mem} %{used\_mem} %{free\_mem} %{shared\_mem} %{cache\_mem} %{availablemem}'} } what i see in kibana dashboard Discover tab: ![image](https://us1.discourse-cdn.com/elastic/original/3X/e/b/ebaf2ad97bdb7ef22c8fb3c3c78beeb79b6447a4.png) is this filter index output correctly mapped or should i use other type of filter, please let me know. ---
**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\ **Post date:** [April 9, 2019, 12:43pm UTC](https://discuss.elastic.co/t/regarding-creating-index-pattern-for-txt-log-files/175979/7 "2019-04-09T12:43:16Z")
> [@Kalyan\_MB](#): > > dissect { > mapping =\> {'message' =\> '%{mem\_type} %{total\_mem} %{used\_mem} %{free\_mem} %{shared\_mem} %{cache\_mem} %{availablemem}'} > } That tells dissect to expect a single space between the entries on the line. Your input has multiple spaces. dissect can handle that using -\> in the field name. ``` dissect { mapping => {'message' => '%{mem_type->} %{total_mem->} %{used_mem->} %{free_mem->} %{shared_mem->} %{cache_mem->} %{availablemem}'} } ``` ---
**Author:** ![Kalyan\_MB](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kalyan_mb/32/42974_2.png) [@Kalyan\_MB](https://discuss.elastic.co/u/Kalyan_MB)\ **Post date:** [April 9, 2019, 12:45pm UTC](https://discuss.elastic.co/t/regarding-creating-index-pattern-for-txt-log-files/175979/8 "2019-04-09T12:45:16Z")
> [@Badger](#): > > put has m @Badger thanks, i will apply this change and try! ---
**Author:** ![Kalyan\_MB](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kalyan_mb/32/42974_2.png) [@Kalyan\_MB](https://discuss.elastic.co/u/Kalyan_MB)\ **Post date:** [April 9, 2019, 2:13pm UTC](https://discuss.elastic.co/t/regarding-creating-index-pattern-for-txt-log-files/175979/9 "2019-04-09T14:13:28Z")
keeping below as the filter: filter { dissect { mapping =\> {'message' =\> '%{mem\_type-\>} %{total\_mem-\>} %{used\_mem-\>} %{free\_mem-\>} %{shared\_mem-\>} %{cache\_mem-\>} %{availablemem-\>}'} } } and input file as: ``` total used free shared buff/cache available Mem: 128132 42349 38321 40870 46062 42720 Swap: 12387 0 12287 ``` what i could see in kibana after index creation is: ![image](https://us1.discourse-cdn.com/elastic/original/3X/2/4/245e3f414013ffc26781c3575b585c96781c6b04.png) here for mem\_type: Mem - total\_mem field is missing ![image](https://us1.discourse-cdn.com/elastic/original/3X/b/f/bfadb148daf9acc927114164116449566c0183c5.png) here for mem\_type:swap - free\_mem is empty can you please comment on this? ---
**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\ **Post date:** [April 9, 2019, 2:46pm UTC](https://discuss.elastic.co/t/regarding-creating-index-pattern-for-txt-log-files/175979/10 "2019-04-09T14:46:09Z")
> [@Kalyan\_MB](#): > > Swap: 12387 0 12287 I am really surprised you do not get a \_dissectfailure for that message. You could make the dissect conditional ``` if [message] =~ /Mem:/ { dissect { mapping => {'message' => '%{mem_type->} %{total_mem->} %{used_mem->} %{free_mem->} %{shared_mem->} %{cache_mem->} %{availablemem}'} } } else if [message] =~ /Swap:/ { dissect { mapping => {'message' => '%{mem_type->} %{total_mem->} %{used_mem->} %{free_mem->}'} } } ``` Or you could use conditional fields in a grok filter ``` grok { match => { "message" => "%{WORD:mem_type}:\s+%{NUMBER:total_mem:int}\s+%{NUMBER:used_mem:int}\s+%{NUMBER:free_mem:int}(\s+%{NUMBER:shared_mem:int}\s+%{NUMBER:cache_mem:int}\s+%{NUMBER:availablemem:int})?" } } ``` Note the use of ( )? to make the last 3 fields optional. ---
**Author:** ![Kalyan\_MB](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kalyan_mb/32/42974_2.png) [@Kalyan\_MB](https://discuss.elastic.co/u/Kalyan_MB)\ **Post date:** [April 10, 2019, 5:04am UTC](https://discuss.elastic.co/t/regarding-creating-index-pattern-for-txt-log-files/175979/11 "2019-04-10T05:04:03Z")
> [@Badger](#): > > if [message] =~ /Mem:/ { dissect { mapping =\> {'message' =\> '%{mem\_type-\>} %{total\_mem-\>} %{used\_mem-\>} %{free\_mem-\>} %{shared\_mem-\>} %{cache\_mem-\>} %{availablemem}'} } } else if [message] =~ /Swap:/ { dissect { mapping =\> {'message' =\> '%{mem\_type-\>} %{total\_mem-\>} %{used\_mem-\>} %{free\_mem-\>}'} } } thank you 🙂 @Badger ---
**Author:** ![Kalyan\_MB](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kalyan_mb/32/42974_2.png) [@Kalyan\_MB](https://discuss.elastic.co/u/Kalyan_MB)\ **Post date:** [April 10, 2019, 7:00am UTC](https://discuss.elastic.co/t/regarding-creating-index-pattern-for-txt-log-files/175979/12 "2019-04-10T07:00:58Z")
> [@Kalyan\_MB](#): > > hosts =\> ["[http://x.x.x.x:9200](http://x.x.x.x:9200)"] > index =\> "perform1" as i am giving output to kibana as above, will i be able to locate the index file in the ELK stack machine? if yes, where can i locate in case of .tar package installation on RHEL. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [May 8, 2019, 7:00am UTC](https://discuss.elastic.co/t/regarding-creating-index-pattern-for-txt-log-files/175979/13 "2019-05-08T07:00:59Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.