# ELK setup for Apache Application Logs **URL:** \ **Category:** Beats\ **Tags:** filebeat\ **Created:** [May 21, 2018, 8:43pm UTC](https://discuss.elastic.co/t/elk-setup-for-apache-application-logs/132697 "2018-05-21T20:43:54Z")\ **Posts on this page:** 7\ **Page:** 1
**Author:** ![rdurai](https://avatars.discourse-cdn.com/v4/letter/r/dbc845/32.png) [@rdurai](https://discuss.elastic.co/u/rdurai)\ **Post date:** [May 21, 2018, 8:43pm UTC](https://discuss.elastic.co/t/elk-setup-for-apache-application-logs/132697/1 "2018-05-21T20:43:54Z")
Hello, another newbie here. I am bit confused about setting up the ELK for Apache2 application log aggregation. My setup is as follows, all components are version 6.2.3; Apache Logs(3X) --\> Filebeat --\> Logstash --\> ElasticSearch + Kibana I am able to see data in kibana, and by adding "multiline.pattern: '^[[:space:]]|^Caused by" in filebeat.yml the lines in stack trace are all together. My problem is with the rest of the lines in an event. They seem to split into separate events and they appear in random order in Kibana. Now I am trying to enable apache2 module, which is supposed keep all lines in an event together. I am not sure if I am doing this right. Added following lines to filebeat.yml #========================== Modules configuration ============================ filebeat.modules: #------------------------------- Apache2 Module ------------------------------ module: apache2 Access logs access: enabled: true Set custom paths for the log files. If left empty, Filebeat will choose the paths depending on your OS. var.paths: ["/opt/apache-tomcat-8.0.46/logs/catalina.out"] error: enabled: true Questions; Is the section 'filebeat.prospects:' is still required, which section is "var.paths" picked up from? Is "multiline.pattern: '^[[:space:]]|^Caused by'" still required? Are there an example configuration available out there for apache log collection. I am looking for exact files to be modified and any help with what should be modified. Filebeat was installed from RPM. My filebeat installation is here-\>\>\> /usr/share/filebeat/bin/filebeat So far I have been tinkering with this file only-\>\>\> /etc/filebeat/filebeat.yml Thanks for your help. ---
**Author:** ![harshbajaj16](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/harshbajaj16/32/44970_2.png) [@harshbajaj16](https://discuss.elastic.co/u/harshbajaj16)\ **Post date:** [May 22, 2018, 3:47am UTC](https://discuss.elastic.co/t/elk-setup-for-apache-application-logs/132697/2 "2018-05-22T03:47:40Z")
Hi, > [@](#): > > Questions; > Is the section 'filebeat.prospects:' is still required, which section is "var.paths" picked up from? > Is "multiline.pattern: '\[1\]|^Caused by'" still required? No, there is no prospector required for the same and below is the format for apache2 modules which is working fine at my end. and also check indentation in your filebeat.yml properly. ![image](https://us1.discourse-cdn.com/elastic/original/3X/4/5/45f80e6a4fa1182b270367936971b286a70b43e6.png) Please refer this and let me know if you are still getting errors. Please paste you full config file here with filebeat logs if getting any error. Regards, * * * 1. [:space:] ---
**Author:** ![rdurai](https://avatars.discourse-cdn.com/v4/letter/r/dbc845/32.png) [@rdurai](https://discuss.elastic.co/u/rdurai)\ **Post date:** [May 22, 2018, 4:44pm UTC](https://discuss.elastic.co/t/elk-setup-for-apache-application-logs/132697/3 "2018-05-22T16:44:55Z")
Hi Harsh, thanks for the pointers. I tried this, the data path is establish. I can see the apache logs in Kibana. BUT each line in a stack trace appears as a separate event. My filebeat.yml file is below. A few things I tried here are Alternating between 'access logs' and 'error logs'. Comment and uncomment - multiline.pattern, multiline.negate, multiline.match Nothing seems to help. My goal is; We have events that start with one of the following headers [GD-WEB], [GD-BACKGROUND], [GD-IMP], [GD-SYNC], [GD-SLR-INDX]. I would like all lines after any one of these header to stay together as one event. Until the next header is reached I hope apache2 module in filebeat can keep the lines in stack trace and any other lines that make up an event together. Thanks for your help again. -rD #========================== Modules configuration ============================ filebeat.modules: #------------------------------- Apache2 Module ------------------------------ - module: apache2 - type: log ---
**Author:** ![harshbajaj16](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/harshbajaj16/32/44970_2.png) [@harshbajaj16](https://discuss.elastic.co/u/harshbajaj16)\ **Post date:** [May 23, 2018, 4:09am UTC](https://discuss.elastic.co/t/elk-setup-for-apache-application-logs/132697/4 "2018-05-23T04:09:12Z")
Hi, > [@](#): > > We have events that start with one of the following headers > [GD-WEB], [GD-BACKGROUND], [GD-IMP], [GD-SYNC], [GD-SLR-INDX]. I would like all lines after any one of these header to stay together as one event. Until the next header is reached Please refer below links for the multi line pattern matching with examples. [https://www.elastic.co/guide/en/beats/filebeat/master/\_examples\_of\_multiline\_configuration.html](https://www.elastic.co/guide/en/beats/filebeat/master/_examples_of_multiline_configuration.html) > **[Manage multiline messages | Filebeat Reference \[8.11\] | Elastic](https://www.elastic.co/guide/en/beats/filebeat/current/multiline-examples.html)** Regards, ---
**Author:** ![rdurai](https://avatars.discourse-cdn.com/v4/letter/r/dbc845/32.png) [@rdurai](https://discuss.elastic.co/u/rdurai)\ **Post date:** [May 29, 2018, 6:50pm UTC](https://discuss.elastic.co/t/elk-setup-for-apache-application-logs/132697/5 "2018-05-29T18:50:56Z")
Hi Harash, thanks again. But I am still have problems. I am adding a few more folks to email just so they can see the stuff we have tried. Hope you don’t mind. I tried the following scheme from the first link in your message. ![](https://us1.discourse-cdn.com/elastic/original/3X/5/e/5ed7fb8e568daa7c9414f2b60866a12cdeb29ae4.jpg) In my case I used “[GD-WEB], [GD-BACKGROUND], [GD-IMP], [GD-SYNC], [GD-SLR-INDX].” As starting points instead of timestamp, please see yml file for details. It did not help. Now when an event has java stack trace, **the lines from stack trace appear as separate events.** I have attached 3 files to this message; 1. ``` See file ELKDebug_CatalinaOut.txt, for an excerpt of an event, that I would like to capture as 1 event. ``` 2. ``` See file ELKDebug_KibanaOut.txt (copied from kibana screen and pasted in this file). Notice how each line of a stack trace appears as a separate line. ``` 3. ``` Yml file in use. ``` Please let me know if you see any problems. Matt, Tristan and Kerry, this is the summary of what I am trying to do with ELK. Any help to reach the goal soon will be helpful. Thanks for your time Richard Durai ---
**Author:** ![harshbajaj16](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/harshbajaj16/32/44970_2.png) [@harshbajaj16](https://discuss.elastic.co/u/harshbajaj16)\ **Post date:** [May 31, 2018, 3:42am UTC](https://discuss.elastic.co/t/elk-setup-for-apache-application-logs/132697/6 "2018-05-31T03:42:01Z")
Hi @rdurai, i'm not able see your yml and attached configuration files. Request you to provide the YML and other details like what o/p you are getting and also explain with example what exactly you need. it will help to understand more about your use case. Regards, ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [June 28, 2018, 3:42am UTC](https://discuss.elastic.co/t/elk-setup-for-apache-application-logs/132697/7 "2018-06-28T03:42:08Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.