I want to use filebeat to ship developers application logs to elasticsearch running in a developers servers. Developers application generates a log file in a particular folder. Whenever the log file reaches a configured size, then a new log file is started like app.log.$date1, app.log.$date2, app.log.$date3 etc. Can I make filebeat read new log files whenever the new log file is started.
Most important I have to see content of the application log files. Right now I can see only message.
Also, is there a possibility I can only send a subset of traces from these application log files to elasticsearch or does it require logstash to do this?
If you need send some metadata, you can use
Thank you for quick replay. Below my filebeat configuration:
=========================== Filebeat inputs =============================
Each - is an input. Most options can be set at the input level, so
you can use different inputs for various configurations.
Below are the input specific configurations.
- type: log
Change to true to enable this input configuration.
Paths that should be crawled and fetched. Glob based paths.
The regexp Pattern that has to be matched. The example pattern matches all lines starting with [
Defines if the pattern set under pattern should be negated or not. Default is false.
Match can be set to "after" or "before". It is used to define if lines should be append to a pattern
that was (not) matched before or after or as long as a pattern is not matched based on negate.
Note: After is the equivalent to previous and before is the equivalent to to next in Logstash
============================= Filebeat modules ===============================
Glob pattern for configuration loading
Set to true to enable config reloading
#==================== Elasticsearch template setting ==========================
#============================== Kibana =====================================
Starting with Beats version 6.0.0, the dashboards are loaded via the Kibana API.
This requires a Kibana endpoint configuration.
Scheme and port can be left out and will be set to the default (http and 5601)
In case you specify and additional path, the scheme is required: http://localhost:5601/path
IPv6 addresses should always be defined as: https://[2001:db8::1]:5601
#-------------------------- Elasticsearch output ------------------------------
Array of hosts to connect to.
What I can see from Kibana:
Use a wildcard
Ok, that doesn't make sense It's a typical wildcard, I thought it was implicit, you can try
/var/log/* to ingest all files in that folder (just to double check, I mean in the
path object )
Better change the log files name such as: $date1.app.log $date2.app.log
So, the configuration of filebeat is simpler. just /var/log/*.log
Another suggestion, if you want to put the message contents of the apps log to the meaningfull fields, use logstash to handle the specific log format, also, if there are not syslog format, better put into another folder, such as /var/log/specificapps/*.log
Then use grok from logstash filter to dissect the message content into fields in the index.
I changed path to /var/log/*. Same result. I can see only /var/log/syslog and /var/log/auth.log file. No messages from my files.
I changed path to /var/log/awsapi/*.log and used logstash to handle specific log format. Noting happened and it getting worth:
I can’t see even message with my files and loosed the server(awsapi02) for dashboard:
Did you already add the path into the filebeat conf?
Yes, I did
St. System Administrator
Folio Financial, Inc
Sr. System Administrator
Folio Financial, Inc
Please do not use email to transmit orders for securities or for other time-sensitive messages. Broker-dealer securities products and services are offered only through Folio Investments, Inc. Member FINRA/SIPC. Folio Investments, Inc. conducts business under various trade names, including: Folio Financial, Folio Institutional, Folio Investing, FolioFirst, VIAFolio, Folio Client, and DirectIndexer. Folio Investments, Inc. is a subsidiary of Folio Financial, Inc. and an affiliate of First Affirmative Financial Network, LLC and Folio Research, LLC. Investment advisory services are provided through First Affirmative Financial Network, LLC and Folio Research, LLC.
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.