Hi,
I'm running logstash on linux machine as a service. I have 2 config files through which data is indexing.
Now, I want this logstash configs to run as a service ( so that indexing will not stop after restarting the machine).
If I place the config files in /etc/logstash/conf.d folder, will those config files also run along with the logstash service?
OR Is there a way to do so?
Please help in understanding the process.
Regards,
Dharma Sanjay Reddy M.
If I place the config files in /etc/logstash/conf.d folder, will those config files also run along with the logstash service?
Yes, most Logstash installations are configured to load all files in /etc/logstash/conf.d.
Thanks @magnusbaeck for the prompt response.
So you mean to say, if I place my config files in conf.d folder, then no need to run those config files manually. Is my understanding Correct?
If you run logstash as a service, yes.
Thanks @magnus for the clarification.
I have installed it as a service. But data is not indexed. When I run it manually, am able to index the data.
In addition to that, when I run the config file manually, using sudo bin/logstash -f /etc/logstash/conf.d/logstash_adlog_server.conf it says WARNING: Could not find logstash.yml which is typically located in $LS_HOME/config or /etc/logstash. You can specify the path using --path.settings. Continuing using the defaults Could not find log4j2 configuration at path /usr/share/logstash/config/log4j2.properties. Using default config which logs errors to the console
Am I missing anything.
Please find my logstash.yml file below
# Settings file in YAML
#
# Settings can be specified either in hierarchical form, e.g.:
#
# pipeline:
# batch:
# size: 125
# delay: 5
#
# Or as flat keys:
#
# pipeline.batch.size: 125
# pipeline.batch.delay: 5
#
# ------------ Node identity ------------
#
# Use a descriptive name for the node:
#
# node.name: test
#
# If omitted the node name will default to the machine's host name
#
# ------------ Data path ------------------
#
# Which directory should be used by logstash and its plugins
# for any persistent needs. Defaults to LOGSTASH_HOME/data
#
path.data: /var/lib/logstash
#
# ------------ Pipeline Settings --------------
#
# Set the number of workers that will, in parallel, execute the filters+outputs
# stage of the pipeline.
#
# This defaults to the number of the host's CPU cores.
#
# pipeline.workers: 2
#
# How many workers should be used per output plugin instance
#
# pipeline.output.workers: 1
#
# How many events to retrieve from inputs before sending to filters+workers
#
# pipeline.batch.size: 125
#
# How long to wait before dispatching an undersized batch to filters+workers
# Value is in milliseconds.
#
# pipeline.batch.delay: 5
#
# Force Logstash to exit during shutdown even if there are still inflight
# events in memory. By default, logstash will refuse to quit until all
# received events have been pushed to the outputs.
#
# WARNING: enabling this can lead to data loss during shutdown
#
# pipeline.unsafe_shutdown: false
#
# ------------ Pipeline Configuration Settings --------------
#
# Where to fetch the pipeline configuration for the main pipeline
#
path.config: /etc/logstash/conf.d
#
# Pipeline configuration string for the main pipeline
#
# config.string:
#
# At startup, test if the configuration is valid and exit (dry run)
#
# config.test_and_exit: false
#
# Periodically check if the configuration has changed and reload the pipeline
# This can also be triggered manually through the SIGHUP signal
#
# config.reload.automatic: false
#
# How often to check if the pipeline configuration has changed (in seconds)
#
# config.reload.interval: 3
#
# Show fully compiled configuration as debug log message
# NOTE: --log.level must be 'debug'
#
# config.debug: false
#
# When enabled, process escaped characters such as \n and \" in strings in the
# pipeline configuration files.
#
# config.support_escapes: false
#
# ------------ Module Settings ---------------
# Define modules here. Modules definitions must be defined as an array.
# The simple way to see this is to prepend each `name` with a `-`, and keep
# all associated variables under the `name` they are associated with, and
# above the next, like this:
#
# modules:
# - name: MODULE_NAME
# var.PLUGINTYPE1.PLUGINNAME1.KEY1: VALUE
# var.PLUGINTYPE1.PLUGINNAME1.KEY2: VALUE
# var.PLUGINTYPE2.PLUGINNAME1.KEY1: VALUE
# var.PLUGINTYPE3.PLUGINNAME3.KEY1: VALUE
#
# Module variable names must be in the format of
#
# var.PLUGIN_TYPE.PLUGIN_NAME.KEY
#
# modules:
#
# ------------ Queuing Settings --------------
#
# Internal queuing model, "memory" for legacy in-memory based queuing and
# "persisted" for disk-based acked queueing. Defaults is memory
#
# queue.type: memory
#
# If using queue.type: persisted, the directory path where the data files will be stored.
# Default is path.data/queue
#
# path.queue:
#
# If using queue.type: persisted, the page data files size. The queue data consists of
# append-only data files separated into pages. Default is 250mb
#
# queue.page_capacity: 250mb
#
# If using queue.type: persisted, the maximum number of unread events in the queue.
# Default is 0 (unlimited)
#
# queue.max_events: 0
#
# If using queue.type: persisted, the total capacity of the queue in number of bytes.
# If you would like more unacked events to be buffered in Logstash, you can increase the
# capacity using this setting. Please make sure your disk drive has capacity greater than
# the size specified here. If both max_bytes and max_events are specified, Logstash will pick
# whichever criteria is reached first
# Default is 1024mb or 1gb
#
# queue.max_bytes: 1024mb
#
# If using queue.type: persisted, the maximum number of acked events before forcing a checkpoint
# Default is 1024, 0 for unlimited
#
# queue.checkpoint.acks: 1024
#
# If using queue.type: persisted, the maximum number of written events before forcing a checkpoint
# Default is 1024, 0 for unlimited
#
# queue.checkpoint.writes: 1024
#
# If using queue.type: persisted, the interval in milliseconds when a checkpoint is forced on the head page
# Default is 1000, 0 for no periodic checkpoint.
#
# queue.checkpoint.interval: 1000
#
# ------------ Dead-Letter Queue Settings --------------
# Flag to turn on dead-letter queue.
#
# dead_letter_queue.enable: false
# If using dead_letter_queue.enable: true, the maximum size of each dead letter queue. Entries
# will be dropped if they would increase the size of the dead letter queue beyond this setting.
# Default is 1024mb
# dead_letter_queue.max_bytes: 1024mb
# If using dead_letter_queue.enable: true, the directory path where the data files will be stored.
# Default is path.data/dead_letter_queue
#
# path.dead_letter_queue:
#
# ------------ Metrics Settings --------------
#
# Bind address for the metrics REST endpoint
#
# http.host: "127.0.0.1"
#
# Bind port for the metrics REST endpoint, this option also accept a range
# (9600-9700) and logstash will pick up the first available ports.
#
# http.port: 9600-9700
#
# ------------ Debugging Settings --------------
#
# Options for log.level:
# * fatal
# * error
# * warn
# * info (default)
# * debug
# * trace
#
# log.level: info
path.logs: /var/log/logstash
#
# ------------ Other Settings --------------
#
# Where to find custom plugins
# path.plugins: []
xpack.monitoring.enabled: false
Please help to resolve this issue.
Logstash normally runs as the logstash user so the problem could be that it doesn't have permissions to read the log files. What's in your configuration files? logstash.yml isn't relevant.
Please find the config file below
input
{
file {
path => "/data/logs/genlog/genlog.xml"
start_position => "beginning"
sincedb_path => "/tmp/sincedb/genlog_sincedb"
}
}
filter
{
if [message] =~ "^<\?xml"
{
drop {}
}
xml
{
remove_namespaces => true
source => "message"
target => "event"
xpath =>
[
"/Event/System/Computer/text()", "Computer",
"/Event/System/EventID/text()", "EventID",
"/Event/AAtool/EventSummary/text()", "EventSummary",
"/Event/AAtool/Level/text()", "Level",
"/Event/AAtool/Description/text()", "Description",
"/Event/AAtool/PotentialCriticality/text()", "PotentialCriticality",
"/Event/EventData/IpAddress/text()", "IpAddress",
"/Event/EventData/RenderingInfo/text()", "RenderingInfo",
"/Event/EventData/SubjectLogonId/text()", "SubjectLogonId"
]
}
}
output
{
elasticsearch
{
hosts => "121.23.45.32"
index => "genlog-%{+YYYYMM}"
template => "/usr/share/logstash/bin/genlog.json"
template_name => "genlog*"
workers => 1
user => logstash_internal
password => changeme
}
}
And does the logstash user have read access to genlog.xml? What about /tmp/sincedb/genlog_sincedb? Have you tried bumping the log level to get more clues about what's going on?
yeah @magnusbaeck, everyone has read access.
I haven't tried on that part on sincedb file. Can you please guide me how to do so.
This is what I get from the sincedb file.
12885109798 0 64768 1295700
8589944958 0 64768 1703070
12885109826 0 64768 1078750
12885110043 0 64768 379453
2147862023 0 64768 472366
6446692648 0 64768 1018861
8590208206 0 64768 1285765
718 0 64768 2278671
~
everyone has read access.
How do you know? Have you checked the permissions of all directories?
Can you please guide me how to do so.
How to do what? The format of the sincedb file is described in the file input documentation.
I have checked the permissions of the file and corresponding directories as well. All the directories have read access.
So have you tried bumping the log level to get more clues about what's going on?
Going through the logs, will let u know the status in a while
@magnusbaeck on one server its working fine. But on other server when I was checking, there is not read access to the sincedb and template files.
I have given the access but it is not indexing. Please let me know what all files require read access.
@magnusbaeck Please find the logstash-plain.log below
[2017-11-03T17:44:13,369][INFO ][logstash.modules.scaffold] Initializing module {:module_name=>"fb_apache", :directory=>"/usr/share/logstash/modules/fb_apache/configuration"}
[2017-11-03T17:44:13,394][INFO ][logstash.modules.scaffold] Initializing module {:module_name=>"netflow", :directory=>"/usr/share/logstash/modules/netflow/configuration"}
[2017-11-03T17:44:14,324][INFO ][logstash.outputs.elasticsearch] Elasticsearch pool URLs updated {:changes=>{:removed=>[], :added=>[http://logstash_internal:xxxxxx@121.23.45.32:9200/]}}
[2017-11-03T17:44:14,325][INFO ][logstash.outputs.elasticsearch] Running health check to see if an Elasticsearch connection is working {:healthcheck_url=>http://logstash_internal:xxxxxx@121.23.45.32:9200/, :path=>"/"}
[2017-11-03T17:44:14,557][WARN ][logstash.outputs.elasticsearch] Restored connection to ES instance {:url=>"http://logstash_internal:xxxxxx@121.23.45.32:9200/"}
[2017-11-03T17:44:14,667][INFO ][logstash.outputs.elasticsearch] Using mapping template from {:path=>"/usr/share/logstash/bin/genlog.json"}
[2017-11-03T17:44:14,679][INFO ][logstash.outputs.elasticsearch] Attempting to install template {:manage_template=>{"template"=>"genlog*", "order"=>1, "settings"=>{"number_of_shards"=>5}, "mappings"=>{"genlog*"=>{"dynamic_templates"=>[{"string_fields"=>{"mapping"=>{"index"=>"analyzed", "omit_norms"=>true, "type"=>"string", "fields"=>{"raw"=>{"index"=>"not_analyzed", "ignore_above"=>256, "type"=>"string"}}}, "match_mapping_type"=>"string", "match"=>"*"}}, {"boolean_fields"=>{"mapping"=>{"type"=>"boolean"}, "match"=>"*is_*"}}], "_all"=>{"enabled"=>true}, "properties"=>{"location"=>{"type"=>"geo_point"}, "clientip"=>{"type"=>"ip"}}}}}}
[2017-11-03T17:44:14,691][INFO ][logstash.outputs.elasticsearch] New Elasticsearch output {:class=>"LogStash::Outputs::ElasticSearch", :hosts=>["//121.23.45.32"]}
[2017-11-03T17:44:15,030][INFO ][logstash.filters.geoip ] Using geoip database {:path=>"/usr/share/logstash/vendor/bundle/jruby/1.9/gems/logstash-filter-geoip-4.3.1-java/vendor/GeoLite2-City.mmdb"}
[2017-11-03T17:44:15,062][INFO ][logstash.pipeline ] Starting pipeline {"id"=>"main", "pipeline.workers"=>4, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>5, "pipeline.max_inflight"=>500}
[2017-11-03T17:44:15,602][INFO ][logstash.pipeline ] Pipeline main started
[2017-11-03T17:44:15,674][INFO ][logstash.agent ] Successfully started Logstash API endpoint {:port=>9600}
For me from here everything looks fine.
When I run manually its indexing fine, but when it is going through /etc/logstash/conf.d folder, data is not indexing.
I have given read access to /data/logs/genlog/genlog.xml, /tmp/sincedb/genlog_sincedb, /usr/share/logstash/bin/genlog.json. But still not able to index the data
Please let me know what all files need read Access.
That log is with the normal log level. I'd be interested to see the debug-level log.
My bad @magnusbaeck.
It was the file permissions that was blocking logstash to access the logs. I have checked all the directories, but missed out the log permissions.
Thanks again for the help.
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.
© 2020. All Rights Reserved - Elasticsearch
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant logo are trademarks of the Apache Software Foundation in the United States and/or other countries.