# Have to run logstash config file automatically **URL:** \ **Category:** Logstash\ **Created:** [November 2, 2017, 12:38pm UTC](https://discuss.elastic.co/t/have-to-run-logstash-config-file-automatically/106195 "2017-11-02T12:38:29Z")\ **Posts on this page:** 18\ **Page:** 1
**Author:** ![sanju1323](https://avatars.discourse-cdn.com/v4/letter/s/5daacb/32.png) [@sanju1323](https://discuss.elastic.co/u/sanju1323)\ **Post date:** [November 2, 2017, 12:38pm UTC](https://discuss.elastic.co/t/have-to-run-logstash-config-file-automatically/106195/1 "2017-11-02T12:38:30Z")
Hi, I'm running logstash on linux machine as a service. I have 2 config files through which data is indexing. Now, I want this logstash configs to run as a service ( so that indexing will not stop after restarting the machine). If I place the config files in /etc/logstash/conf.d folder, will those config files also run along with the logstash service? OR Is there a way to do so? Please help in understanding the process. Regards, Dharma Sanjay Reddy M. ---
**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\ **Post date:** [November 2, 2017, 1:01pm UTC](https://discuss.elastic.co/t/have-to-run-logstash-config-file-automatically/106195/2 "2017-11-02T13:01:12Z")
> If I place the config files in /etc/logstash/conf.d folder, will those config files also run along with the logstash service? Yes, most Logstash installations are configured to load all files in /etc/logstash/conf.d. ---
**Author:** ![sanju1323](https://avatars.discourse-cdn.com/v4/letter/s/5daacb/32.png) [@sanju1323](https://discuss.elastic.co/u/sanju1323)\ **Post date:** [November 2, 2017, 1:31pm UTC](https://discuss.elastic.co/t/have-to-run-logstash-config-file-automatically/106195/3 "2017-11-02T13:31:13Z")
Thanks @magnusbaeck for the prompt response. So you mean to say, if I place my config files in conf.d folder, then no need to run those config files manually. Is my understanding Correct? ---
**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\ **Post date:** [November 3, 2017, 6:19am UTC](https://discuss.elastic.co/t/have-to-run-logstash-config-file-automatically/106195/4 "2017-11-03T06:19:01Z")
If you run logstash as a service, yes. ---
**Author:** ![sanju1323](https://avatars.discourse-cdn.com/v4/letter/s/5daacb/32.png) [@sanju1323](https://discuss.elastic.co/u/sanju1323)\ **Post date:** [November 3, 2017, 6:23am UTC](https://discuss.elastic.co/t/have-to-run-logstash-config-file-automatically/106195/5 "2017-11-03T06:23:00Z")
Thanks @magnus for the clarification. I have installed it as a service. But data is not indexed. When I run it manually, am able to index the data. In addition to that, when I run the config file manually, using `sudo bin/logstash -f /etc/logstash/conf.d/logstash_adlog_server.conf` it says `WARNING: Could not find logstash.yml which is typically located in $LS_HOME/config or /etc/logstash. You can specify the path using --path.settings. Continuing using the defaults Could not find log4j2 configuration at path /usr/share/logstash/config/log4j2.properties. Using default config which logs errors to the console` Am I missing anything. Please find my logstash.yml file below ```auto # Settings file in YAML # # Settings can be specified either in hierarchical form, e.g.: # # pipeline: # batch: # size: 125 # delay: 5 # # Or as flat keys: # # pipeline.batch.size: 125 # pipeline.batch.delay: 5 # # ------------ Node identity ------------ # # Use a descriptive name for the node: # # node.name: test # # If omitted the node name will default to the machine's host name # # ------------ Data path ------------------ # # Which directory should be used by logstash and its plugins # for any persistent needs. Defaults to LOGSTASH_HOME/data # path.data: /var/lib/logstash # # ------------ Pipeline Settings -------------- # # Set the number of workers that will, in parallel, execute the filters+outputs # stage of the pipeline. # # This defaults to the number of the host's CPU cores. # # pipeline.workers: 2 # # How many workers should be used per output plugin instance # # pipeline.output.workers: 1 # # How many events to retrieve from inputs before sending to filters+workers # # pipeline.batch.size: 125 # # How long to wait before dispatching an undersized batch to filters+workers # Value is in milliseconds. # # pipeline.batch.delay: 5 # # Force Logstash to exit during shutdown even if there are still inflight # events in memory. By default, logstash will refuse to quit until all # received events have been pushed to the outputs. # # WARNING: enabling this can lead to data loss during shutdown # # pipeline.unsafe_shutdown: false # # ------------ Pipeline Configuration Settings -------------- # # Where to fetch the pipeline configuration for the main pipeline # path.config: /etc/logstash/conf.d # # Pipeline configuration string for the main pipeline # # config.string: # # At startup, test if the configuration is valid and exit (dry run) # # config.test_and_exit: false # # Periodically check if the configuration has changed and reload the pipeline # This can also be triggered manually through the SIGHUP signal # # config.reload.automatic: false # # How often to check if the pipeline configuration has changed (in seconds) # # config.reload.interval: 3 # # Show fully compiled configuration as debug log message # NOTE: --log.level must be 'debug' # # config.debug: false # # When enabled, process escaped characters such as \n and \" in strings in the # pipeline configuration files. # # config.support_escapes: false # # ------------ Module Settings --------------- # Define modules here. Modules definitions must be defined as an array. # The simple way to see this is to prepend each `name` with a `-`, and keep # all associated variables under the `name` they are associated with, and # above the next, like this: # # modules: # - name: MODULE_NAME # var.PLUGINTYPE1.PLUGINNAME1.KEY1: VALUE # var.PLUGINTYPE1.PLUGINNAME1.KEY2: VALUE # var.PLUGINTYPE2.PLUGINNAME1.KEY1: VALUE # var.PLUGINTYPE3.PLUGINNAME3.KEY1: VALUE # # Module variable names must be in the format of # # var.PLUGIN_TYPE.PLUGIN_NAME.KEY # # modules: # # ------------ Queuing Settings -------------- # # Internal queuing model, "memory" for legacy in-memory based queuing and # "persisted" for disk-based acked queueing. Defaults is memory # # queue.type: memory # # If using queue.type: persisted, the directory path where the data files will be stored. # Default is path.data/queue # # path.queue: # # If using queue.type: persisted, the page data files size. The queue data consists of # append-only data files separated into pages. Default is 250mb # # queue.page_capacity: 250mb # # If using queue.type: persisted, the maximum number of unread events in the queue. # Default is 0 (unlimited) # # queue.max_events: 0 # # If using queue.type: persisted, the total capacity of the queue in number of bytes. # If you would like more unacked events to be buffered in Logstash, you can increase the # capacity using this setting. Please make sure your disk drive has capacity greater than # the size specified here. If both max_bytes and max_events are specified, Logstash will pick # whichever criteria is reached first # Default is 1024mb or 1gb # # queue.max_bytes: 1024mb # # If using queue.type: persisted, the maximum number of acked events before forcing a checkpoint # Default is 1024, 0 for unlimited # # queue.checkpoint.acks: 1024 # # If using queue.type: persisted, the maximum number of written events before forcing a checkpoint # Default is 1024, 0 for unlimited # # queue.checkpoint.writes: 1024 # # If using queue.type: persisted, the interval in milliseconds when a checkpoint is forced on the head page # Default is 1000, 0 for no periodic checkpoint. # # queue.checkpoint.interval: 1000 # # ------------ Dead-Letter Queue Settings -------------- # Flag to turn on dead-letter queue. # # dead_letter_queue.enable: false # If using dead_letter_queue.enable: true, the maximum size of each dead letter queue. Entries # will be dropped if they would increase the size of the dead letter queue beyond this setting. # Default is 1024mb # dead_letter_queue.max_bytes: 1024mb # If using dead_letter_queue.enable: true, the directory path where the data files will be stored. # Default is path.data/dead_letter_queue # # path.dead_letter_queue: # # ------------ Metrics Settings -------------- # # Bind address for the metrics REST endpoint # # http.host: "127.0.0.1" # # Bind port for the metrics REST endpoint, this option also accept a range # (9600-9700) and logstash will pick up the first available ports. # # http.port: 9600-9700 # # ------------ Debugging Settings -------------- # # Options for log.level: # * fatal # * error # * warn # * info (default) # * debug # * trace # # log.level: info path.logs: /var/log/logstash # # ------------ Other Settings -------------- # # Where to find custom plugins # path.plugins: [] xpack.monitoring.enabled: false ``` Please help to resolve this issue. ---
**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\ **Post date:** [November 3, 2017, 6:26am UTC](https://discuss.elastic.co/t/have-to-run-logstash-config-file-automatically/106195/6 "2017-11-03T06:26:24Z")
Logstash normally runs as the logstash user so the problem could be that it doesn't have permissions to read the log files. What's in your configuration files? logstash.yml isn't relevant. ---
**Author:** ![sanju1323](https://avatars.discourse-cdn.com/v4/letter/s/5daacb/32.png) [@sanju1323](https://discuss.elastic.co/u/sanju1323)\ **Post date:** [November 3, 2017, 6:32am UTC](https://discuss.elastic.co/t/have-to-run-logstash-config-file-automatically/106195/7 "2017-11-03T06:32:21Z")
Please find the config file below ```auto input { file { path => "/data/logs/genlog/genlog.xml" start_position => "beginning" sincedb_path => "/tmp/sincedb/genlog_sincedb" } } filter { if [message] =~ "^<\?xml" { drop {} } xml { remove_namespaces => true source => "message" target => "event" xpath => [ "/Event/System/Computer/text()", "Computer", "/Event/System/EventID/text()", "EventID", "/Event/AAtool/EventSummary/text()", "EventSummary", "/Event/AAtool/Level/text()", "Level", "/Event/AAtool/Description/text()", "Description", "/Event/AAtool/PotentialCriticality/text()", "PotentialCriticality", "/Event/EventData/IpAddress/text()", "IpAddress", "/Event/EventData/RenderingInfo/text()", "RenderingInfo", "/Event/EventData/SubjectLogonId/text()", "SubjectLogonId" ] } } output { elasticsearch { hosts => "121.23.45.32" index => "genlog-%{+YYYYMM}" template => "/usr/share/logstash/bin/genlog.json" template_name => "genlog*" workers => 1 user => logstash_internal password => changeme } } ``` ---
**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\ **Post date:** [November 3, 2017, 6:40am UTC](https://discuss.elastic.co/t/have-to-run-logstash-config-file-automatically/106195/8 "2017-11-03T06:40:01Z")
And does the logstash user have read access to genlog.xml? What about /tmp/sincedb/genlog\_sincedb? Have you tried bumping the log level to get more clues about what's going on? ---
**Author:** ![sanju1323](https://avatars.discourse-cdn.com/v4/letter/s/5daacb/32.png) [@sanju1323](https://discuss.elastic.co/u/sanju1323)\ **Post date:** [November 3, 2017, 6:51am UTC](https://discuss.elastic.co/t/have-to-run-logstash-config-file-automatically/106195/9 "2017-11-03T06:51:02Z")
yeah @magnusbaeck, everyone has read access. I haven't tried on that part on sincedb file. Can you please guide me how to do so. This is what I get from the sincedb file. ```auto 12885109798 0 64768 1295700 8589944958 0 64768 1703070 12885109826 0 64768 1078750 12885110043 0 64768 379453 2147862023 0 64768 472366 6446692648 0 64768 1018861 8590208206 0 64768 1285765 718 0 64768 2278671 ~ ``` ---
**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\ **Post date:** [November 3, 2017, 6:53am UTC](https://discuss.elastic.co/t/have-to-run-logstash-config-file-automatically/106195/10 "2017-11-03T06:53:07Z")
> everyone has read access. How do you know? Have you checked the permissions of all directories? > Can you please guide me how to do so. How to do what? The format of the sincedb file is described in the file input documentation. ---
**Author:** ![sanju1323](https://avatars.discourse-cdn.com/v4/letter/s/5daacb/32.png) [@sanju1323](https://discuss.elastic.co/u/sanju1323)\ **Post date:** [November 3, 2017, 6:56am UTC](https://discuss.elastic.co/t/have-to-run-logstash-config-file-automatically/106195/11 "2017-11-03T06:56:37Z")
I have checked the permissions of the file and corresponding directories as well. All the directories have read access. ---
**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\ **Post date:** [November 3, 2017, 7:03am UTC](https://discuss.elastic.co/t/have-to-run-logstash-config-file-automatically/106195/12 "2017-11-03T07:03:22Z")
So have you tried bumping the log level to get more clues about what's going on? ---
**Author:** ![sanju1323](https://avatars.discourse-cdn.com/v4/letter/s/5daacb/32.png) [@sanju1323](https://discuss.elastic.co/u/sanju1323)\ **Post date:** [November 3, 2017, 7:59am UTC](https://discuss.elastic.co/t/have-to-run-logstash-config-file-automatically/106195/13 "2017-11-03T07:59:28Z")
Going through the logs, will let u know the status in a while ---
**Author:** ![sanju1323](https://avatars.discourse-cdn.com/v4/letter/s/5daacb/32.png) [@sanju1323](https://discuss.elastic.co/u/sanju1323)\ **Post date:** [November 3, 2017, 8:48am UTC](https://discuss.elastic.co/t/have-to-run-logstash-config-file-automatically/106195/14 "2017-11-03T08:48:28Z")
@magnusbaeck on one server its working fine. But on other server when I was checking, there is not read access to the sincedb and template files. I have given the access but it is not indexing. Please let me know what all files require read access. ---
**Author:** ![sanju1323](https://avatars.discourse-cdn.com/v4/letter/s/5daacb/32.png) [@sanju1323](https://discuss.elastic.co/u/sanju1323)\ **Post date:** [November 3, 2017, 10:00am UTC](https://discuss.elastic.co/t/have-to-run-logstash-config-file-automatically/106195/15 "2017-11-03T10:00:15Z")
@magnusbaeck Please find the logstash-plain.log below ```auto [2017-11-03T17:44:13,369][INFO][logstash.modules.scaffold] Initializing module {:module_name=>"fb_apache", :directory=>"/usr/share/logstash/modules/fb_apache/configuration"} [2017-11-03T17:44:13,394][INFO][logstash.modules.scaffold] Initializing module {:module_name=>"netflow", :directory=>"/usr/share/logstash/modules/netflow/configuration"} [2017-11-03T17:44:14,324][INFO][logstash.outputs.elasticsearch] Elasticsearch pool URLs updated {:changes=>{:removed=>[], :added=>[http://logstash_internal:xxxxxx@121.23.45.32:9200/]}} [2017-11-03T17:44:14,325][INFO][logstash.outputs.elasticsearch] Running health check to see if an Elasticsearch connection is working {:healthcheck_url=>http://logstash_internal:xxxxxx@121.23.45.32:9200/, :path=>"/"} [2017-11-03T17:44:14,557][WARN][logstash.outputs.elasticsearch] Restored connection to ES instance {:url=>"http://logstash_internal:xxxxxx@121.23.45.32:9200/"} [2017-11-03T17:44:14,667][INFO][logstash.outputs.elasticsearch] Using mapping template from {:path=>"/usr/share/logstash/bin/genlog.json"} [2017-11-03T17:44:14,679][INFO][logstash.outputs.elasticsearch] Attempting to install template {:manage_template=>{"template"=>"genlog*", "order"=>1, "settings"=>{"number_of_shards"=>5}, "mappings"=>{"genlog*"=>{"dynamic_templates"=>[{"string_fields"=>{"mapping"=>{"index"=>"analyzed", "omit_norms"=>true, "type"=>"string", "fields"=>{"raw"=>{"index"=>"not_analyzed", "ignore_above"=>256, "type"=>"string"}}}, "match_mapping_type"=>"string", "match"=>"*"}}, {"boolean_fields"=>{"mapping"=>{"type"=>"boolean"}, "match"=>"*is_*"}}], "_all"=>{"enabled"=>true}, "properties"=>{"location"=>{"type"=>"geo_point"}, "clientip"=>{"type"=>"ip"}}}}}} [2017-11-03T17:44:14,691][INFO][logstash.outputs.elasticsearch] New Elasticsearch output {:class=>"LogStash::Outputs::ElasticSearch", :hosts=>["//121.23.45.32"]} [2017-11-03T17:44:15,030][INFO][logstash.filters.geoip] Using geoip database {:path=>"/usr/share/logstash/vendor/bundle/jruby/1.9/gems/logstash-filter-geoip-4.3.1-java/vendor/GeoLite2-City.mmdb"} [2017-11-03T17:44:15,062][INFO][logstash.pipeline] Starting pipeline {"id"=>"main", "pipeline.workers"=>4, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>5, "pipeline.max_inflight"=>500} [2017-11-03T17:44:15,602][INFO][logstash.pipeline] Pipeline main started [2017-11-03T17:44:15,674][INFO][logstash.agent] Successfully started Logstash API endpoint {:port=>9600} ``` For me from here everything looks fine. When I run manually its indexing fine, but when it is going through /etc/logstash/conf.d folder, data is not indexing. I have given read access to `/data/logs/genlog/genlog.xml`, `/tmp/sincedb/genlog_sincedb`, `/usr/share/logstash/bin/genlog.json`. But still not able to index the data Please let me know what all files need read Access. ---
**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\ **Post date:** [November 3, 2017, 10:41am UTC](https://discuss.elastic.co/t/have-to-run-logstash-config-file-automatically/106195/16 "2017-11-03T10:41:56Z")
That log is with the normal log level. I'd be interested to see the debug-level log. ---
**Author:** ![sanju1323](https://avatars.discourse-cdn.com/v4/letter/s/5daacb/32.png) [@sanju1323](https://discuss.elastic.co/u/sanju1323)\ **Post date:** [November 10, 2017, 9:27am UTC](https://discuss.elastic.co/t/have-to-run-logstash-config-file-automatically/106195/17 "2017-11-10T09:27:52Z")
My bad @magnusbaeck. It was the file permissions that was blocking logstash to access the logs. I have checked all the directories, but missed out the log permissions. Thanks again for the help. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [December 8, 2017, 9:28am UTC](https://discuss.elastic.co/t/have-to-run-logstash-config-file-automatically/106195/18 "2017-12-08T09:28:02Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.