# Unable to create logstash index in Kibana **URL:** \ **Category:** Logstash\ **Created:** [March 9, 2020, 7:22pm UTC](https://discuss.elastic.co/t/unable-to-create-logstash-index-in-kibana/222790 "2020-03-09T19:22:29Z")\ **Posts on this page:** 8\ **Page:** 1
**Author:** ![gmorales](https://avatars.discourse-cdn.com/v4/letter/g/f4b2a3/32.png) [@gmorales](https://discuss.elastic.co/u/gmorales)\ **Post date:** [March 9, 2020, 7:22pm UTC](https://discuss.elastic.co/t/unable-to-create-logstash-index-in-kibana/222790/1 "2020-03-09T19:22:29Z")
I am a newbie, I inherited an older elasticsearch, kibana, logstash, filebeat server and it is currently working fine, however I wanted to run the latest and went ahead and installed learticsearch, kibana, logstash, filebeat on a new Ubuntu Linux server but I am unable to create the logstash index in kibana, it does not show up when I try to create the index. Any help would be highly appreciated. root@elastic\_syslog:/var/log/logstash# root@elastic\_syslog:/var/log/logstash# sudo service logstash status â logstash.service - logstash Loaded: loaded (/etc/systemd/system/logstash.service; disabled; vendor preset: enabled) Active: active (running) since Mon 2020-03-09 19:19:32 UTC; 14s ago Main PID: 17435 (java) Tasks: 22 (limit: 4915) CGroup: /system.slice/logstash.service ââ17435 /usr/bin/java -Xms1g -Xmx1g -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -Djava.awt.headless=true -Dfile.encoding=UTF-8 -Djruby.compile. Mar 09 19:19:32 elastic\_syslog systemd[1]: Started logstash. Mar 09 19:19:45 elastic\_syslog logstash[17435]: Sending Logstash logs to /var/log/logstash which is now configured via log4j2.properties Mar 09 19:19:45 elastic\_syslog logstash[17435]: [2020-03-09T19:19:45,706][INFO][logstash.runner] Starting Logstash {"logstash.version"=\>"7.6.1"} root@elastic\_syslog:/var/log/logstash# curl -XGET "[http://localhost:9200/\_cat/indices](http://localhost:9200/_cat/indices)" green open .kibana\_task\_manager\_1 i8eCVQkASeyxa3UL3lepEA 1 0 2 1 31.4kb 31.4kb yellow open filebeat-7.6.1-2020.03.05-000001 UcTKM9nfQ3qFK\_cO2zzgcg 1 1 49821 0 10.1mb 10.1mb green open ilm-history-1-000001 O3n92t7SRky7ahtWvjddJQ 1 0 18 0 25.3kb 25.3kb green open .apm-agent-configuration TCj94zuGRkug0fdlBGzv1A 1 0 0 0 283b 283b green open kibana\_sample\_data\_logs IwftZXkMRq-cRN1cie-b1Q 1 0 14074 0 11.4mb 11.4mb green open .kibana\_1 aM8WEHR1RQCd\_5S2d4y0kA 1 0 302 26 474.2kb 474.2kb } [2020-03-09T15:47:46,602][ERROR][logstash.agent] Failed to execute action {:action=\>LogStash::PipelineAction::Create/pipeline\_id:main, :exception=\>"Java::JavaLang::IllegalStateException", :message =\>"Unable to configure plugins: (ConfigurationError) Something is wrong with your configuration.", :backtrace=\>["org.logstash.config.ir.CompiledPipeline.(CompiledPipeline.java:103)", "org.logstash.exec ution.JavaBasePipelineExt.initialize(JavaBasePipelineExt.java:60)", "org.logstash.execution.JavaBasePipelineExt$INVOKER$i$1$0$initialize.call(JavaBasePipelineExt$INVOKER$i$1$0$initialize.gen)", "org.jruby.in ternal.runtime.methods.JavaMethod$JavaMethodN.call(JavaMethod.java:837)", "org.jruby.ir.runtime.IRRuntimeHelpers.instanceSuper(IRRuntimeHelpers.java:1169)", "org.jruby.ir.runtime.IRRuntimeHelpers.instanceSup erSplatArgs(IRRuntimeHelpers.java:1156)", "org.jruby.ir.targets.InstanceSuperInvokeSite.invoke(InstanceSuperInvokeSite.java:39)", "usr.share.logstash.logstash\_minus\_core.lib.logstash.java\_pipeline.RUBY$metho d$initialize$0(/usr/share/logstash/logstash-core/lib/logstash/java\_pipeline.rb:27)", "org.jruby.internal.runtime.methods.CompiledIRMethod.call(CompiledIRMethod.java:84)", "org.jruby.internal.runtime.methods. MixedModeIRMethod.call(MixedModeIRMethod.java:70)", "org.jruby.runtime.callsite.CachingCallSite.cacheAndCall(CachingCallSite.java:332)", "org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java: 86)", "org.jruby.RubyClass.newInstance(RubyClass.java:915)", "org.jruby.RubyClass$INVOKER$i$newInstance.call(RubyClass$INVOKER$i$newInstance.gen)", "org.jruby.ir.targets.InvokeSite.invoke(InvokeSite.java:183 )", "usr.share.logstash.logstash\_minus\_core.lib.logstash.pipeline\_action.create.RUBY$method$execute$0(/usr/share/logstash/logstash-core/lib/logstash/pipeline\_action/create.rb:36)", "usr.share.logstash.logsta sh\_minus\_core.lib.logstash.pipeline\_action.create.RUBY$method$execute$0$ **VARARGS** (/usr/share/logstash/logstash-core/lib/logstash/pipeline\_action/create.rb)", "org.jruby.internal.runtime.methods.CompiledIRM ethod.call(CompiledIRMethod.java:84)", "org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:70)", "org.jruby.ir.targets.InvokeSite.invoke(InvokeSite.java:183)", "usr.share.logsta sh.logstash\_minus\_core.lib.logstash.agent.RUBY$block$converge\_state$2(/usr/share/logstash/logstash-core/lib/logstash/agent.rb:326)", "org.jruby.runtime.CompiledIRBlockBody.callDirect(CompiledIRBlockBody.java :136)", "org.jruby.runtime.IRBlockBody.call(IRBlockBody.java:77)", "org.jruby.runtime.IRBlockBody.call(IRBlockBody.java:71)", "org.jruby.runtime.Block.call(Block.java:125)", "org.jruby.RubyProc.call(RubyProc .java:274)", "org.jruby.internal.runtime.RubyRunnable.run(RubyRunnable.java:105)", "java.lang.Thread.run(Thread.java:748)"]} [2020-03-09T15:47:46,616][ERROR][logstash.agent] An exception happened when converging configuration {:exception=\>LogStash::Error, :message=\>"Don't know how to handle `Java::JavaLang::IllegalState Exception` for `PipelineAction::Create
`", :backtrace=\>["org/logstash/execution/ConvergeResultExt.java:109:in `create'", "org/logstash/execution/ConvergeResultExt.java:37:in `add'", "/usr/share/logstash /logstash-core/lib/logstash/agent.rb:339:in `block in converge_state'"]} [2020-03-09T15:47:46,647][FATAL][logstash.runner] An unexpected error occurred! {:error=>#, :backtrace=>["org/logstash/execution/ConvergeResultExt.java:109:in `create'", "org/logstash/execution/ConvergeResultExt.java:37:in `add'", "/usr/share/logstash/logstash-core/lib/logstash/agent.r b:339:in `block in converge\_state'"]} [2020-03-09T15:47:46,664][ERROR][org.logstash.Logstash] java.lang.IllegalStateException: Logstash stopped processing because of an error: (SystemExit) exit [2020-03-09T15:48:00,274][INFO][logstash.runner] Starting Logstash {"logstash.version"=\>"7.6.1"} [2020-03-09T15:48:02,778][INFO][org.reflections.Reflections] Reflections took 34 ms to scan 1 urls, producing 20 keys and 40 values [2020-03-09T15:48:03,085][ERROR][logstash.inputs.beats] Invalid setting for beats input plugin: input { beats { # This setting must be a path # File does not exist or cannot be opened /etc/pki/tls/certs/logstash-forwarder.crt ssl\_certificate =\> "/etc/pki/tls/certs/logstash-forwarder.crt" ... } } [2020-03-09T15:48:03,087][ERROR][logstash.inputs.beats] Invalid setting for beats input plugin: input { beats { # This setting must be a path # File does not exist or cannot be opened /etc/pki/tls/private/logstash-forwarder.key ssl\_key =\> "/etc/pki/tls/private/logstash-forwarder.key" ---
**Author:** ![gmorales](https://avatars.discourse-cdn.com/v4/letter/g/f4b2a3/32.png) [@gmorales](https://discuss.elastic.co/u/gmorales)\ **Post date:** [March 9, 2020, 7:26pm UTC](https://discuss.elastic.co/t/unable-to-create-logstash-index-in-kibana/222790/2 "2020-03-09T19:26:44Z")
![image](https://us1.discourse-cdn.com/elastic/original/3X/9/6/9683dac96f3be336d75764d2de73b74ac8b0c234.png) ![image](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1aeeb08136ade156dda5ef5dd05efa390663ad8c.png) ![image](https://us1.discourse-cdn.com/elastic/original/3X/a/5/a5cba9bcf546e291790022debe27d6e42551432d.png) ---
**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\ **Post date:** [March 9, 2020, 8:41pm UTC](https://discuss.elastic.co/t/unable-to-create-logstash-index-in-kibana/222790/3 "2020-03-09T20:41:42Z")
Please don't post pictures of text, they are difficult to read, impossible to search and some people may not be even able to see them 🙂 ---
**Author:** ![gmorales](https://avatars.discourse-cdn.com/v4/letter/g/f4b2a3/32.png) [@gmorales](https://discuss.elastic.co/u/gmorales)\ **Post date:** [March 9, 2020, 8:57pm UTC](https://discuss.elastic.co/t/unable-to-create-logstash-index-in-kibana/222790/4 "2020-03-09T20:57:52Z")
understood thanks mark, I also posted actual images of the same... ---
**Author:** ![gmorales](https://avatars.discourse-cdn.com/v4/letter/g/f4b2a3/32.png) [@gmorales](https://discuss.elastic.co/u/gmorales)\ **Post date:** [March 9, 2020, 9:07pm UTC](https://discuss.elastic.co/t/unable-to-create-logstash-index-in-kibana/222790/5 "2020-03-09T21:07:57Z")
sorry I misread your previous Email, ---
**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\ **Post date:** [March 9, 2020, 10:02pm UTC](https://discuss.elastic.co/t/unable-to-create-logstash-index-in-kibana/222790/6 "2020-03-09T22:02:41Z")
It'd be helpful if you posted your entire config. Please also format your code/logs/config using the `` button in the text editor, or markdown style back ticks. It helps to make things easy to read which helps us help you 🙂 ---
**Author:** ![gmorales](https://avatars.discourse-cdn.com/v4/letter/g/f4b2a3/32.png) [@gmorales](https://discuss.elastic.co/u/gmorales)\ **Post date:** [March 10, 2020, 3:17pm UTC](https://discuss.elastic.co/t/unable-to-create-logstash-index-in-kibana/222790/7 "2020-03-10T15:17:02Z")
Thanks Mark, here are the config files. root@elastic\_syslog:/etc/logstash# vi logstash-sample.conf # Sample Logstash configuration for creating a simple # Beats -\> Logstash -\> Elasticsearch pipeline. ``` input { beats { port => 5044 } } output { elasticsearch { hosts => ["http://localhost:9200"] index => "%{[@metadata][beat]}-%{[@metadata][version]}-%{+YYYY.MM.dd}" #user => "elastic" #password => "changeme" } } ``` * * * ``` root@elastic_syslog:/etc/logstash# more logstash.yml # Settings file in YAML # # Settings can be specified either in hierarchical form, e.g.: # # pipeline: # batch: # size: 125 # delay: 5 # # Or as flat keys: # # pipeline.batch.size: 125 # pipeline.batch.delay: 5 # # ------------ Node identity ------------ # # Use a descriptive name for the node: # # node.name: test # # If omitted the node name will default to the machine's host name # # ------------ Data path ------------------ # # Which directory should be used by logstash and its plugins # for any persistent needs. Defaults to LOGSTASH_HOME/data # path.data: /var/lib/logstash # # ------------ Pipeline Settings -------------- # # The ID of the pipeline. # # pipeline.id: main # # Set the number of workers that will, in parallel, execute the filters+outputs # stage of the pipeline. # # This defaults to the number of the host's CPU cores. # # pipeline.workers: 2 # # How many events to retrieve from inputs before sending to filters+workers # # pipeline.batch.size: 125 # # How long to wait in milliseconds while polling for the next event # before dispatching an undersized batch to filters+outputs # # pipeline.batch.delay: 50 # # Force Logstash to exit during shutdown even if there are still inflight # events in memory. By default, logstash will refuse to quit until all # received events have been pushed to the outputs. # # WARNING: enabling this can lead to data loss during shutdown # # pipeline.unsafe_shutdown: false # # ------------ Pipeline Configuration Settings -------------- # # Where to fetch the pipeline configuration for the main pipeline # # path.config: # # Pipeline configuration string for the main pipeline # # config.string: # # At startup, test if the configuration is valid and exit (dry run) # # config.test_and_exit: false # # Periodically check if the configuration has changed and reload the pipeline # This can also be triggered manually through the SIGHUP signal # # config.reload.automatic: false # # How often to check if the pipeline configuration has changed (in seconds) # # config.reload.interval: 3s # # Show fully compiled configuration as debug log message # NOTE: --log.level must be 'debug' # # config.debug: false # # When enabled, process escaped characters such as \n and \" in strings in the # pipeline configuration files. # # config.support_escapes: false # # ------------ Module Settings --------------- # Define modules here. Modules definitions must be defined as an array. # The simple way to see this is to prepend each `name` with a `-`, and keep # all associated variables under the `name` they are associated with, and # above the next, like this: # # modules: # - name: MODULE_NAME # var.PLUGINTYPE1.PLUGINNAME1.KEY1: VALUE # var.PLUGINTYPE1.PLUGINNAME1.KEY2: VALUE # var.PLUGINTYPE2.PLUGINNAME1.KEY1: VALUE # var.PLUGINTYPE3.PLUGINNAME3.KEY1: VALUE # # Module variable names must be in the format of # # var.PLUGIN_TYPE.PLUGIN_NAME.KEY # # modules: # # ------------ Cloud Settings --------------- # Define Elastic Cloud settings here. # Format of cloud.id is a base64 value e.g. dXMtZWFzdC0xLmF3cy5mb3VuZC5pbyRub3RhcmVhbCRpZGVudGlmaWVy # and it may have an label prefix e.g. staging:dXMtZ... # This will overwrite 'var.elasticsearch.hosts' and 'var.kibana.host' # cloud.id: # # Format of cloud.auth is: : # This is optional # If supplied this will overwrite 'var.elasticsearch.username' and 'var.elasticsearch.password' # If supplied this will overwrite 'var.kibana.username' and 'var.kibana.password' # cloud.auth: elastic: # # ------------ Queuing Settings -------------- # # Internal queuing model, "memory" for legacy in-memory based queuing and # "persisted" for disk-based acked queueing. Defaults is memory # # queue.type: memory # # If using queue.type: persisted, the directory path where the data files will be stored. # Default is path.data/queue # # path.queue: # # If using queue.type: persisted, the page data files size. The queue data consists of # append-only data files separated into pages. Default is 64mb # # queue.page_capacity: 64mb # # If using queue.type: persisted, the maximum number of unread events in the queue. # Default is 0 (unlimited) # # queue.max_events: 0 # # If using queue.type: persisted, the total capacity of the queue in number of bytes. # If you would like more unacked events to be buffered in Logstash, you can increase the # capacity using this setting. Please make sure your disk drive has capacity greater than # the size specified here. If both max_bytes and max_events are specified, Logstash will pick # whichever criteria is reached first # Default is 1024mb or 1gb # # queue.max_bytes: 1024mb # # If using queue.type: persisted, the maximum number of acked events before forcing a checkpoint # Default is 1024, 0 for unlimited # # queue.checkpoint.acks: 1024 # # If using queue.type: persisted, the maximum number of written events before forcing a checkpoint # Default is 1024, 0 for unlimited # # queue.checkpoint.writes: 1024 # # If using queue.type: persisted, the interval in milliseconds when a checkpoint is forced on the head page # Default is 1000, 0 for no periodic checkpoint. # # queue.checkpoint.interval: 1000 # # ------------ Dead-Letter Queue Settings -------------- # Flag to turn on dead-letter queue. # # dead_letter_queue.enable: false # If using dead_letter_queue.enable: true, the maximum size of each dead letter queue. Entries # will be dropped if they would increase the size of the dead letter queue beyond this setting. # Default is 1024mb # dead_letter_queue.max_bytes: 1024mb # If using dead_letter_queue.enable: true, the directory path where the data files will be stored. # Default is path.data/dead_letter_queue # # path.dead_letter_queue: # # ------------ Metrics Settings -------------- # # Bind address for the metrics REST endpoint # # http.host: "127.0.0.1" # # Bind port for the metrics REST endpoint, this option also accept a range # (9600-9700) and logstash will pick up the first available ports. # # http.port: 9600-9700 # # ------------ Debugging Settings -------------- # # Options for log.level: # * fatal # * error # * warn # * info (default) # * debug # * trace # # log.level: info path.logs: /var/log/logstash # # ------------ Other Settings -------------- # # Where to find custom plugins # path.plugins: [] # # Flag to output log lines of each pipeline in its separate log file. Each log filename contains the pipeline.name # Default is false # pipeline.separate_logs: false # # ------------ X-Pack Settings (not applicable for OSS build)-------------- # # X-Pack Monitoring # https://www.elastic.co/guide/en/logstash/current/monitoring-logstash.html #xpack.monitoring.enabled: false #xpack.monitoring.elasticsearch.username: logstash_system #xpack.monitoring.elasticsearch.password: password #xpack.monitoring.elasticsearch.hosts: ["https://es1:9200", "https://es2:9200"] # an alternative to hosts + username/password settings is to use cloud_id/cloud_auth #xpack.monitoring.elasticsearch.cloud_id: monitoring_cluster_id:xxxxxxxxxx #xpack.monitoring.elasticsearch.cloud_auth: logstash_system:password #xpack.monitoring.elasticsearch.ssl.certificate_authority: ["/path/to/ca.crt"] #xpack.monitoring.elasticsearch.ssl.truststore.path: path/to/file #xpack.monitoring.elasticsearch.ssl.truststore.password: password #xpack.monitoring.elasticsearch.ssl.keystore.path: /path/to/file #xpack.monitoring.elasticsearch.ssl.keystore.password: password #xpack.monitoring.elasticsearch.ssl.verification_mode: certificate #xpack.monitoring.elasticsearch.sniffing: false #xpack.monitoring.collection.interval: 10s #xpack.monitoring.collection.pipeline.details.enabled: true # # X-Pack Management # https://www.elastic.co/guide/en/logstash/current/logstash-centralized-pipeline-management.html #xpack.management.enabled: false #xpack.management.pipeline.id: ["main", "apache_logs"] #xpack.management.elasticsearch.username: logstash_admin_user #xpack.management.elasticsearch.password: password #xpack.management.elasticsearch.hosts: ["https://es1:9200", "https://es2:9200"] # an alternative to hosts + username/password settings is to use cloud_id/cloud_auth #xpack.management.elasticsearch.cloud_id: management_cluster_id:xxxxxxxxxx #xpack.management.elasticsearch.cloud_auth: logstash_admin_user:password #xpack.management.elasticsearch.ssl.certificate_authority: ["/path/to/ca.crt"] #xpack.management.elasticsearch.ssl.truststore.path: /path/to/file #xpack.management.elasticsearch.ssl.truststore.password: password #xpack.management.elasticsearch.ssl.keystore.path: /path/to/file #xpack.management.elasticsearch.ssl.keystore.password: password #xpack.management.elasticsearch.ssl.verification_mode: certificate #xpack.management.elasticsearch.sniffing: false #xpack.management.logstash.poll_interval: 5s ``` * * * ``` root@elastic_syslog:/etc/logstash# more pipelines.yml # This file is where you define your pipelines. You can define multiple. # For more information on multiple pipelines, see the documentation: # https://www.elastic.co/guide/en/logstash/current/multiple-pipelines.html - pipeline.id: main path.config: "/etc/logstash/conf.d/*.conf" ``` * * * ``` root@elastic_syslog:/etc/logstash# vi startup.options ################################################################################ # These settings are ONLY used by $LS_HOME/bin/system-install to create a custom # startup script for Logstash and is not used by Logstash itself. It should # automagically use the init system (systemd, upstart, sysv, etc.) that your # Linux distribution uses. # # After changing anything here, you need to re-run $LS_HOME/bin/system-install # as root to push the changes to the init script. ################################################################################ # Override Java location #JAVACMD=/usr/bin/java # Set a home directory LS_HOME=/usr/share/logstash # logstash settings directory, the path which contains logstash.yml LS_SETTINGS_DIR=/etc/logstash # Arguments to pass to logstash LS_OPTS="--path.settings ${LS_SETTINGS_DIR}" # Arguments to pass to java LS_JAVA_OPTS="" # pidfiles aren't used the same way for upstart and systemd; this is for sysv users. LS_PIDFILE=/var/run/logstash.pid # user and group id to be invoked as LS_USER=logstash LS_GROUP=logstash # Enable GC logging by uncommenting the appropriate lines in the GC logging # section in jvm.options LS_GC_LOG_FILE=/var/log/logstash/gc.log # Open file limit LS_OPEN_FILES=16384 # Nice level LS_NICE=19 # Change these to have the init script named and described differently # This is useful when running multiple instances of Logstash on the same # physical box or vm SERVICE_NAME="logstash" SERVICE_DESCRIPTION="logstash" # If you need to run a command or script before launching Logstash, put it # between the lines beginning with `read` and `EOM`, and uncomment those lines. ### ## read -r -d '' PRESTART << EOM ## EOM ``` ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [April 7, 2020, 3:17pm UTC](https://discuss.elastic.co/t/unable-to-create-logstash-index-in-kibana/222790/8 "2020-04-07T15:17:10Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.