# Index not showing up in Kibana while using Logstash **URL:** \ **Category:** Logstash\ **Created:** [June 19, 2024, 9:03am UTC](https://discuss.elastic.co/t/index-not-showing-up-in-kibana-while-using-logstash/361717 "2024-06-19T09:03:32Z")\ **Posts on this page:** 7\ **Page:** 1
**Author:** ![dou07](https://avatars.discourse-cdn.com/v4/letter/d/c89c15/32.png) [@dou07](https://discuss.elastic.co/u/dou07)\ **Post date:** [June 19, 2024, 9:03am UTC](https://discuss.elastic.co/t/index-not-showing-up-in-kibana-while-using-logstash/361717/1 "2024-06-19T09:03:32Z")
Hello everyone, I want to load a json file into elasticsearch using Logstash. I am following all the steps but the index is still not showing up in Kibana. For reference, this is my Logstash configuration file : ```auto input { file { path => "my_path/smartlight_and_weather_data.json" start_position => "beginning" sincedb_path => "NUL" codec => json_lines { target => "doc" } } } output { elasticsearch { hosts => ["https://localhost:9200"] user => "my_user" password => "my_pw" index => "smartlight_and_weather_data" ssl => true cacert => "my_cacert_path.crt" ssl_verification_mode => "full" } stdout { codec => rubydebug } } ``` And this is an example of my last Logstash logs : ```auto [2024-06-14T16:53:57,430][INFO][logstash.javapipeline][main] Starting pipeline {:pipeline_id=>"main", "pipeline.workers"=>16, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>50, "pipeline.max_inflight"=>2000, "pipeline.sources"=>["C:/Users/ASUS/Downloads/logstash-8.13.4/logstash.conf"], :thread=>"#"} [2024-06-14T16:53:58,183][INFO][logstash.javapipeline][main] Pipeline Java execution initialization time {"seconds"=>0.75} [2024-06-14T16:53:58,208][INFO][logstash.javapipeline][main] Pipeline started {"pipeline.id"=>"main"} [2024-06-14T16:53:58,213][INFO][filewatch.observingtail][main][4bbffa1e730280c0aab5af074cec475fa2279412768d859e69c5355d0ce667c1] START, creating Discoverer, Watch with file and sincedb collections [2024-06-14T16:53:58,219][INFO][logstash.agent] Pipelines running {:count=>1, :running_pipelines=>[:main], :non_running_pipelines=>[]} ``` It says the pipeline is running successfully but i'm still not seeing the index in Kibana. Any help to solve this would be appreciated ! ---
**Author:** ![rtwolfe94022](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/rtwolfe94022/32/132253_2.png) [@rtwolfe94022](https://discuss.elastic.co/u/rtwolfe94022)\ **Post date:** [June 19, 2024, 2:21pm UTC](https://discuss.elastic.co/t/index-not-showing-up-in-kibana-while-using-logstash/361717/2 "2024-06-19T14:21:12Z")
Official Elastic Search 8.14 Repo via JARaaS Hybrid RAG - Documentation - code current 6/19/2024 [AI Generated] **Note: Sources at the end of the response** It looks like your Logstash configuration and the logs provided indicate that Logstash is starting and the pipeline is running. However, the data may not be reaching Elasticsearch or being indexed properly. Here are several steps you can take to troubleshoot and resolve this issue: 1. **Check Logstash Output** : First, check if Logstash is reading the file and outputting it to the console with the `rubydebug` codec. This will help you verify if Logstash is ingesting the data properly from the file. 2. **Elasticsearch Connectivity** : Make sure Logstash can connect to Elasticsearch. Test the connection from the machine running Logstash to Elasticsearch using a simple curl command: 3. **File Input Configuration** : Ensure the file path is correct and Logstash has permissions to read the file. Also, verify the JSON is valid. 4. **Index Creation in Elasticsearch** : Check if the index is being created in Elasticsearch. You can do this by making a request to Elasticsearch directly: 5. **Logstash Logging** : It seems like you're seeing only part of the logs. Increase the verbosity of the logs to debug level to get more information. To do this, you might need to adjust Logstash's `log4j2.properties` file (usually found in the `config` directory): 6. **Elasticsearch Output Configuration** : Ensure your `elasticsearch` output configuration is correct and matches your Elasticsearch setup. Here's an updated version of your Logstash config with some added debug options: ```plaintext input { file { path => "/path/to/your/json/smartlight_and_weather_data.json" start_position => "beginning" sincedb_path => "NUL" codec => json_lines { target => "doc" } } } output { elasticsearch { hosts => ["https://localhost:9200"] user => "my_user" password => "my_pw" index => "smartlight_and_weather_data" ssl => true cacert => "/path/to/your/ca.crt" ssl_verification_mode => "full" } stdout { codec => rubydebug } } ``` _ **Other Considerations:** _ 1. **Data Format** : Ensure each line in your JSON file is a valid JSON object. Example of JSON lines: ```json {"field1": "value1", "field2": "value2"} {"field1": "value3", "field2": "value4"} ``` 1. **Elasticsearch Index Mappings** : If there are specific mappings required, you might need to pre-configure the index in Elasticsearch to ensure the data is ingested correctly. If you've followed all these steps and still don't see the index, it could be helpful to: - Check Elasticsearch logs for any potential issues on the Elasticsearch side. - Verify if there are any errors or warnings in Logstash logs that might provide more context. This methodical approach should help you identify the point of failure or misconfiguration. ---
**Author:** ![leandrojmp](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/leandrojmp/32/107231_2.png) [@leandrojmp](https://discuss.elastic.co/u/leandrojmp)\ **Post date:** [June 19, 2024, 2:33pm UTC](https://discuss.elastic.co/t/index-not-showing-up-in-kibana-while-using-logstash/361717/3 "2024-06-19T14:33:58Z")
> [@dou07](#): > > `path => "my_path/smartlight_and_weather_data.json"` Are you using the full path? What does your log looks like inside the file? Can you share some sample messages? It has more than one line? ---
**Author:** ![dou07](https://avatars.discourse-cdn.com/v4/letter/d/c89c15/32.png) [@dou07](https://discuss.elastic.co/u/dou07)\ **Post date:** [June 19, 2024, 7:34pm UTC](https://discuss.elastic.co/t/index-not-showing-up-in-kibana-while-using-logstash/361717/4 "2024-06-19T19:34:30Z")
Thank you for your reply ! Yes, I am actually using the full path for my json file. These are the last logs i see in the file and it has more than one line this way : ```auto ["C:/Users/ASUS/Downloads/logstash-8.13.4/logstash.conf"], :thread=>"#"} [2024-06-19T20:31:31,259][INFO][logstash.javapipeline][main] Pipeline Java execution initialization time {"seconds"=>0.69} [2024-06-19T20:31:31,279][INFO][logstash.javapipeline][main] Pipeline started {"pipeline.id"=>"main"} [2024-06-19T20:31:31,287][INFO][filewatch.observingtail][main][aafdc8a076f0d84187724ad15f4f91129fa77cd97b20f7fe408df0b4d6edf363] START, creating Discoverer, Watch with file and sincedb collections [2024-06-19T20:31:31,290][INFO][logstash.agent] Pipelines running {:count=>1, :running_pipelines=>[:main], :non_running_pipelines=>[]} ``` ---
**Author:** ![dou07](https://avatars.discourse-cdn.com/v4/letter/d/c89c15/32.png) [@dou07](https://discuss.elastic.co/u/dou07)\ **Post date:** [June 19, 2024, 7:54pm UTC](https://discuss.elastic.co/t/index-not-showing-up-in-kibana-while-using-logstash/361717/5 "2024-06-19T19:54:07Z")
Thank you for all of your suggestions! I have tried the 5th and 6th steps but it's unfortunately still not showing up in Kibana and the logs are not giving me much information. For reference, these are logstash's logs : ```auto ["C:/Users/ASUS/Downloads/logstash-8.13.4/logstash.conf"], :thread=>"#"} [2024-06-19T20:47:22,172][INFO][logstash.javapipeline][main] Pipeline Java execution initialization time {"seconds"=>0.82} [2024-06-19T20:47:22,198][INFO][logstash.javapipeline][main] Pipeline started {"pipeline.id"=>"main"} [2024-06-19T20:47:22,206][INFO][filewatch.observingtail][main][aafdc8a076f0d84187724ad15f4f91129fa77cd97b20f7fe408df0b4d6edf363] START, creating Discoverer, Watch with file and sincedb collections [2024-06-19T20:47:22,219][INFO][logstash.agent] Pipelines running {:count=>1, :running_pipelines=>[:main], :non_running_pipelines=>[]} ``` ---
**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\ **Post date:** [June 19, 2024, 8:58pm UTC](https://discuss.elastic.co/t/index-not-showing-up-in-kibana-while-using-logstash/361717/6 "2024-06-19T20:58:41Z")
> [@dou07](#): > > ```auto > file { > path => "my_path/smartlight_and_weather_data.json" > start_position => "beginning" > sincedb_path => "NUL" > codec => json_lines { target => "doc" } > } > > ``` Do not use a json\_lines codec with a file input. The file input reads newline delimited lines from the file, strips off the newline, then feeds them to the codec. The codec then waits forever, collecting lines until it gets a newline (which it will never receive). Change the code to json. ---
**Author:** ![dou07](https://avatars.discourse-cdn.com/v4/letter/d/c89c15/32.png) [@dou07](https://discuss.elastic.co/u/dou07)\ **Post date:** [June 19, 2024, 10:46pm UTC](https://discuss.elastic.co/t/index-not-showing-up-in-kibana-while-using-logstash/361717/7 "2024-06-19T22:46:07Z")
Thank you a lot for this ! As per your suggestion, I just changed codec =\> json and it worked perfectly. The index is finally showing up in Kibana.