Index not showing up in Kibana while using Logstash

Elastic Stack Logstash
1

Hello everyone, I want to load a json file into elasticsearch using Logstash.
I am following all the steps but the index is still not showing up in Kibana.
For reference, this is my Logstash configuration file :

input {
  file {
    path => "my_path/smartlight_and_weather_data.json"
    start_position => "beginning"
    sincedb_path => "NUL"
    codec => json_lines { target => "doc" }
  }
} 

output {
  elasticsearch {
    hosts => ["https://localhost:9200"]
    user => "my_user"
    password => "my_pw"
    index => "smartlight_and_weather_data"
    ssl => true
    cacert => "my_cacert_path.crt"
    ssl_verification_mode => "full"
  }
  stdout {
    codec => rubydebug
  }
}

And this is an example of my last Logstash logs :

[2024-06-14T16:53:57,430][INFO ][logstash.javapipeline    ][main] Starting pipeline {:pipeline_id=>"main", "pipeline.workers"=>16, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>50, "pipeline.max_inflight"=>2000, "pipeline.sources"=>["C:/Users/ASUS/Downloads/logstash-8.13.4/logstash.conf"], :thread=>"#<Thread:0xd6f3b41 C:/Users/ASUS/Downloads/logstash-8.13.4/logstash-core/lib/logstash/java_pipeline.rb:134 run>"}
[2024-06-14T16:53:58,183][INFO ][logstash.javapipeline    ][main] Pipeline Java execution initialization time {"seconds"=>0.75}
[2024-06-14T16:53:58,208][INFO ][logstash.javapipeline    ][main] Pipeline started {"pipeline.id"=>"main"}
[2024-06-14T16:53:58,213][INFO ][filewatch.observingtail  ][main][4bbffa1e730280c0aab5af074cec475fa2279412768d859e69c5355d0ce667c1] START, creating Discoverer, Watch with file and sincedb collections
[2024-06-14T16:53:58,219][INFO ][logstash.agent           ] Pipelines running {:count=>1, :running_pipelines=>[:main], :non_running_pipelines=>[]}

It says the pipeline is running successfully but i'm still not seeing the index in Kibana.
Any help to solve this would be appreciated !

2

Official Elastic Search 8.14 Repo via JARaaS Hybrid RAG - Documentation - code current 6/19/2024 [AI Generated]

Note: Sources at the end of the response

It looks like your Logstash configuration and the logs provided indicate that Logstash is starting and the pipeline is running. However, the data may not be reaching Elasticsearch or being indexed properly. Here are several steps you can take to troubleshoot and resolve this issue:

  1. Check Logstash Output: First, check if Logstash is reading the file and outputting it to the console with the rubydebug codec. This will help you verify if Logstash is ingesting the data properly from the file.

  2. Elasticsearch Connectivity: Make sure Logstash can connect to Elasticsearch. Test the connection from the machine running Logstash to Elasticsearch using a simple curl command:

    curl -u my_user:my_pw --cacert my_cacert_path.crt https://localhost:9200

    You should see a response from Elasticsearch.

  3. File Input Configuration: Ensure the file path is correct and Logstash has permissions to read the file. Also, verify the JSON is valid.

  4. Index Creation in Elasticsearch: Check if the index is being created in Elasticsearch. You can do this by making a request to Elasticsearch directly:

    curl -u my_user:my_pw --cacert my_cacert_path.crt https://localhost:9200/_cat/indices?v

    Look for your smartlight_and_weather_data index in the list.

  5. Logstash Logging: It seems like you're seeing only part of the logs. Increase the verbosity of the logs to debug level to get more information. To do this, you might need to adjust Logstash's log4j2.properties file (usually found in the config directory):

    status = debug

  6. Elasticsearch Output Configuration: Ensure your elasticsearch output configuration is correct and matches your Elasticsearch setup.

Here's an updated version of your Logstash config with some added debug options:

input {
  file {
    path => "/path/to/your/json/smartlight_and_weather_data.json"
    start_position => "beginning"
    sincedb_path => "NUL"
    codec => json_lines { target => "doc" }
  }
}

output {
  elasticsearch {
    hosts => ["https://localhost:9200"]
    user => "my_user"
    password => "my_pw"
    index => "smartlight_and_weather_data"
    ssl => true
    cacert => "/path/to/your/ca.crt"
    ssl_verification_mode => "full"
  }
  stdout {
    codec => rubydebug
  }
}

Other Considerations:

  1. Data Format: Ensure each line in your JSON file is a valid JSON object. Example of JSON lines:
{"field1": "value1", "field2": "value2"}
{"field1": "value3", "field2": "value4"}
  1. Elasticsearch Index Mappings: If there are specific mappings required, you might need to pre-configure the index in Elasticsearch to ensure the data is ingested correctly.

If you've followed all these steps and still don't see the index, it could be helpful to:

  • Check Elasticsearch logs for any potential issues on the Elasticsearch side.
  • Verify if there are any errors or warnings in Logstash logs that might provide more context.

This methodical approach should help you identify the point of failure or misconfiguration.

1 Like
3

Are you using the full path?

What does your log looks like inside the file? Can you share some sample messages?

It has more than one line?

1 Like
4

Thank you for your reply !
Yes, I am actually using the full path for my json file.
These are the last logs i see in the file and it has more than one line this way :

["C:/Users/ASUS/Downloads/logstash-8.13.4/logstash.conf"], :thread=>"#<Thread:0x1ca3adff C:/Users/ASUS/Downloads/logstash-8.13.4/logstash-core/lib/logstash/java_pipeline.rb:134 run>"}
[2024-06-19T20:31:31,259][INFO ][logstash.javapipeline    ][main] Pipeline Java execution initialization time {"seconds"=>0.69}
[2024-06-19T20:31:31,279][INFO ][logstash.javapipeline    ][main] Pipeline started {"pipeline.id"=>"main"}
[2024-06-19T20:31:31,287][INFO ][filewatch.observingtail  ][main][aafdc8a076f0d84187724ad15f4f91129fa77cd97b20f7fe408df0b4d6edf363] START, creating Discoverer, Watch with file and sincedb collections
[2024-06-19T20:31:31,290][INFO ][logstash.agent           ] Pipelines running {:count=>1, :running_pipelines=>[:main], :non_running_pipelines=>[]}
5

Thank you for all of your suggestions!
I have tried the 5th and 6th steps but it's unfortunately still not showing up in Kibana and the logs are not giving me much information.
For reference, these are logstash's logs :

["C:/Users/ASUS/Downloads/logstash-8.13.4/logstash.conf"], :thread=>"#<Thread:0x7a92ba3d C:/Users/ASUS/Downloads/logstash-8.13.4/logstash-core/lib/logstash/java_pipeline.rb:134 run>"}
[2024-06-19T20:47:22,172][INFO ][logstash.javapipeline    ][main] Pipeline Java execution initialization time {"seconds"=>0.82}
[2024-06-19T20:47:22,198][INFO ][logstash.javapipeline    ][main] Pipeline started {"pipeline.id"=>"main"}
[2024-06-19T20:47:22,206][INFO ][filewatch.observingtail  ][main][aafdc8a076f0d84187724ad15f4f91129fa77cd97b20f7fe408df0b4d6edf363] START, creating Discoverer, Watch with file and sincedb collections
[2024-06-19T20:47:22,219][INFO ][logstash.agent           ] Pipelines running {:count=>1, :running_pipelines=>[:main], :non_running_pipelines=>[]}
6

Do not use a json_lines codec with a file input. The file input reads newline delimited lines from the file, strips off the newline, then feeds them to the codec. The codec then waits forever, collecting lines until it gets a newline (which it will never receive).

Change the code to json.

2 Likes
7

Thank you a lot for this !
As per your suggestion, I just changed codec => json and it worked perfectly. The index is finally showing up in Kibana.