# Index not creating **URL:** \ **Category:** Logstash\ **Tags:** docker\ **Created:** [September 27, 2021, 12:31pm UTC](https://discuss.elastic.co/t/index-not-creating/285232 "2021-09-27T12:31:21Z")\ **Posts on this page:** 18\ **Page:** 1
**Author:** ![jubin03](https://avatars.discourse-cdn.com/v4/letter/j/85e7bf/32.png) [@jubin03](https://discuss.elastic.co/u/jubin03)\ **Post date:** [September 27, 2021, 12:31pm UTC](https://discuss.elastic.co/t/index-not-creating/285232/1 "2021-09-27T12:31:21Z")
Following with this topic [Index not creating - #17 by jubin03](https://discuss.elastic.co/t/index-not-creating/282345/17) I have created a new ELK and fetched data and it is working fine and able to create index patter, but the existing one is not working. Could somebody have any idea? ---
**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\ **Post date:** [September 28, 2021, 4:28am UTC](https://discuss.elastic.co/t/index-not-creating/285232/2 "2021-09-28T04:28:34Z")
Did you follow up on [Index not creating - #18 by aaron-nimocks](https://discuss.elastic.co/t/index-not-creating/282345/18)? ---
**Author:** ![jubin03](https://avatars.discourse-cdn.com/v4/letter/j/85e7bf/32.png) [@jubin03](https://discuss.elastic.co/u/jubin03)\ **Post date:** [September 28, 2021, 6:05pm UTC](https://discuss.elastic.co/t/index-not-creating/285232/3 "2021-09-28T18:05:23Z")
Yes, I have done the same and no error was found while running logstash, but no index pattern was created. Please find the logs ```auto [[main]-pipeline-manager] INFO logstash.javapipeline - Starting pipeline {:pipeline_id=>"main", "pipeline.workers"=>2, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>50, "pipeline.max_inflight"=>250, "pipeline.sources"=>["/usr/share/logstash/config/conf.d/1password-filter.conf", "/usr/share/logstash/config/conf.d/filebeat-filter.conf", "/usr/share/logstash/config/conf.d/logstash.conf", "/usr/share/logstash/config/conf.d/pfsense-filter.conf", "/usr/share/logstash/config/conf.d/webhook-filter.conf"], :thread=>"#"} 18:21:28.920 [[main]-pipeline-manager] INFO logstash.javapipeline - Pipeline Java execution initialization time {"seconds"=>1.71} 18:21:29.047 [[main]-pipeline-manager] INFO logstash.inputs.beats - Starting input listener {:address=>"0.0.0.0:5044"} 18:21:29.105 [[main]-pipeline-manager] INFO logstash.javapipeline - Pipeline started {"pipeline.id"=>"main"} 18:21:29.125 [[main]"0.0.0.0:3233", :ssl=>"false"} 18:21:29.159 [Agent thread] INFO logstash.agent - Pipelines running {:count=>1, :running_pipelines=>[:main], :non_running_pipelines=>[]} 18:21:29.235 [[main]"0.0.0.0:5044"} 18:21:29.236 [[main]"0.0.0.0:5044", :receive_buffer_bytes=>"106496", :queue_size=>"2000"} ` ``` ---
**Author:** ![aaron-nimocks](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/aaron-nimocks/32/73965_2.png) [@aaron-nimocks](https://discuss.elastic.co/u/aaron-nimocks)\ **Post date:** [September 28, 2021, 6:25pm UTC](https://discuss.elastic.co/t/index-not-creating/285232/4 "2021-09-28T18:25:07Z")
The idea behind changing your output from Elasticsearch to stdout is to test your data and if it's flowing. When you do the below and run it you should see records on the screen being processed. If you do not, then your issue is in your filter or input. What results do you see when you run it? ```auto output { stdout {} } ``` ---
**Author:** ![jubin03](https://avatars.discourse-cdn.com/v4/letter/j/85e7bf/32.png) [@jubin03](https://discuss.elastic.co/u/jubin03)\ **Post date:** [September 29, 2021, 6:17am UTC](https://discuss.elastic.co/t/index-not-creating/285232/5 "2021-09-29T06:17:31Z")
Yes, I have enabled as you said, and can see other filter data flowing, but not from the JSON file. ---
**Author:** ![aaron-nimocks](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/aaron-nimocks/32/73965_2.png) [@aaron-nimocks](https://discuss.elastic.co/u/aaron-nimocks)\ **Post date:** [September 29, 2021, 10:45am UTC](https://discuss.elastic.co/t/index-not-creating/285232/6 "2021-09-29T10:45:33Z")
Can you post your entire conf? If you ran the below and data didn't flow then it most likely isn't Logstash and is your source files not having the data you expect it to. ```auto input { file { path => "/path/*.json" start_position => "beginning" sincedb_path => "/dev/null" } } output { stdout {} } ``` ---
**Author:** ![jubin03](https://avatars.discourse-cdn.com/v4/letter/j/85e7bf/32.png) [@jubin03](https://discuss.elastic.co/u/jubin03)\ **Post date:** [September 29, 2021, 11:01am UTC](https://discuss.elastic.co/t/index-not-creating/285232/7 "2021-09-29T11:01:13Z")
Please find the configurations Logstash conf ```auto # Filebeat input input { beats { port => 5044 } } #File input input { file { path => "/path/*.json" start_position => "beginning" sincedb_path => "/dev/null" } } # webhook input { http { port => port number type => "webhook" } } #udp syslogs stream via 5044 input { udp { type => "syslog" port => 5044 } } output { if [type] == "syslog" { elasticsearch { ssl => true ssl_certificate_verification => false user => username password => "password" action => "index" hosts => ["https://elk.com:9200"] index => "logstash-pfsense-%{+YYYY.MM.dd}" } } else if [type] == "signinattempts" { elasticsearch { ssl => true ssl_certificate_verification => false user => username password => "password" action => "index" hosts => ["https://elk.com:9200"] index => "events-%{+YYYY.MM.dd}" } } else if [type] == "webhook" { elasticsearch { ssl => true ssl_certificate_verification => false user => username password => "password" action => "index" hosts => ["https://elk.com:9200"] index => "webhook-%{+YYYY.MM.dd}" } } else { elasticsearch { ssl => true ssl_certificate_verification => false user => username password => "password" action => "index" hosts => ["https://elk.com:9200"] index => "%{[@metadata][beat]}-%{+YYYY.MM.dd}" } } stdout { codec => rubydebug } } ``` Filter.conf ```auto filter { if [type] == "signinattempts" { json { source => "message" } } } ``` ---
**Author:** ![aaron-nimocks](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/aaron-nimocks/32/73965_2.png) [@aaron-nimocks](https://discuss.elastic.co/u/aaron-nimocks)\ **Post date:** [September 29, 2021, 11:05am UTC](https://discuss.elastic.co/t/index-not-creating/285232/8 "2021-09-29T11:05:25Z")
None of your inputs set type to `signinattempts`. So type will never == signinattempts. ---
**Author:** ![jubin03](https://avatars.discourse-cdn.com/v4/letter/j/85e7bf/32.png) [@jubin03](https://discuss.elastic.co/u/jubin03)\ **Post date:** [September 29, 2021, 11:08am UTC](https://discuss.elastic.co/t/index-not-creating/285232/9 "2021-09-29T11:08:22Z")
I haven't added type input for webhook that filter also same as this and working fine. Could you please suggest what changes required on the conf ---
**Author:** ![aaron-nimocks](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/aaron-nimocks/32/73965_2.png) [@aaron-nimocks](https://discuss.elastic.co/u/aaron-nimocks)\ **Post date:** [September 29, 2021, 11:12am UTC](https://discuss.elastic.co/t/index-not-creating/285232/10 "2021-09-29T11:12:18Z")
I don't understand the problem still. If you are expecting data from all json files from that path to be processed by Logstash but running this exact config does not produce records then I would look at your json files and not your conf to start with. ```auto input { file { path => "/path/*.json" start_position => "beginning" sincedb_path => "/dev/null" } } output { stdout {} } ``` ---
**Author:** ![jubin03](https://avatars.discourse-cdn.com/v4/letter/j/85e7bf/32.png) [@jubin03](https://discuss.elastic.co/u/jubin03)\ **Post date:** [September 29, 2021, 11:17am UTC](https://discuss.elastic.co/t/index-not-creating/285232/11 "2021-09-29T11:17:26Z")
I have created a script to fetch some events data to a single JSON file via API auth also checked. Due to security concerns, I can't share the JSON file. Seems like the data inside the file is not reading logstash ---
**Author:** ![aaron-nimocks](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/aaron-nimocks/32/73965_2.png) [@aaron-nimocks](https://discuss.elastic.co/u/aaron-nimocks)\ **Post date:** [September 29, 2021, 11:31am UTC](https://discuss.elastic.co/t/index-not-creating/285232/12 "2021-09-29T11:31:19Z")
I would verify that is valid JSON in that file. Also run Logstash in `--debug` mode to see if it give you any other hints. Are the JSON records 1 per line or multiline? Can you give a sample of how it's structured without using the actual data? It should look like this with how your input is setup. ```auto {"message": "value", "another": "field"} {"message": "value", "another": "field"} {"message": "value", "another": "field"} {"message": "value", "another": "field"} {"message": "value", "another": "field"} ``` ---
**Author:** ![jubin03](https://avatars.discourse-cdn.com/v4/letter/j/85e7bf/32.png) [@jubin03](https://discuss.elastic.co/u/jubin03)\ **Post date:** [September 29, 2021, 11:39am UTC](https://discuss.elastic.co/t/index-not-creating/285232/13 "2021-09-29T11:39:23Z")
This is the file sample content. I have removed the datas only ```auto {"uuid":"","session_uuid":"","timestamp":"2021-07-02T03:55:35Z","country":"US","category":"success","type":"mfa_ok","details":null,"target_user":{"uuid":"","name":"","email":""},"client":{"app_name":" Browser Extension","app_version":"20166","platform_name":"Firefox","platform_version":"68.0","os_name":"Windows","os_version":"10.0","ip_address":""}} ``` ---
**Author:** ![aaron-nimocks](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/aaron-nimocks/32/73965_2.png) [@aaron-nimocks](https://discuss.elastic.co/u/aaron-nimocks)\ **Post date:** [September 29, 2021, 11:52am UTC](https://discuss.elastic.co/t/index-not-creating/285232/14 "2021-09-29T11:52:50Z")
Taking your sample data and running the below conf produces the below output and works. Next part to verify would be the `path` to your json files. **Conf** ```auto input { file { path => "/Applications/elastic/logstash/config/data1.json" sincedb_path => "/dev/null" start_position => "beginning" codec => json } } output { stdout { codec => json_lines } } ``` **Output** ```auto { "path": "/Applications/elastic/logstash/config/data1.json", "target_user": { "uuid": "", "email": "", "name": "" }, "@version": "1", "@timestamp": "2021-09-29T11:48:21.644Z", "details": null, "uuid": "", "timestamp": "2021-07-02T03:55:35Z", "client": { "app_version": "20166", "app_name": " Browser Extension", "platform_name": "Firefox", "ip_address": "", "os_name": "Windows", "platform_version": "68.0", "os_version": "10.0" }, "host": "Aarons-MBP", "category": "success", "type": "mfa_ok", "country": "US", "session_uuid": "" } ``` ---
**Author:** ![jubin03](https://avatars.discourse-cdn.com/v4/letter/j/85e7bf/32.png) [@jubin03](https://discuss.elastic.co/u/jubin03)\ **Post date:** [September 29, 2021, 11:56am UTC](https://discuss.elastic.co/t/index-not-creating/285232/15 "2021-09-29T11:56:02Z")
I have added this file in a path and the permission is chmod 777 ---
**Author:** ![aaron-nimocks](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/aaron-nimocks/32/73965_2.png) [@aaron-nimocks](https://discuss.elastic.co/u/aaron-nimocks)\ **Post date:** [September 29, 2021, 12:04pm UTC](https://discuss.elastic.co/t/index-not-creating/285232/16 "2021-09-29T12:04:33Z")
I would run just the below using `--debug` option and post results. ```auto input { file { path => "/Applications/elastic/logstash/config/data1.json" sincedb_path => "/dev/null" start_position => "beginning" codec => json } } output { stdout { codec => json_lines } } ``` ---
**Author:** ![jubin03](https://avatars.discourse-cdn.com/v4/letter/j/85e7bf/32.png) [@jubin03](https://discuss.elastic.co/u/jubin03)\ **Post date:** [October 14, 2021, 6:22pm UTC](https://discuss.elastic.co/t/index-not-creating/285232/17 "2021-10-14T18:22:40Z")
Now I'm able to create an index, but in Kibana the index shows the data from beats not from the JSON file. I have used the same data which I mentioned in the previous chat Conf file ```auto #events input { file { path => "/path/events.json" sincedb_path => "/dev/null" start_position => "beginning" codec => json } } filter { json { source => "message" } } output { elasticsearch { ssl => true ssl_certificate_verification => false user => password => action => "index" hosts => ["https://url:9200"] index => "events-%{+YYYY.MM.dd}" } } ``` FYI: It is dockerized ELK. No errors were found from docker logs ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [November 11, 2021, 6:22pm UTC](https://discuss.elastic.co/t/index-not-creating/285232/18 "2021-11-11T18:22:59Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.