Seems that Logstash stop shipping logs to ES / Problem into the logstash config

[Rodolphe_Ghio]

Hello everyone, I'm new in the ELK community. I'm trying to setup a kibana sandbox (linux vm). So I've put data into the logstash. I've created index / mapping thanks to the Dev tool.
But when I'm trying to set a default index, kibana don't find any index

So here's the two index I've created.

.
Can anyone help me ? or give me a clue ?

EDITED:
It seems that something is misconfigured on the Logstash end
Here's my Input file and my output

Input

input {
  beats {
     port => 5044
     tags => "fromBeats"
  }
  file {
    path => [
	"/home/secunix/logs/*",
	"/tech/*"
	]
    start_position => "beginning"

    sincedb_path => "/dev/null"
  }
  tcp {
    port => 5514
    type => "syslog"
    tags => "from Syslog-ng"
  } 
}

filter {
    grok {
      match => ["message", "<(?<sys_priority>\d+?)>(?<syslog_timestamp>%{CISCOTIMESTAMP})\s(?<logsource>%{URIHOST})(\s(?:(?<application>.*?)(%(?<project>.*?))?))?:(?:\s)?(?<logmessage>.*$)"]
    }
    if [logmessage] {
      mutate {
        replace => [ "message", "%{logmessage}" ]
        remove_field => [ "logmessage" ]
      }
    }
    if [project]  {
      mutate {
        replace => [ "type", "%{project}" ]
        remove_field => [ "project" ]
      }
    }else if [application] {
      mutate {
        lowercase => [ "application" ]
      }
      mutate {
        gsub => [ "application", " ", "_" ]
      }
      mutate {
        replace => [ "type", "%{application}" ]
      }
    }else {
      mutate {
        replace => [ "type", "uknapp" ]
        add_field => { "application" => "uknapp" }
      }
    }
}

Output

output {
   elasticsearch {
        hosts => ["localhost:9200"] //host pour push l’index
        index => "logstash-%{+YYYY.MM.dd}" //Nom de l’index à la sortie
        #+++ sa Added by scr-sop-af-config-elksandbox. Do not remove this line.
user => "logstash" //Logs de connexion, ne pas toucher
#--- sa Added by scr-sop-af-config-elksandbox. Do not remove this line.
        #+++ sa Added by scr-sop-af-config-elksandbox. Do not remove this line.
password => "logstash"
#--- sa Added by scr-sop-af-config-elksandbox. Do not remove this line.

   }
}

and this is the kinf of logs i've :

{"index":{"_index":"shakespeare","_id":0}}
{"type":"act","line_id":1,"play_name":"Henry IV", "speech_number":"","line_number":"","speaker":"","text_entry":"ACT I"}

[azasypkin]

Hi @Rodolphe_Ghio,

I see the indices you're referring to don't have any docs in them, that's the reason why you don't see them in Kibana UI. That means you need to have at least on document in the index to be able to create a pattern for it in Kibana.

You can follow https://github.com/elastic/kibana/issues/35481 where possible improvements are discussed.

--
Oleg

[Rodolphe_Ghio]

@azasypkin

have any doc

Thanks for yyour answer but how can I add or link data to those index.
Cause in my point of view I've added some logs into the folder wich is into the input file in logstash.

[azasypkin]

@Rodolphe_Ghio

Cause in my point of view I've added some logs into the folder wich is into the input file in logstash.

If you don't see any documents in logstash indices, probably something in misconfigured on the Logstash end or there is nothing to ingest. Let me forward your question to the Logstash hub. In the meantime please share more info on your logstash setup/configuration.

--
Oleg

[Rodolphe_Ghio]

Thanks a lot.

[Badger]

@Rodolphe_Ghio

file {
    path => [
        "/home/secunix/logs/",
        "/tech/"
    ]
    start_position => "beginning"
    sincedb_path => "/dev/null"
}

If you are really giving the file input directories it will not do anything, you need to give it paths to files, not directories.

Please edit your post, select the configuration and click on </> in the toolbar above the edit pane to blockquote it. Then make sure that the configuration in the preview panel on the right matches your configuration.

[Rodolphe_Ghio]

Ok, but that's why I've put the "*" behind the "/" cause I think it select all the files in the directories. What do you think about this ? Do you think it will not work ?

file {
    path => [
        "/home/secunix/logs/*",
        "/tech/*"
    ]
    start_position => "beginning"
    sincedb_path => "/dev/null"
}

[Badger]

I would expect that to work. You could run with "--log.level trace" on the logstash command line and see what filewatch has to say.

[Rodolphe_Ghio]

Sorry, but I'm not sure to understand, where is the logstash command line, or where I've to type "--log.level trace" cause I've only access to the dev tool and my linux vm.

[Badger]

If you cannot modify the command line used to start logstash you could follow the instructions at the tail end of this post. Read the whole post.

[Rodolphe_Ghio]

As you can see I don't have any TRACE loggers

[Badger]

It would be up to you to use the API to modify the filewatch loggers to TRACE, as the post I linked to explained.

The strange thing is you do not have any loggers for inputs, filters, or outputs, which suggests you do not have any pipelines configured.

[Rodolphe_Ghio]

Bump

[Rodolphe_Ghio]

My conf file was wrong