Since we moved 7.15 "Pushing flush onto pipeline." and files are not parsed

[fabeau]

We deployed logstash config files on a new server running 7.15 from a 7.11 server working perfectly .

We validated .conf files in command line and everything is fine.

we parse some log files locally on the server and send it out to elastic.

On the 7.15 server (Centos) there are no error and continuous logs "Pushing flush onto pipeline :

[2021-10-13T09:17:25,878][DEBUG][logstash.outputs.file    ] Starting flush cycle
[2021-10-13T09:17:26,922][DEBUG][filewatch.sincedbcollection] writing sincedb (delta since last write = 15)
[2021-10-13T09:17:26,922][TRACE][filewatch.sincedbcollection] sincedb_write: /var/lib/logstash/plugins/inputs/file/.sincedb_ea87b331bdacc3b708b1438bb498d83f (time = 2021-10-13 09:17:26 -0500)
[2021-10-13T09:17:26,922][TRACE][filewatch.sincedbcollection] non_atomic_write:  {:time=>2021-10-13 09:17:26 -0500}
[2021-10-13T09:17:27,879][DEBUG][logstash.outputs.file    ] Starting flush cycle
[2021-10-13T09:17:29,879][DEBUG][logstash.outputs.file    ] Starting flush cycle
[2021-10-13T09:17:30,460][DEBUG][org.logstash.execution.PeriodicFlush] Pushing flush onto pipeline

That make me believe something goes wrong in pipelines settings.

As well, pipelines setting are exactly the same on working/non-working servers:

- pipeline.id: readme-v11_pipeline
  path.config: "/etc/logstash/conf.d/readme-v11.conf"
  pipeline.workers: 8
  pipeline.batch.size: 1024
  queue.type: persisted
  pipeline.ecs_compatibility: v1

We also noticed :
sincedb files are empty ( despite no right issues or file management problems)
completed
running conf file in command line is fine , this is only in service and conf.d context

Before going deeper in explanation , I just wanted to know if this continious flush is normal or not and mandatory pipeline and logstash service configuration check to perfom to point out this issue.

[Badger]

@fabeau

[DEBUG][org.logstash.execution.PeriodicFlush]

As it says, this is a periodic flush. I think it happens every five seconds. This is normal.

[fabeau]

Agree, but this is not explaining the issue whereas running 7.15 make not working the configuration file ( or the pipelines , I am not really sure).

I did not found so much insights in logs :

[2021-10-13T14:05:17,775][DEBUG][logstash.config.source.multilocal] Reading pipeline configurations from YAML {:location=>"/etc/logstash/pipelines.yml"}
[2021-10-13T14:05:17,779][DEBUG][logstash.config.source.local.configpathloader] Skipping the following files while reading config since they don't match the specified glob pattern {:files=>[]}
[2021-10-13T14:05:17,779][DEBUG][logstash.config.source.local.configpathloader] Reading config file {:config_file=>"/etc/logstash/conf.d/readme-v11.conf"}
[2021-10-13T14:05:17,779][DEBUG][org.logstash.config.ir.PipelineConfig] -------- Logstash Config ---------
[2021-10-13T14:05:17,779][DEBUG][org.logstash.config.ir.PipelineConfig] Config from source, source: LogStash::Config::Source::MultiLocal, pipeline_id:: readme-v11_pipeline
[2021-10-13T14:05:17,779][DEBUG][org.logstash.config.ir.PipelineConfig] Config string, protocol: file, id: /etc/logstash/conf.d/readme-v11.conf
[2021-10-13T14:05:17,779][DEBUG][org.logstash.config.ir.PipelineConfig]

## Logstash config for logs collected by MDT using kubernetes API/kubectl


input {
## for tests/tuning
#       http {
#       host => "10.86.81.204"
#       port => "8080"
##      codec => json{}
#       }

  file {
          path => ['/root/analyser/classic/v11/*/*/*.txt']
          mode => "read"
          start_position => "beginning"
          file_completed_action => "log"
          file_completed_log_path => '/tmp/completed_file-readme.txt'
          check_archive_validity => false
          file_sort_by => 'path'
          exclude => ["*.csv" , "*.tar" , "*.tar" ,"gz.node*"]
          codec => multiline { pattern => "^endofFile" negate => true what => previous auto_flush_interval => 1 }

  }
}
(continued without any error)....

(according to last answer, I commented auto_flush_internal without any impact).