Logstash "elasticsearch" input doesn't start

Elastic Stack Logstash
1

Hi!
I'm trying to add new fields in my index documents which are calculated from data already present in the documents. For example, I have two timestamps and I have to calculate the difference in seconds.
I wrote a logstash configuration file that take data from elatsicserach index and put the output updating documents.
It seems that the routine doesn't start, even if I've specified the tag schedule or scroll (not both of them).
This is imy input:

input {
  elasticsearch {
    hosts => "my_host:9200"
    index => "my_index*"
    query => '{ "query": { "bool": { "filter": [ { "exists": { "field": "time_closed" } }, { "exists": { "field": "time_open" } }, { "match_phrase": { "calc_done": false } }, { "range": { "@timestamp": { "gt": "now-3d", "lt": "now" } } } ] } } }'
    tags => ["my_tag"]
#    schedule => "30 * * * *"
    user => "my_elastic_user"
    password => "my_elastic_password"
    scroll => "10m"
    size => 10000
    docinfo => true
  }
}

I tried the query in a curl request, it returns every documents that I expect.

This is my input:

filter {
  if "my_tag" in [tags] {
    .. some calculation
    mutate {
      replace => {
        "[calc_done]" => true
      }
    }
  }
}

This is my output:

output {
  if "my_tag" in [tags] {
    elasticsearch {
      hosts => [ "http://my_host:9200" ]
      user => "my_logstash_user"
      password => "my_logstash_password"
      index => "%{[@metadata][_index]}"
      action => "update"
      doc_as_upsert => true
      document_id => "%{[@metadata][_id]}"
    }
  }
}

I don't see any log for this routine in logstash and elasticserach log file...
In every documents the file "calc_done" is false and no calculations are done...

Thanks in advance
Giacomo

2

Does anyone have suggestions about my problem?

3

I did many tests...
I found an interest thng: if I run logstash configuration file from another instamce of logstash, it works, even with schedule tag in input section!
The command is
./bin/logstash -f <pth_of conf_file>

On Internet I found suggestions about file rights, so I checked them and there are no difference between this file and the others tha work correctly!

I tried to run the sceond instance of logstash with user logstash and it didn't work, but I don't know if it's a good test...
I tried also to change configuration file owner and group to logstash but I've not had any improvements...

4

I resolved my problem and I want to write the solution fo rpeople who have my problem.
There's no errors on logstash configuration file, it was a concept error.

In my first ingest of data I put the tag "my_first_tag" in every documents.
In this routine I put the tag "my_tag".

So documents have both "my_tag" and "my_first_tag" and they enter in two different sections "filter" and two different sections "output".

Another problem was that user logstash didn't have ownership of file /etc/default/logstash and that the directory /etc/logstash/ had group "root" instead "logstash".

5

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.