Module processor being cached?

[danimaiochi (Daniel Maiochi)]

Hi!
I'm developing a new module for filebeat that I'm calling ruby.
I noticed that my pipeline keeps using the same processor pattern it was configured on the first version.
this is the first version of my module:

"processors": [
{
  "grok": {
    "field": "message",
    "patterns": [
      "%{NUMBER:ruby.requests.timestamp}"
    ]
  }
}
]

so since I was using NUMBER for the timestamp I got the timestamp field = "02" - OK, THAT'S EXPECTED! I'M HUMAN

after noticing that I changed for:

"processors": [
{
  "grok": {
    "field": "message",
    "patterns": [
      "%{TIMESTAMP_ISO8601:ruby.requests.timestamp}"
    ]
  }
}
]

but I still got just the last number.
I added, changed and removed fields but keep getting the same result!

later I changed my modules/ruby folder to modules/ruby2 (and the filebeat.yml to use ruby2) and voila! it worked!

That's why I think there's some sort of cache being held and is not loading my patter everytime.
can you guys help me with that?
there's some sort of config to disable or a way to clear this "cache" thing?

Thanks in advance!
(Yeah, I restarted both services after changing the processor, even rebooted the computer)
(using windows )

[kvch (Noémi Ványi)]

By default templates are not updated automatically. It must be set in setup.template.overwrite option to true. Or you can delete it manually by using Kibana's Dev tools or CURL (see: https://www.elastic.co/guide/en/elasticsearch/reference/current/indices-templates.html#delete).

[danimaiochi (Daniel Maiochi)]

@kvch

setup.template.overwrite

the problem isn't with the template (btw, I'm using setup.template.overwrite already)
the problem is that Filebeat keeps using the old grok pattern.

if I have the:

"patterns": [
  "%{TIMESTAMP_ISO8601:ruby.requests.timestamp}"
]

my ruby.requests.timestamp will be like 2017-09-07 14:29:30

and if I change the grok to

"patterns": [
  "lalalalalala"
]

instead of getting an error, I'll get the same result

[kvch (Noémi Ványi)]

Then you pipeline is not updated. Pipelines can be deleted manually using the Dev tools of Kibana using DELETE _ingest/pipeline/your-pipeline-id. See "more": https://www.elastic.co/guide/en/elasticsearch/reference/current/delete-pipeline-api.html

Also, developing grok expressions could be easier using Simulate Pipeline API. More info on it: https://www.elastic.co/guide/en/elasticsearch/reference/5.5/simulate-pipeline-api.html

[danimaiochi (Daniel Maiochi)]

yeap!
exactly what I was looking for, I deleted the pipelines and it worked

there's a way to always replace the pipeline?

Thank you so much Noémi!