# Custom Fields Value From Path Element **URL:** \ **Category:** Beats\ **Tags:** filebeat\ **Created:** [January 30, 2017, 4:30am UTC](https://discuss.elastic.co/t/custom-fields-value-from-path-element/73153 "2017-01-30T04:30:47Z")\ **Posts on this page:** 5\ **Page:** 1
**Author:** ![gariels](https://avatars.discourse-cdn.com/v4/letter/g/a9adbd/32.png) [@gariels](https://discuss.elastic.co/u/gariels)\ **Post date:** [January 30, 2017, 4:30am UTC](https://discuss.elastic.co/t/custom-fields-value-from-path-element/73153/1 "2017-01-30T04:30:47Z")
Let's say I have this prospector definitions: ``` filebeat.prospectors: # log for `app-a` - input_type: log paths: - /progs/app-a/*/tomcat/*/logs/app-a.log - /progs/app-a/*/tomcat/*/logs/catalina.out tags: ["tomcat"] fields: source_program: app-a multiline.pattern: "^[[:digit:]]+|^$" multiline.negate: true multiline.match: after # log for `app-b` - input_type: log paths: - /progs/app-b/*/tomcat/*/logs/app-b.log - /progs/app-b/*/tomcat/*/logs/catalina.out tags: ["tomcat"] fields: source_program: app-b multiline.pattern: "^[[:digit:]]+|^$" multiline.negate: true multiline.match: after ``` As you can see I basically add a _source\_program_ field with the name of the application. Our applications deployment is automated and the deployment path follows this convention: ``` /progs/{APPLICATION_NAME}/{DEPLOYMENT_VERSION}/tomcat/{TOMCAT_VERSION}/ ``` My question is, could I use some sort automated value for fields using elements of path, like named-group in regular expression? For example could I do something like this? ``` - input_type: log paths: - /progs/app-b/*/tomcat/*/logs/app-b.log - /progs/app-b/*/tomcat/*/logs/catalina.out tags: ["tomcat"] fields: source_program: `path.element[1]` deploy_version: `path.element[2]` tomcat_version: `path.element[4]` multiline.pattern: "^[[:digit:]]+|^$" multiline.negate: true multiline.match: after ``` Is it possible to achieve this using only Filebeat and Elastisearch? ie. without using logstash. Thank you. ---
**Author:** ![steffens](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/steffens/32/79630_2.png) [@steffens](https://discuss.elastic.co/u/steffens)\ **Post date:** [January 30, 2017, 11:57am UTC](https://discuss.elastic.co/t/custom-fields-value-from-path-element/73153/2 "2017-01-30T11:57:41Z")
have you had a look at elasticsearch ingest node? ---
**Author:** ![gariels](https://avatars.discourse-cdn.com/v4/letter/g/a9adbd/32.png) [@gariels](https://discuss.elastic.co/u/gariels)\ **Post date:** [January 31, 2017, 3:28am UTC](https://discuss.elastic.co/t/custom-fields-value-from-path-element/73153/3 "2017-01-31T03:28:35Z")
Thank you for your reply. I'm not familiar with Elasticsearch Ingest Node, so far from browsing the documentation it seems what I need is the Append Processor. I am not familiar with how to split field value by path element separator, and then get the individual parts. It seems I also have to do a little bit logic to exclude other data from this treatment, we have other logs which does not follow the path convention (which is why use tags like "webserver", "tomcat", etc.), and I'm not sure how to do that with this "pipeline" thing. A small example would be nice, thank you. Anyway, if it is too convoluted we might have to bite the bullet and run logstash (something that I want to avoid). ---
**Author:** ![steffens](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/steffens/32/79630_2.png) [@steffens](https://discuss.elastic.co/u/steffens)\ **Post date:** [February 1, 2017, 12:44pm UTC](https://discuss.elastic.co/t/custom-fields-value-from-path-element/73153/4 "2017-02-01T12:44:12Z")
you can make the pipeline optional. See [pipeline](https://www.elastic.co/guide/en/beats/filebeat/current/elasticsearch-output.html#_pipeline) and [pipelines](https://www.elastic.co/guide/en/beats/filebeat/current/elasticsearch-output.html#_pipelines). If no `pipeline` is empty, no pipeline will be selected. e.g. ```auto filebeat.prospectors: - paths: ["/progs/app-b/*/tomcat/*/logs/app-b.log"] fields.pipeline: "analyze_source" .. - paths: ["/progs/app-b/tomcat/non-standard/logs/app-b.log"] ... output.elasticsearch: ... pipeline: '%{[fields.pipeline]}' ``` With this configuration, all events from first prospector will be send to the `analyze_source` pipeline and all events from second prospector will not be send to any ingest pipeline. in elasticsearch you can try the grok filter. It's basically regular expressions on steroids, supporting 'templates', extracing fields, and converting strings to (e.g. numeric) types. The grok filter even supports multiple patterns, in case you have different schemas. Ingest node also has some 'failure' handling in case of content being unparseable by grok. You can use this one as well and always send all events to the pipeline. I recommend to test in the kibana console using the [simulate API](https://www.elastic.co/guide/en/elasticsearch/reference/current/simulate-pipeline-api.html). If you really need something custom, you can use the [script](https://www.elastic.co/guide/en/elasticsearch/reference/current/script-processor.html) processor with painless([[1](https://www.elastic.co/blog/painless-a-new-scripting-language)], [[2](https://www.elastic.co/guide/en/elasticsearch/reference/current/modules-scripting-painless.html)], [[3](https://www.elastic.co/guide/en/elasticsearch/reference/current/modules-scripting-painless-debugging.html)]). I don't see how Append processor is what you need. If `grok` is not doing the trick, you can try the `split` processor + `script` to assign the individual fields (on the other hand I think painless let's you use some JAVA API, that is you can use [split](https://docs.oracle.com/javase/7/docs/api/java/lang/String.html#split(java.lang.String))) . ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [March 1, 2017, 12:44pm UTC](https://discuss.elastic.co/t/custom-fields-value-from-path-element/73153/5 "2017-03-01T12:44:41Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.