# Array support for YML environment values **URL:** \ **Category:** Beats\ **Tags:** filebeat\ **Created:** [February 27, 2017, 4:38pm UTC](https://discuss.elastic.co/t/array-support-for-yml-environment-values/76657 "2017-02-27T16:38:27Z")\ **Posts on this page:** 7\ **Page:** 1
**Author:** ![PicoCreator](https://avatars.discourse-cdn.com/v4/letter/p/a6a055/32.png) [@PicoCreator](https://discuss.elastic.co/u/PicoCreator)\ **Post date:** [February 27, 2017, 4:38pm UTC](https://discuss.elastic.co/t/array-support-for-yml-environment-values/76657/1 "2017-02-27T16:38:27Z")
For example, the environment value of ```auto LOG_EXCLUDE_LINES='["^A", "^B"]' ``` With yml configuration line ```auto exclude_lines: "${LOG_EXCLUDE_LINES}" ``` Is currently not possible, as it will be replaced with quotes. Though, there probably need to be a very specific new syntax for such a scenario. ---
**Author:** ![steffens](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/steffens/32/79630_2.png) [@steffens](https://discuss.elastic.co/u/steffens)\ **Post date:** [February 28, 2017, 12:02am UTC](https://discuss.elastic.co/t/array-support-for-yml-environment-values/76657/2 "2017-02-28T00:02:52Z")
filebeat version? these should work: if you set `exclude_lines: ${LOG_EXCLUDE_LINES}` (without the double quotes) then you can configure the array in your environment variable using a 'superset' of json. e.g. ```auto LOG_EXCLUDE_LINES='^A, ^B' ``` when inserting environment variables, an input value is constructed from environment variable plus surrounding string and parsed afterwards. By using quotes in your configuration file you did force the setting to be a string value, but no array. configuration file format, plus environment variable usage is [documented here](https://www.elastic.co/guide/en/beats/libbeat/current/config-file-format-env-vars.html). Unfortunately the docs is missing the syntax support for 'advanced' objects via environment variables or command line (`-E`). [Best documentation I can find is in code](https://github.com/elastic/go-ucfg/blob/master/internal/parse/parse.go#L23). You can basically pass json via environment variables (or CLI) with small "enhancements": - strings can be unquoted, single-quoted or double-quoted (being lax on quotation makes it easier to handle quotation requirements on shell) - arrays at top-level do not require `[]`, just use `,` to separate the elements. Note: do not use double-quotes `"` for regular expressions, as `\` will be interpreted as escape character. New github issue: [https://github.com/elastic/beats/issues/3686](https://github.com/elastic/beats/issues/3686) ---
**Author:** ![PicoCreator](https://avatars.discourse-cdn.com/v4/letter/p/a6a055/32.png) [@PicoCreator](https://discuss.elastic.co/u/PicoCreator)\ **Post date:** [February 28, 2017, 2:39am UTC](https://discuss.elastic.co/t/array-support-for-yml-environment-values/76657/3 "2017-02-28T02:39:01Z")
Currently deploying: 5.2.1 But makes sense, i was getting the errors from the "[]" array brackets, modifying directly from previous examples. This should work. Cheers! ---
**Author:** ![PicoCreator](https://avatars.discourse-cdn.com/v4/letter/p/a6a055/32.png) [@PicoCreator](https://discuss.elastic.co/u/PicoCreator)\ **Post date:** [March 1, 2017, 3:07am UTC](https://discuss.elastic.co/t/array-support-for-yml-environment-values/76657/4 "2017-03-01T03:07:23Z")
Slightly off topic. Does FileBeat optimises the array of exclude regex queries to evaluate as a single pass? or as individual passes for each query. Currently in my docker FileBeat container [https://hub.docker.com/r/picoded/docker-filebeat/](https://hub.docker.com/r/picoded/docker-filebeat/) I worked around the array problem, by doing the following. ```auto exclude_lines: [".*(([0-9]{2}\[(KNL|IKE|MGR|NET)\])|(POST /_bulk HTTP/1.1\" 200 [0-9]* \"-\" \"Go-http-client/1.1)|(level\=debug.*io\.rancher)).*"] ``` Breaking the regex down, so its more readable. Gives this. ```auto .*( ([0-9]{2}\[(KNL|IKE|MGR|NET)\])| (POST /_bulk HTTP/1.1\" 200 [0-9]* \"-\" \"Go-http-client/1.1)| (level\=debug.*io\.rancher) ).* ``` Originally I was hoping to set it up as an array of the following. But faced the YML array setup issue through docker environment variables. ```auto [ ".*[0-9]{2}\[(KNL|IKE|MGR|NET)\].*", ".*(POST /_bulk HTTP/1.1\" 200 [0-9]* \"-\" \"Go-http-client/1.1).*", ".*(level\=debug.*io\.rancher).*" ] ``` Subsequently, I actually tried both currently. But perhaps at my scale, or even maybe at any scale. I couldn't reliably measure any differences. But the question is back to, if each query in the array evaluated separately, or consolidated together and called in a single pass. I probably keep my regex query together for a single pass, if it isn't so. ---
**Author:** ![steffens](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/steffens/32/79630_2.png) [@steffens](https://discuss.elastic.co/u/steffens)\ **Post date:** [March 1, 2017, 5:47pm UTC](https://discuss.elastic.co/t/array-support-for-yml-environment-values/76657/5 "2017-03-01T17:47:42Z")
Well, we're talking micro-optimizations here. No idea how big an impact this will have with all filebeat machinery in place. Using `exclude_lines`, every pattern will be executed after another. By putting all patterns into one big regular expression, there might be some room for improvements. On the other hand, if you can simplify the patterns to be mostly strings, with 5.3 some other improvements might kick in replacing an `O(n*n)` algorithm with `O(n*log(n))`. This [PR](https://github.com/elastic/beats/pull/2433) introduces a string-matcher trying to optimize some common patterns (unfortunately your patterns doesn't fit into the 'easy, common' patterns supported yet). See [commit message](https://github.com/elastic/beats/pull/2433/commits/c683077a10f9a28c0e308057f7ea4b25ba81ee0b) for some details. Changing pattern 2 to `'POST /_bulk HTTP/1.1" 200 .* "-" "Go-http-client/1\.1'`, pattern 2 and 3 might be potential targets for additional matcher optimizations (not yet implemented, feel free to open [enhancement request](https://github.com/elastic/beats/issues)). That is, long-term using arrays might be the better option. Some notes on your regular expressions: 1. When string-matching, the regular expressions already search for a sub-string in the input stream. Do not use `.*` at the beginning or end of a regular expressions, as this increases the search-space required for back-tracking. The result will be the same. E.g. see this micro-benchmark result (searching for substring 'PATTERN') from mentioned PR: ```auto BenchmarkPatterns/Name=contains_'PATTERN'_with_'.*,_Matcher=Regex,_Content=mixed-4 50000 39312 ns/op BenchmarkPatterns/Name=contains_'PATTERN',_Matcher=Regex,_Content=mixed-4 1000000 1660 ns/op ``` 1. using `()` introduces a capture-group. as we do not want to capture any content via the regular expression use a non-capture-group `(?:)`. 2. the stdlib regex engine (well, go1.8 at least) tries to drive all 'NFA threads' of execution in parallel on stream of input. As neither of the regular expressions have a common prefix, I'd assume the both types, array style and using on big regexp via `|`, makes no real difference (same big O). Line filtering happens on already buffered (in memory) line + regex automaton needs to allocate some 'thread-state' (well, it's using a memory pool I think) per `|` sub-term. Optimizations (1) and (2) will be automatically available in upcoming 5.3 release. I have a tool to convert a regular expression (after parsing + default optimizations in stdlib) to graphviz for visual inspection: [https://github.com/urso/anareg](https://github.com/urso/anareg) Applying this to your regex: ```auto ./anareg '.*(([0-9]{2}\[(KNL|IKE|MGR|NET)\])|(POST /_bulk HTTP/1.1" 200 [0-9]* "-" "Go-http-client/1.1)|(level=debug.*io\.rancher)).*' | dot -Tpng | imgcat ``` I get: ![](https://us1.discourse-cdn.com/elastic/original/2X/1/17f46a578ab342c46b13e69869f5da83d8686c25.png) with some minor optimizations (automatically applied in 5.3 release) this becomes: ```auto ./anareg '(?:\d{2}\[(?:KNL|IKE|MGR|NET)\])|(?:POST /_bulk HTTP/1\.1" 200 [0-9]* "-" "Go-http-client/1\.1)|(?:level=debug.*io\.rancher)' | dot -Tpng | ./imgcat ``` ![](https://us1.discourse-cdn.com/elastic/original/2X/b/b5626a574ebe4ed5bc17b1872f6e055f081fa0d4.png) ---
**Author:** ![PicoCreator](https://avatars.discourse-cdn.com/v4/letter/p/a6a055/32.png) [@PicoCreator](https://discuss.elastic.co/u/PicoCreator)\ **Post date:** [March 3, 2017, 3:49pm UTC](https://discuss.elastic.co/t/array-support-for-yml-environment-values/76657/6 "2017-03-03T15:49:56Z")
Wow, i am actually really really glad automated optimisations for (1) and (2) is already occurring. As much as I would gladly use non capturing group, I worry that the next programmer after me, wouldn't understand it properly. This actually would make the code much easier to maintain. Additionally, i actually forgotten how heavily go was optimised for regex, that literally makes this micro optimisation. In that sense my experience with early day java regex libraries betray me. Because it was measurable differences the performance of a single huge regex, vs many smaller ones. Cheers 😂 ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [March 31, 2017, 3:50pm UTC](https://discuss.elastic.co/t/array-support-for-yml-environment-values/76657/7 "2017-03-31T15:50:01Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.