# How to get rid of nested field and other issues **URL:** \ **Category:** Logstash\ **Created:** [February 20, 2021, 9:59am UTC](https://discuss.elastic.co/t/how-to-get-rid-of-nested-field-and-other-issues/264945 "2021-02-20T09:59:46Z")\ **Posts on this page:** 8\ **Page:** 1
**Author:** ![jkost](https://avatars.discourse-cdn.com/v4/letter/j/e0b2c6/32.png) [@jkost](https://discuss.elastic.co/u/jkost)\ **Post date:** [February 20, 2021, 9:59am UTC](https://discuss.elastic.co/t/how-to-get-rid-of-nested-field-and-other-issues/264945/1 "2021-02-20T09:59:47Z")
Hallo, I 'm using logstash 7.11 and my data have the following format: ```auto { field1: ..., field2: ..., aList: [ { Id ... }, { fielda1 ... }, { fielda2 ... }, { fielda2 ... }, ... ] fieldN:..., fieldN1: ... } ``` I only need data inside `aList` so my pipeline is like so: ```auto input { file { path => "/usr/local/Cellar/logstash-full/7.11.0/data/data.json" codec => "json" sincedb_path => "NULL" } } filter { json { source => "message" } split { field => "[aList]" } if [aList][fielda1] { mutate { add_field => { "fielda1" => "[aList][fielda1]" } } } .... remove_field => ["path", "@version", "@timestamp", "host", "message", "field1", ..., "fieldN1"] rename => { "[aList][Id]" => "[@metadata][Id]" } ... } output { elasticsearch { hosts => ["http://localhost:9200"] index => "mydata" document_id => "%{[@metadata][Id]}" } } ``` 1. I 'm not sure if split filter is needed if I use input codec json. In any case, adding new fields at the root level keeps `aList` in the data. If I drop it, then it won't find any `aList`; I guess the code inside `filter` is not executed sequentially. In short, how do I get rid of `aList` once I have copied all its fields at the upper level? 2. While logstash was processing data since yesterday, today it seems to be stalled and denies to process my conf file. I have removed any `sincedb*` files, even though there shouldn't be any since I use `sincedb_path => "NULL"`. With `--debug` I see that it has compiled the filters but then it repeats this: ```auto [2021-02-20T11:20:59,870][DEBUG][org.logstash.execution.PeriodicFlush][main] Pushing flush onto pipeline. [2021-02-20T11:21:01,243][DEBUG][logstash.instrument.periodicpoller.jvm] collector name {:name=>"ParNew"} [2021-02-20T11:21:01,244][DEBUG][logstash.instrument.periodicpoller.jvm] collector name {:name=>"ConcurrentMarkSweep"} [2021-02-20T11:21:02,148][DEBUG][filewatch.sincedbcollection][main][dfd2caffb9cafaaef3de80e622b99fe8bd7b28ce8d7353878192e580cdaf3cdd] writing sincedb (delta since last write = 15) [2021-02-20T11:21:04,059][DEBUG][logstash.instrument.periodicpoller.cgroup] One or more required cgroup files or directories not found: /proc/self/cgroup, /sys/fs/cgroup/cpuacct, /sys/fs/cgroup/cpu [2021-02-20T11:57:41,083][DEBUG][logstash.instrument.periodicpoller.jvm] collector name {:name=>"ParNew"} [2021-02-20T11:57:41,085][DEBUG][logstash.instrument.periodicpoller.jvm] collector name {:name=>"ConcurrentMarkSweep"} [2021-02-20T11:57:41,596][DEBUG][org.logstash.execution.PeriodicFlush][main] Pushing flush onto pipeline. [2021-02-20T11:57:43,487][DEBUG][filewatch.sincedbcollection][main][038696e67175ced52e017c718a78d1c797905ce61b90eba968c0e2ce28cb4e63] writing sincedb (delta since last write = 15) [2021-02-20T11:57:45,750][DEBUG][logstash.instrument.periodicpoller.cgroup] One or more required cgroup files or directories not found: /proc/self/cgroup, /sys/fs/cgroup/cpuacct, /sys/fs/cgroup/cpu ``` Any ideas? Thank you in advance. ---
**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\ **Post date:** [February 20, 2021, 3:26pm UTC](https://discuss.elastic.co/t/how-to-get-rid-of-nested-field-and-other-issues/264945/2 "2021-02-20T15:26:39Z")
> [@jkost](#): > > `sincedb_path => "NULL"` If you do not want the in-memory sincedb persisted to disk across restarts then on Windows you would us `sincedb_path => "NUL"` (with one L). On UNIX you should use `sincedb_path => "/dev/null"`. ---
**Author:** ![jkost](https://avatars.discourse-cdn.com/v4/letter/j/e0b2c6/32.png) [@jkost](https://discuss.elastic.co/u/jkost)\ **Post date:** [February 20, 2021, 3:33pm UTC](https://discuss.elastic.co/t/how-to-get-rid-of-nested-field-and-other-issues/264945/3 "2021-02-20T15:33:33Z")
Thanks for your reply badger. If I use `/dev/null` in MacOSX I get this error: ```auto Error: Permission denied – Permission denied Exception: Errno::EACCES ``` ---
**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\ **Post date:** [February 20, 2021, 3:41pm UTC](https://discuss.elastic.co/t/how-to-get-rid-of-nested-field-and-other-issues/264945/4 "2021-02-20T15:41:36Z")
That makes me wonder if you are hitting [this](https://discuss.elastic.co/t/logstash-7-9-1-have-issues-in-arm/249596/7) bug. Do you get a stack trace? ---
**Author:** ![jkost](https://avatars.discourse-cdn.com/v4/letter/j/e0b2c6/32.png) [@jkost](https://discuss.elastic.co/u/jkost)\ **Post date:** [February 20, 2021, 3:44pm UTC](https://discuss.elastic.co/t/how-to-get-rid-of-nested-field-and-other-issues/264945/5 "2021-02-20T15:44:57Z")
Regarding the issue that it doesn't import the data to ES anymore, I have found a number of issues about ```auto One or more required cgroup files or directories not found: /proc/self/cgroup, /sys/fs/cgroup/cpuacct, /sys/fs/cgroup/cpu ``` and I don't know what is the solution. Some posts mention that it is not a problem, others that you need to modify `jvm.options` or `logstash.yml`. I don't know what is the final solution, definitely, these files don't exist in MacOSX. Even if I remove `sincedb_path => "NULL"` and use the default path, I still don't see any progress. However, things are worse, because it seems it is not related to ES. Even with this output: ```auto output { # elasticsearch { # hosts => ["http://localhost:9200"] # index => "flight-data" # document_id => "%{[@metadata][Id]}" # } file { path => "/usr/local/Cellar/logstash-full/7.11.0/data/output.json" } stdout { codec => rubydebug } } ``` I don't see any `output.json`. The strange thing is that I didn't change anything to my machine, and yesterday it was working!!! ---
**Author:** ![jkost](https://avatars.discourse-cdn.com/v4/letter/j/e0b2c6/32.png) [@jkost](https://discuss.elastic.co/u/jkost)\ **Post date:** [February 20, 2021, 3:52pm UTC](https://discuss.elastic.co/t/how-to-get-rid-of-nested-field-and-other-issues/264945/6 "2021-02-20T15:52:45Z")
```auto [2021-02-20T17:50:23,793][DEBUG][logstash.agent] Converging pipelines state {:actions_count=>0} [2021-02-20T17:50:24,412][DEBUG][logstash.instrument.periodicpoller.jvm] collector name {:name=>"ParNew"} [2021-02-20T17:50:24,414][DEBUG][logstash.instrument.periodicpoller.jvm] collector name {:name=>"ConcurrentMarkSweep"} [2021-02-20T17:50:25,551][DEBUG][org.logstash.execution.PeriodicFlush][main] Pushing flush onto pipeline. [2021-02-20T17:50:25,597][DEBUG][logstash.outputs.file][main] Starting flush cycle [2021-02-20T17:50:26,786][DEBUG][logstash.config.source.local.configpathloader] Skipping the following files while reading config since they don't match the specified glob pattern {:files=>["/usr/local/Cellar/logstash-full/7.11.0/config/data.conf"]} [2021-02-20T17:50:26,787][DEBUG][logstash.config.source.local.configpathloader] Reading config file {:config_file=>"/usr/local/Cellar/logstash-full/7.11.0/config/flight.conf"} [2021-02-20T17:50:26,792][DEBUG][logstash.agent] Converging pipelines state {:actions_count=>0} [2021-02-20T17:50:27,394][DEBUG][logstash.instrument.periodicpoller.cgroup] One or more required cgroup files or directories not found: /proc/self/cgroup, /sys/fs/cgroup/cpuacct, /sys/fs/cgroup/cpu [2021-02-20T17:50:27,602][DEBUG][logstash.outputs.file][main] Starting flush cycle ``` Why does the log say: ```auto Skipping the following files while reading config since they don't match the specified glob pattern {:files=>["/usr/local/Cellar/logstash-full/7.11.0/config/data.conf"]} [2021-02-20T17:58:05,257][DEBUG][logstash.agent] Converging pipelines state {:actions_count=>0} [2021-02-20T17:58:05,580][DEBUG][org.logstash.execution.PeriodicFlush][main] Pushing flush onto pipeline. [2021-02-20T17:58:05,671][DEBUG][logstash.outputs.file][main] Starting flush cycle ``` ---
**Author:** ![jkost](https://avatars.discourse-cdn.com/v4/letter/j/e0b2c6/32.png) [@jkost](https://discuss.elastic.co/u/jkost)\ **Post date:** [February 20, 2021, 6:17pm UTC](https://discuss.elastic.co/t/how-to-get-rid-of-nested-field-and-other-issues/264945/7 "2021-02-20T18:17:25Z")
Well, let me kinda try to answer my questions: 1. `sincedb_path => "NULL"` isn't the same as `/dev/null` in MacOSX as I thought; it simply creates a file named `NULL` in Logstash folder; it is the same as let the default path. This must have been the reason for the stall. So, unless the `Permission denied` is resolved in Mac, there is no other way than remove the `.sincedb` file each time, and `--config.reload.automatic` won't work. 2. You can get rid of `aList` simply by adding in the end of the `filter` plugin: ```auto mutate { remove_field => ["aList"] } ``` ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [March 20, 2021, 6:18pm UTC](https://discuss.elastic.co/t/how-to-get-rid-of-nested-field-and-other-issues/264945/8 "2021-03-20T18:18:25Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.