# Save search results as a file. Which is continually updated **URL:** \ **Category:** Logstash\ **Tags:** docker\ **Created:** [February 27, 2024, 5:21pm UTC](https://discuss.elastic.co/t/save-search-results-as-a-file-which-is-continually-updated/354261 "2024-02-27T17:21:09Z")\ **Posts on this page:** 10\ **Page:** 1
**Author:** ![sharbich](https://avatars.discourse-cdn.com/v4/letter/s/5daacb/32.png) [@sharbich](https://discuss.elastic.co/u/sharbich)\ **Post date:** [February 27, 2024, 5:21pm UTC](https://discuss.elastic.co/t/save-search-results-as-a-file-which-is-continually-updated/354261/1 "2024-02-27T17:21:09Z")
Hello, i capture logs from a Docker container according to the following pattern. ```auto routes: - multiline+logstash+tcp://logstash.intern.example.com:50000 env: - name: SYSLOG_HOSTNAME value: homeassistant - name: INACTIVITY_TIMEOUT value: 1m - name: MULTILINE_PATTERN value: >- (\d\d(\d\d)?[-/]\d\d[-/]\d\d[T ]\d\d:\d\d:\d\d)|(^s6-rc:)|(^\[\d\d:\d\d:\d\d\])|(\d\d:\d\d:\d\d\ -)|(^[TDIWEF]:) - name: MULTILINE_MATCH value: first - name: INCLUDE_CONTAINERS value: homeassistant - name: LOGSTASH_FIELDS value: source=hame01 ``` Here the logstash config: ```auto root@dsme01:~# cat /etc/logstash/conf.d/syslog/syslog_hame01.conf input { tcp { host => "logstash.intern.example.com" port => 50000 codec => json } } ## Add your filters / logstash plugins configuration here filter { if ([source] == "my-home-assistant") { if ([docker][name] == "/homeassistant") { grok { patterns_dir => ["/usr/share/logstash/pipeline/patterns"] match => { "message" => "%{LOGLEVEL:log_level}%{SPACE}\(%{GREEDYDATA:log_thread}\)%{SPACE}\[%{LOGGER_NAME:log_name}\]%{SPACE}%{GREEDYDATA:log_message}" } } mutate { gsub => ["log_message", "\x1B\[([0-9]{1,2}(;[0-9]{1,2})?)?[m|M|K]", "" ] } if [log_message] =~ /\n/ { mutate { copy => { "log_message" => "log_trace" } } mutate { gsub => ["log_message", "(?m)^([^\n]*)$.*", "\1" ] } } } else { drop { } } } } output { elasticsearch { hosts => "elasticsearch.intern.example.com:9200" user => "elastic" password => "elastic" ssl => true cacert => "/etc/logstash/config/certs/HarbichCA.crt" index => "syslog_hame01-%{+YYYY.MM.dd}" } } ``` I see the Information in Kibana. Can I also see the messages in a file on the ELK host in order to search for the contents with another program (fail2ban)? Greetings from Stefan Harbich ---
**Author:** ![Rios](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/rios/32/95745_2.png) [@Rios](https://discuss.elastic.co/u/Rios)\ **Post date:** [February 27, 2024, 8:22pm UTC](https://discuss.elastic.co/t/save-search-results-as-a-file-which-is-continually-updated/354261/2 "2024-02-27T20:22:36Z")
Well, it's easier to search in Kibana-Discovery 🙂 In order to see what LS will send to ES, you can add in the output: `file { path => "/path/syslog_%{+YYYY-MM-dd}.txt" " }` ---
**Author:** ![sharbich](https://avatars.discourse-cdn.com/v4/letter/s/5daacb/32.png) [@sharbich](https://discuss.elastic.co/u/sharbich)\ **Post date:** [February 27, 2024, 10:07pm UTC](https://discuss.elastic.co/t/save-search-results-as-a-file-which-is-continually-updated/354261/3 "2024-02-27T22:07:12Z")
Hello, Unfortunately that does not work. Here is my adjusted configuration: ```auto file { path => "/var/log/homeassistant/syslog_hame01.log" } ``` and the error in the log: ```auto [2024-02-27T23:00:40,970][ERROR][logstash.agent] Failed to execute action {:action=>LogStash::PipelineAction::Create/pipeline_id:hame01, :exception=>"LogStash::ConfigurationError", :message=>"Expected one of [\\t\\r\\n], \"#\", \"=>\" at line 42, column 10 (byte 1127) after output {\n elasticsearch {\n hosts => \"elasticsearch.intern.example.com:9200\"\n user => \"elastic\"\n password => \"elastic\"\n ssl => true\n cacert => \"/etc/logstash/config/certs/HarbichCA.crt\"\n index => \"syslog_hame01-%{+YYYY.MM.dd}\"\n file ", :backtrace=>["/usr/share/logstash/logstash-core/lib/logstash/compiler.rb:32:in `compile_imperative'", "org/logstash/execution/AbstractPipelineExt.java:239:in `initialize'", "org/logstash/execution/AbstractPipelineExt.java:173:in `initialize'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:48:in `initialize'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline_action/create.rb:49:in `execute'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:386:in `block in converge_state'"]} ``` Where is the mistake? Greetings from Stefan Harbich ---
**Author:** ![Rios](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/rios/32/95745_2.png) [@Rios](https://discuss.elastic.co/u/Rios)\ **Post date:** [February 27, 2024, 10:23pm UTC](https://discuss.elastic.co/t/save-search-results-as-a-file-which-is-continually-updated/354261/4 "2024-02-27T22:23:10Z")
at line 42, column 10 \<-- You have an issue here. Try with this: ```auto output { elasticsearch { hosts => "elasticsearch.intern.example.com:9200" user => "elastic" password => "pass" ssl => true cacert => "/etc/logstash/config/certs/HarbichCA.crt" index => "syslog_hame01-%{+YYYY.MM.dd}" } file { path => "/var/log/homeassistant/syslog_hame01.log" } } ``` ---
**Author:** ![sharbich](https://avatars.discourse-cdn.com/v4/letter/s/5daacb/32.png) [@sharbich](https://discuss.elastic.co/u/sharbich)\ **Post date:** [February 28, 2024, 10:08am UTC](https://discuss.elastic.co/t/save-search-results-as-a-file-which-is-continually-updated/354261/5 "2024-02-28T10:08:10Z")
> [@Rios](#): > > at line 42, column 10 \<-- You have an issue here. It just doesn't work. I checked the permissions. From my point of view everything is fine. I even set the permission to 777. The following users are authorized to write to the directory (elasticsearch, logstash or root). I created the file. Nothing helps. it just doesn't work. no log file is written. Do I have to install a plugin so that the file can be created? What else can I do? ---
**Author:** ![Rios](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/rios/32/95745_2.png) [@Rios](https://discuss.elastic.co/u/Rios)\ **Post date:** [February 28, 2024, 10:37am UTC](https://discuss.elastic.co/t/save-search-results-as-a-file-which-is-continually-updated/354261/6 "2024-02-28T10:37:50Z")
No, you don't have to install file plugin, it's already included in LS installation. Error: "line 42, column 10" means your .conf file is not OK, most likely brackets are no closed. Again, if you run LS as servicectl, your user is "logstash:logstash" and you should change the ownership. If you ran as a process from the command line, then that user needs write permissions. Basically, "file { path ..." will write the same data which goes to ES to local disk. ---
**Author:** ![sharbich](https://avatars.discourse-cdn.com/v4/letter/s/5daacb/32.png) [@sharbich](https://discuss.elastic.co/u/sharbich)\ **Post date:** [February 28, 2024, 11:48am UTC](https://discuss.elastic.co/t/save-search-results-as-a-file-which-is-continually-updated/354261/7 "2024-02-28T11:48:51Z")
> [@Rios](#): > > No, you don't have to install file plugin, it's already included in LS installation. > > Error: "line 42, column 10" means your .conf file is not OK, most likely brackets are no closed. > > Again, if you run LS as servicectl, your user is "logstash:logstash" and you should change the ownership. If you ran as a process from the command line, then that user needs write permissions. > > Basically, "file { path ..." will write the same data which goes to ES to local disk. I've tried everything. Permissions checked. Tried all variants. No file is created under "/var/log/homeassistant/". I just don't know why???? Here my config: ```auto root@dsme01:~# cat /etc/logstash/conf.d/syslog/syslog_hame01.conf input { tcp { host => "logstash.intern.example.com" port => 50000 codec => json } } ## Add your filters / logstash plugins configuration here filter { if ([source] == "my-home-assistant") { if ([docker][name] == "/homeassistant") { grok { patterns_dir => ["/usr/share/logstash/pipeline/patterns"] match => { "message" => "%{LOGLEVEL:log_level}%{SPACE}\(%{GREEDYDATA:log_thread}\)%{SPACE}\[%{LOGGER_NAME:log_name}\]%{SPACE}%{GREEDYDATA:log_message}" } } mutate { gsub => ["log_message", "\x1B\[([0-9]{1,2}(;[0-9]{1,2})?)?[m|M|K]", "" ] } if [log_message] =~ /\n/ { mutate { copy => { "log_message" => "log_trace" } } mutate { gsub => ["log_message", "(?m)^([^\n]*)$.*", "\1" ] } } } else { drop { } } } } output { elasticsearch { hosts => "elasticsearch.intern.example.com:9200" user => "elastic" password => "elastic" ssl => true cacert => "/etc/logstash/config/certs/HarbichCA.crt" index => "syslog_hame01-%{+YYYY.MM.dd}" } file { path => "/var/log/homeassistant/syslog_hame01.log" } } ``` ---
**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\ **Post date:** [February 28, 2024, 6:02pm UTC](https://discuss.elastic.co/t/save-search-results-as-a-file-which-is-continually-updated/354261/8 "2024-02-28T18:02:27Z")
What error message do you get after you changed the .conf file? ---
**Author:** ![sharbich](https://avatars.discourse-cdn.com/v4/letter/s/5daacb/32.png) [@sharbich](https://discuss.elastic.co/u/sharbich)\ **Post date:** [February 28, 2024, 8:16pm UTC](https://discuss.elastic.co/t/save-search-results-as-a-file-which-is-continually-updated/354261/9 "2024-02-28T20:16:13Z")
Hello, i found my error. My "logspout addon" has ended. Therefore no data was provided. Now everything works. Many thanks for the help. Greetings from Stefan Harbich ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [March 27, 2024, 8:16pm UTC](https://discuss.elastic.co/t/save-search-results-as-a-file-which-is-continually-updated/354261/10 "2024-03-27T20:16:51Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.