# Logstash **URL:** https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=344 [Latest](https://discuss.elastic.co/latest.md) · [Categories](https://discuss.elastic.co/categories.md) · [Tags](https://discuss.elastic.co/tags.md) **Page:** 345 --- ## [Logstash + Lumberjack error](https://discuss.elastic.co/t/logstash-lumberjack-error/236574)
**Author:** [@18visions](https://discuss.elastic.co/u/18visions)\ **Replies:** 0\ **Last updated:** [June 10, 2020, 7:49pm UTC](https://discuss.elastic.co/t/logstash-lumberjack-error/236574 "2020-06-10T19:49:50Z")
Hello. I've had some issues connecting to our host. I get this message from journalctl- Is there anything that sticks out in this error message? { :"hosts=\>"\[ "\", "\", "\" … --- ## [Logstash grok IF condition fields](https://discuss.elastic.co/t/logstash-grok-if-condition-fields/236561)
**Author:** [@himalc](https://discuss.elastic.co/u/himalc)\ **Replies:** 0\ **Last updated:** [June 10, 2020, 5:58pm UTC](https://discuss.elastic.co/t/logstash-grok-if-condition-fields/236561 "2020-06-10T17:58:35Z")
Hi, Please let me know, following if condition can run for any docker images. I want to filter postgres-1 docker image logs only. What are those values (\[docker\]\[container\]\[labels\] ) How can I create this type of filter… --- ## [Logstash email alerts](https://discuss.elastic.co/t/logstash-email-alerts/236416)
**Author:** [@hiba](https://discuss.elastic.co/u/hiba)\ **Replies:** 3\ **Last updated:** [June 10, 2020, 5:28pm UTC](https://discuss.elastic.co/t/logstash-email-alerts/236416 "2020-06-10T17:28:11Z")
Hi, I'm trying to configure logstash to send mail . But it seems doesn't work. my config file : input { http\_poller { urls =\> { url =\> "xxxl" } request\_timeout =\> 60 schedule =\> { every =\> "60s" } codec =\> "jso… --- ## [If condition in the logstash output section](https://discuss.elastic.co/t/if-condition-in-the-logstash-output-section/236456)
**Author:** [@dsv](https://discuss.elastic.co/u/dsv)\ **Replies:** 5\ **Last updated:** [June 10, 2020, 5:23pm UTC](https://discuss.elastic.co/t/if-condition-in-the-logstash-output-section/236456 "2020-06-10T17:23:51Z")
Hi Having the latest logstash-7.7.1-1.noarch from the official repo (baseurl=https://artifacts.elastic.co/packages/7.x/yum) In the output section the if "\_grokparsefailure" not in \[tags\] works fine. But any of the … --- ## [Logstash elasticsearch input reads same set of data everytime](https://discuss.elastic.co/t/logstash-elasticsearch-input-reads-same-set-of-data-everytime/236451)
**Author:** [@Ullas7](https://discuss.elastic.co/u/Ullas7)\ **Replies:** 3\ **Last updated:** [June 10, 2020, 4:12pm UTC](https://discuss.elastic.co/t/logstash-elasticsearch-input-reads-same-set-of-data-everytime/236451 "2020-06-10T16:12:22Z")
I am reading data from elasticsearch with below conf input { elasticsearch { hosts =\> "localhost:9200" index =\> "indexName" query =\> '{ "sort": \[ "\_doc" \] }' schedule =\> "/20 \* \* \* \* \*" doci… --- ## [Join two log files using aggregate filter of logstash to enter combined event in kibana](https://discuss.elastic.co/t/join-two-log-files-using-aggregate-filter-of-logstash-to-enter-combined-event-in-kibana/236536)
**Author:** [@Rushabh\_Doshi](https://discuss.elastic.co/u/Rushabh_Doshi)\ **Replies:** 0\ **Last updated:** [June 10, 2020, 3:34pm UTC](https://discuss.elastic.co/t/join-two-log-files-using-aggregate-filter-of-logstash-to-enter-combined-event-in-kibana/236536 "2020-06-10T15:34:53Z")
I have used the below aggregate plugin for logstash to join two log files: Log1: PAGE,PAGEDESCRIPTION Log 2: PAGE,HTTPAGENT I WANT THE FINAL OUTPUT in kibana as: PAGE,PAGEDESCRIPTION,HTTPAGENT. The aggregate plugin sni… --- ## [Poll: What Java version do you use with Logstash?](https://discuss.elastic.co/t/poll-what-java-version-do-you-use-with-logstash/236523)
**Author:** [@roaksoax](https://discuss.elastic.co/u/roaksoax)\ **Replies:** 0\ **Last updated:** [June 10, 2020, 2:25pm UTC](https://discuss.elastic.co/t/poll-what-java-version-do-you-use-with-logstash/236523 "2020-06-10T14:25:13Z")
Hi All, The Logstash team is looking at the future of our JDK support and we were wondering what JDK (whether that's OpenJDK, AdoptOpenJDK or Zulu) version you are using. poll --- ## [Cannot update plugin logstash-filter-grok](https://discuss.elastic.co/t/cannot-update-plugin-logstash-filter-grok/236388)
**Author:** [@nhnicwaller](https://discuss.elastic.co/u/nhnicwaller)\ **Replies:** 1\ **Last updated:** [June 10, 2020, 1:50pm UTC](https://discuss.elastic.co/t/cannot-update-plugin-logstash-filter-grok/236388 "2020-06-10T13:50:55Z")
I have this command as part of my process to build a docker image for Logstash. It worked fine a month ago, and today it is failing. logstash-plugin update logstash-filter-grok Here's the exact error message: An error… --- ## [Hyphen-minus with a new line](https://discuss.elastic.co/t/hyphen-minus-with-a-new-line/236477)
**Author:** [@Burga](https://discuss.elastic.co/u/Burga)\ **Replies:** 1\ **Last updated:** [June 10, 2020, 11:57am UTC](https://discuss.elastic.co/t/hyphen-minus-with-a-new-line/236477 "2020-06-10T11:57:02Z")
Hi , I'm trying to grok log with a multiple lines and Hyphen-minus and got stuck 2020-06-10 10:06:59,821 WARN \[ImageFilterEffect:thread-1\] \[confluence.image.effects.ImageFilterTask\] rotateWhenExifExist Could not retrie… --- ## [Logstash pipeline systax errors after upgrading from 7.1.1. to 7.7.0](https://discuss.elastic.co/t/logstash-pipeline-systax-errors-after-upgrading-from-7-1-1-to-7-7-0/236479)
**Author:** [@bhavesh.vasoya](https://discuss.elastic.co/u/bhavesh.vasoya)\ **Replies:** 6\ **Last updated:** [June 10, 2020, 11:40am UTC](https://discuss.elastic.co/t/logstash-pipeline-systax-errors-after-upgrading-from-7-1-1-to-7-7-0/236479 "2020-06-10T11:40:28Z")
\------------------- Some information ------------------- we have running ELK stack 7.1.1. logstash parser written in multiple files under "/etc/logstash/conf.d" directory Now i have upgrade ELK stack 7.1.1 to 7.7.0 and… --- ## [Multiline events based on identical fields](https://discuss.elastic.co/t/multiline-events-based-on-identical-fields/235564)
**Author:** [@WDK](https://discuss.elastic.co/u/WDK)\ **Replies:** 4\ **Last updated:** [June 10, 2020, 10:44am UTC](https://discuss.elastic.co/t/multiline-events-based-on-identical-fields/235564 "2020-06-10T10:44:55Z")
Hi all, I have a not so straight forward request of a user to process progress logfiles. The catch is that the want the log files multilined in a specific pattern. All lines that have the exact same timestamp, P-id and… --- ## [Logstash throws error INVALID\_FETCH\_SESSION\_EPOCH after upgrading from 6.5 to 6.8.8](https://discuss.elastic.co/t/logstash-throws-error-invalid-fetch-session-epoch-after-upgrading-from-6-5-to-6-8-8/236482)
**Author:** [@vijay\_kaali](https://discuss.elastic.co/u/vijay_kaali)\ **Replies:** 0\ **Last updated:** [June 10, 2020, 10:07am UTC](https://discuss.elastic.co/t/logstash-throws-error-invalid-fetch-session-epoch-after-upgrading-from-6-5-to-6-8-8/236482 "2020-06-10T10:07:49Z")
We get \[org.apache.kafka.clients.FetchSessionHandler\] \[Consumer clientId=logstash-0, groupId=logstash\] Node 0 was unable to process the fetch request with (sessionId=1844422694, epoch=11343): INVALID\_FETCH\_SESSION\_EPOCH … --- ## [Xml attribute parsing through Logstash + file beat](https://discuss.elastic.co/t/xml-attribute-parsing-through-logstash-file-beat/236435)
**Author:** [@Puneeth\_S\_B\_Gowda1](https://discuss.elastic.co/u/Puneeth_S_B_Gowda1)\ **Replies:** 1\ **Last updated:** [June 10, 2020, 9:34am UTC](https://discuss.elastic.co/t/xml-attribute-parsing-through-logstash-file-beat/236435 "2020-06-10T09:34:23Z")
This is my xml \M1S\ from above one i need M2S attribute value as new column in Kibana Can you share sample conf which will help for my above xml please My xml is from text file in… --- ## [Join two log files using aggregate filter of logstash](https://discuss.elastic.co/t/join-two-log-files-using-aggregate-filter-of-logstash/236468)
**Author:** [@Rushabh\_Doshi](https://discuss.elastic.co/u/Rushabh_Doshi)\ **Replies:** 0\ **Last updated:** [June 10, 2020, 9:06am UTC](https://discuss.elastic.co/t/join-two-log-files-using-aggregate-filter-of-logstash/236468 "2020-06-10T09:06:09Z")
I have used the below aggregate plugin for logstash to join two log files: Log1: PAGE,PAGEDESCRIPTION Log 2: PAGE,HTTPAGENT I WANT THE FINAL OUTPUT in kibana as: PAGE,PAGEDESCRIPTION,HTTPAGENT. The aggregate plugin sni… --- ## [Extract a subset of an big Index](https://discuss.elastic.co/t/extract-a-subset-of-an-big-index/236360)
**Author:** [@ginu](https://discuss.elastic.co/u/ginu)\ **Replies:** 4\ **Last updated:** [June 10, 2020, 8:24am UTC](https://discuss.elastic.co/t/extract-a-subset-of-an-big-index/236360 "2020-06-10T08:24:51Z")
Hi I have a bit index journalbeat-\*... we would like to have an extract of it which we would like to keep for months with only a few fields... what i the best way to achieve this? --- ## [Flooded /var/log/messages](https://discuss.elastic.co/t/flooded-var-log-messages/236442)
**Author:** [@zeleznypa](https://discuss.elastic.co/u/zeleznypa)\ **Replies:** 0\ **Last updated:** [June 10, 2020, 6:26am UTC](https://discuss.elastic.co/t/flooded-var-log-messages/236442 "2020-06-10T06:26:04Z")
On our servers, the /var/log/messages file is flooded with messages like this: Jun 10 08:16:46 SERVER logstash: 2020-06-10 08:16:42,551 Converge PipelineAction::Create\ WARN The Logger slowlog.logstas… --- ## [Too many if else if in output](https://discuss.elastic.co/t/too-many-if-else-if-in-output/236441)
**Author:** [@cling1988](https://discuss.elastic.co/u/cling1988)\ **Replies:** 0\ **Last updated:** [June 10, 2020, 6:21am UTC](https://discuss.elastic.co/t/too-many-if-else-if-in-output/236441 "2020-06-10T06:21:32Z")
Hi All I am currently using Logstash version 7.4 My Logstash output is to Kusto. And it has the if-else condition on the output, for the different condition will output to different Kusto database. The if-else checki… --- ## [HTTP poller SSL cert issue](https://discuss.elastic.co/t/http-poller-ssl-cert-issue/236064)
**Author:** [@jb789](https://discuss.elastic.co/u/jb789)\ **Replies:** 1\ **Last updated:** [June 9, 2020, 8:17pm UTC](https://discuss.elastic.co/t/http-poller-ssl-cert-issue/236064 "2020-06-09T20:17:42Z")
Could not get http poller to work in pulling a public API data input { http\_poller { urls =\> { urlname =\> "https://pricing.us-east-1.amazonaws.com/offers/v1.0/aws/AmazonRedshif… --- ## [Indexing multiple DBs](https://discuss.elastic.co/t/indexing-multiple-dbs/236380)
**Author:** [@squareEngine](https://discuss.elastic.co/u/squareEngine)\ **Replies:** 0\ **Last updated:** [June 9, 2020, 3:53pm UTC](https://discuss.elastic.co/t/indexing-multiple-dbs/236380 "2020-06-09T15:53:40Z")
Hi, We're planning to introduce Elasticsearch in our product, to provide better performance while searching data. Currently we have around 200 db instances with the same schema, containing data about tenants in our app. … --- ## [Outdated Documentation](https://discuss.elastic.co/t/outdated-documentation/236382)
**Author:** [@Kren42](https://discuss.elastic.co/u/Kren42)\ **Replies:** 0\ **Last updated:** [June 9, 2020, 3:59pm UTC](https://discuss.elastic.co/t/outdated-documentation/236382 "2020-06-09T15:59:00Z")
The documentation for the CEF codec located below is out of date. https://www.elastic.co/guide/en/logstash/current/plugins-codecs-cef.html There is at least one undocumented option: 'raw\_data\_field'. How does this go … --- ## [Logstash permissions](https://discuss.elastic.co/t/logstash-permissions/236355)
**Author:** [@PraveenKT](https://discuss.elastic.co/u/PraveenKT)\ **Replies:** 0\ **Last updated:** [June 9, 2020, 2:03pm UTC](https://discuss.elastic.co/t/logstash-permissions/236355 "2020-06-09T14:03:48Z")
What are the correct user group, rwx permissions should i give to "/etc/logstash/config.d/" directory. I am always getting permission errors. --- ## [Replace @timestamp to time from actual log file](https://discuss.elastic.co/t/replace-timestamp-to-time-from-actual-log-file/236330)
**Author:** [@Youssef\_SBAI](https://discuss.elastic.co/u/Youssef_SBAI)\ **Replies:** 11\ **Last updated:** [June 9, 2020, 2:02pm UTC](https://discuss.elastic.co/t/replace-timestamp-to-time-from-actual-log-file/236330 "2020-06-09T14:02:56Z")
Hello it is possible to Change @timestamp to time from actual log file I use this config filter { date { match =\> \["timestamp", "ISO8601"\] } } But it does not work --- ## [Filebeat is not able to connect to logstash](https://discuss.elastic.co/t/filebeat-is-not-able-to-connect-to-logstash/236066)
**Author:** [@NetRajan](https://discuss.elastic.co/u/NetRajan)\ **Replies:** 4\ **Last updated:** [June 9, 2020, 1:59pm UTC](https://discuss.elastic.co/t/filebeat-is-not-able-to-connect-to-logstash/236066 "2020-06-09T13:59:16Z")
I am using filebeat version 7.6.2 Trying to connect to logstash and getting following error. Failed to connect to backoff(async(tcp://dnslogstash.net:5044)): EOF logstash is configured with SSL certificate. I am able… --- ## [Logstash filter ruby](https://discuss.elastic.co/t/logstash-filter-ruby/235677)
**Author:** [@ajayraghuraj](https://discuss.elastic.co/u/ajayraghuraj)\ **Replies:** 9\ **Last updated:** [June 9, 2020, 1:05pm UTC](https://discuss.elastic.co/t/logstash-filter-ruby/235677 "2020-06-09T13:05:06Z")
Hi How do I add a new field which is a percentage calculation using ruby filter ? new field called 'MemValuePct' should contain the value of 'type\_instance' in percentage example MemValuePct \[ free / (free +cached … --- ## [Extracting JSON key and values to separated fields](https://discuss.elastic.co/t/extracting-json-key-and-values-to-separated-fields/236289)
**Author:** [@mihailo](https://discuss.elastic.co/u/mihailo)\ **Replies:** 1\ **Last updated:** [June 9, 2020, 12:13pm UTC](https://discuss.elastic.co/t/extracting-json-key-and-values-to-separated-fields/236289 "2020-06-09T12:13:51Z")
Hi, I need help with processing JSON file. My input looks like this (I need to keep it intact): input { s3 { bucket =\> "${S3\_BUCKET}" access\_key\_id =\> "${S3\_ACCESS\_KEY}" secret\_access\_key =\> "${S3\_S… --- ## [Date timestamp not matching](https://discuss.elastic.co/t/date-timestamp-not-matching/236277)
**Author:** [@calanon](https://discuss.elastic.co/u/calanon)\ **Replies:** 11\ **Last updated:** [June 9, 2020, 12:02pm UTC](https://discuss.elastic.co/t/date-timestamp-not-matching/236277 "2020-06-09T12:02:36Z")
I am having trouble trying to get this timestamp to match, any ideas? \[20200609 084347\] --- ## [Aggregate filter doesn't find last line of file](https://discuss.elastic.co/t/aggregate-filter-doesnt-find-last-line-of-file/235962)
**Author:** [@anoop](https://discuss.elastic.co/u/anoop)\ **Replies:** 2\ **Last updated:** [June 9, 2020, 12:00pm UTC](https://discuss.elastic.co/t/aggregate-filter-doesnt-find-last-line-of-file/235962 "2020-06-09T12:00:22Z")
Aggregate filter doesn't find last line of file I am trying to identify the last line of my log files using aggregate timeout. Then I am setting set it a "\_aggregatetimeout" tag. (I want to later use it in a 'elapsed' … --- ## [Handling MQRFH2 header in Logstash](https://discuss.elastic.co/t/handling-mqrfh2-header-in-logstash/236318)
**Author:** [@rohitarorait82](https://discuss.elastic.co/u/rohitarorait82)\ **Replies:** 0\ **Last updated:** [June 9, 2020, 11:11am UTC](https://discuss.elastic.co/t/handling-mqrfh2-header-in-logstash/236318 "2020-06-09T11:11:33Z")
Hi Guys, Can you please suggest , how to handle incoming MQRFH2 header in logstash, this is going to the dead letter queue, whenever there is an MQRFH2 header in request --- ## [How to check if then else whether nummeric between 2 signs of a string](https://discuss.elastic.co/t/how-to-check-if-then-else-whether-nummeric-between-2-signs-of-a-string/236310)
**Author:** [@Datakids](https://discuss.elastic.co/u/Datakids)\ **Replies:** 0\ **Last updated:** [June 9, 2020, 10:07am UTC](https://discuss.elastic.co/t/how-to-check-if-then-else-whether-nummeric-between-2-signs-of-a-string/236310 "2020-06-09T10:07:36Z")
Hi there, I have this in my logs: check code '' is relevant check code '589' is relevant What I want to do in logstash: if \[field\] !~ check code '/^\[0-9\]+$/' is relavant! { mutate { update =\> {"sta… --- ## [Logstash log error format](https://discuss.elastic.co/t/logstash-log-error-format/236300)
**Author:** [@MagnusSectra](https://discuss.elastic.co/u/MagnusSectra)\ **Replies:** 0\ **Last updated:** [June 9, 2020, 9:43am UTC](https://discuss.elastic.co/t/logstash-log-error-format/236300 "2020-06-09T09:43:47Z")
We have a lot of hosts adding data to our logstash. When errors appear it would be nice to somehow get "sender" information. Is that possible? Example: Say that suddenly a XML parsing error appears in the logstash-plai… [Previous page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=343) [Next page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=345)