# Logstash **URL:** https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=556 [Latest](https://discuss.elastic.co/latest.md) · [Categories](https://discuss.elastic.co/categories.md) · [Tags](https://discuss.elastic.co/tags.md) **Page:** 557 --- ## [Parse two log files into Single index in ES via Logstash](https://discuss.elastic.co/t/parse-two-log-files-into-single-index-in-es-via-logstash/179077)
**Author:** [@sjabiulla](https://discuss.elastic.co/u/sjabiulla)\ **Replies:** 1\ **Last updated:** [April 30, 2019, 2:24pm UTC](https://discuss.elastic.co/t/parse-two-log-files-into-single-index-in-es-via-logstash/179077 "2019-04-30T14:24:10Z")
Hi, I have two log files and I want to merge the data from two log files and index the data into the same document. Log file 1 format: \ \ \ \ Log file 2 format: … --- ## [Logstash http output to Elasticsearch don\`t work if both http and https applied](https://discuss.elastic.co/t/logstash-http-output-to-elasticsearch-don-t-work-if-both-http-and-https-applied/179088)
**Author:** [@Hawchik\_Pupkur](https://discuss.elastic.co/u/Hawchik_Pupkur)\ **Replies:** 8\ **Last updated:** [April 30, 2019, 2:13pm UTC](https://discuss.elastic.co/t/logstash-http-output-to-elasticsearch-don-t-work-if-both-http-and-https-applied/179088 "2019-04-30T14:13:15Z")
input { beats { port =\> 5044 } } output { elasticsearch { hosts =\> \["https://localhost:9200","https://1.2.3.4:9200","https://2.3.4.5:9200"\] ssl\_certificate\_verification =\> "false" index =\> "dev-%{\[beat.version\]}… --- ## [Logstash kafka input problem(Unable to create Kafka consumer from given configuration )](https://discuss.elastic.co/t/logstash-kafka-input-problem-unable-to-create-kafka-consumer-from-given-configuration/178373)
**Author:** [@ahmetkumas1](https://discuss.elastic.co/u/ahmetkumas1)\ **Replies:** 1\ **Last updated:** [April 30, 2019, 11:38am UTC](https://discuss.elastic.co/t/logstash-kafka-input-problem-unable-to-create-kafka-consumer-from-given-configuration/178373 "2019-04-30T11:38:03Z")
Hello everyone, I'm having trouble with logstash when I want to use logstash to log kafka topics here is the full output. \[2019-04-25T09:21:18,836\]\[WARN \]\[org.apache.kafka.clients.ClientUtils\] Couldn't resolve server l… --- ## [Problem with crashing UDP Sender in Logstash](https://discuss.elastic.co/t/problem-with-crashing-udp-sender-in-logstash/179032)
**Author:** [@pipari](https://discuss.elastic.co/u/pipari)\ **Replies:** 3\ **Last updated:** [April 30, 2019, 8:10am UTC](https://discuss.elastic.co/t/problem-with-crashing-udp-sender-in-logstash/179032 "2019-04-30T08:10:42Z")
Hi, We use logstash to send UDP data through a secure interface that only allows communication in one direction. It worked very well intil I introduced a little more load to it. Running ELK stack 6.6.0 with filebeat an… --- ## [Need some examples for Aggregate filter Concept](https://discuss.elastic.co/t/need-some-examples-for-aggregate-filter-concept/179020)
**Author:** [@Smack\_Alpha](https://discuss.elastic.co/u/Smack_Alpha)\ **Replies:** 0\ **Last updated:** [April 30, 2019, 6:22am UTC](https://discuss.elastic.co/t/need-some-examples-for-aggregate-filter-concept/179020 "2019-04-30T06:22:36Z")
I want to know the concept of aggregate filter. I read the guide of aggregate filter in elastic website. I came to know that it merges the similar events with one. For example: if i have a data like this, { "country\_… --- ## [\[LOGSTASH\] crashes with pipeline exception (5.6.10)](https://discuss.elastic.co/t/logstash-crashes-with-pipeline-exception-5-6-10/179018)
**Author:** [@Karthik\_Mahadevan](https://discuss.elastic.co/u/Karthik_Mahadevan)\ **Replies:** 0\ **Last updated:** [April 30, 2019, 6:09am UTC](https://discuss.elastic.co/t/logstash-crashes-with-pipeline-exception-5-6-10/179018 "2019-04-30T06:09:06Z")
we see no issue when starting logstash, it was running for months without any issue. suddenly we see traceback provided below and logstash in state to reject incoming logs. traceback: '\[2019-04-24T22:20:18,855\]\[ERROR\]… --- ## [Webhdfs Exception: undefined method \`read\_uint32'](https://discuss.elastic.co/t/webhdfs-exception-undefined-method-read-uint32/179009)
**Author:** [@huagang](https://discuss.elastic.co/u/huagang)\ **Replies:** 1\ **Last updated:** [April 30, 2019, 3:42am UTC](https://discuss.elastic.co/t/webhdfs-exception-undefined-method-read-uint32/179009 "2019-04-30T03:42:58Z")
ENV： Ambari 2.7 HDP 3.1 Enabled Kerberos and AD It then installs gssapi for logstash according to the operation of the use kerberos webhdfs start error with exception "no such file to load -- gssapi" dengshaochun hos… --- ## [GEO Mapping and IPV6 addresses](https://discuss.elastic.co/t/geo-mapping-and-ipv6-addresses/174875)
**Author:** [@fhansen](https://discuss.elastic.co/u/fhansen)\ **Replies:** 2\ **Last updated:** [April 1, 2019, 9:30pm UTC](https://discuss.elastic.co/t/geo-mapping-and-ipv6-addresses/174875 "2019-04-01T21:30:51Z")
I'm struggling with getting IPV6 addresses to correctly show up in the GEO mapping. I've created the following configuration for Logstash: input { snmp { walk => \["1.3.6.1.2.1.31.1.1.1"\] hosts => \[{host => "… --- ## [Direct buffer memory error when creating a connection](https://discuss.elastic.co/t/direct-buffer-memory-error-when-creating-a-connection/178993)
**Author:** [@liling](https://discuss.elastic.co/u/liling)\ **Replies:** 0\ **Last updated:** [April 29, 2019, 9:25pm UTC](https://discuss.elastic.co/t/direct-buffer-memory-error-when-creating-a-connection/178993 "2019-04-29T21:25:08Z")
Hi, I'm using a filebeat -\> logstash -\> logstash -\> elasticsearch -\>kibana stack. Logstash is version 6.7.1 and the others are also latest 6.x versions. On the downstream logstash, I saw this warning message when it ke… --- ## [Logstash with Localstack](https://discuss.elastic.co/t/logstash-with-localstack/178992)
**Author:** [@xduh](https://discuss.elastic.co/u/xduh)\ **Replies:** 0\ **Last updated:** [April 29, 2019, 9:22pm UTC](https://discuss.elastic.co/t/logstash-with-localstack/178992 "2019-04-29T21:22:04Z")
Hi, I have a project with a pipeline who get input from a SQS and output to ES. In order to test the pipeline I was trying to start a localstack (https://github.com/localstack/localstack) container to provide sqs service… --- ## [Running logstash on elastic cloud](https://discuss.elastic.co/t/running-logstash-on-elastic-cloud/178899)
**Author:** [@IV\_jph](https://discuss.elastic.co/u/IV_jph)\ **Replies:** 2\ **Last updated:** [April 29, 2019, 8:18pm UTC](https://discuss.elastic.co/t/running-logstash-on-elastic-cloud/178899 "2019-04-29T20:18:01Z")
Hi I was wondering how to configure logstash to run on the elastic cloud solution? I can see through the documentation that it's possible to configure logstash to run against a cloud id, however I would like to run the … --- ## [Need a Logstash filter to query a rest api/tcp on conditional statement and return JSON response](https://discuss.elastic.co/t/need-a-logstash-filter-to-query-a-rest-api-tcp-on-conditional-statement-and-return-json-response/178819)
**Author:** [@stcdarrell](https://discuss.elastic.co/u/stcdarrell)\ **Replies:** 5\ **Last updated:** [April 29, 2019, 7:47pm UTC](https://discuss.elastic.co/t/need-a-logstash-filter-to-query-a-rest-api-tcp-on-conditional-statement-and-return-json-response/178819 "2019-04-29T19:47:13Z")
hi, i'd love some help, i've googled all i know to google, read the logstash documents.. and i'm still not finding what i need. i am bringing in a ton of logs from various sources dealing with network traffic. i'd like … --- ## [Invalid FieldReference](https://discuss.elastic.co/t/invalid-fieldreference/178721)
**Author:** [@Samvid\_Kulkarni](https://discuss.elastic.co/u/Samvid_Kulkarni)\ **Replies:** 2\ **Last updated:** [April 29, 2019, 7:14pm UTC](https://discuss.elastic.co/t/invalid-fieldreference/178721 "2019-04-29T19:14:17Z")
I have a field named AccountName which has values like this abc$, abc234$ (just two values in field). i am using mutate filter to split it into parts and then use add\_field to insert values into another field. I have lo… --- ## [Logstash and index recreation](https://discuss.elastic.co/t/logstash-and-index-recreation/178975)
**Author:** [@Elena\_Kochetkova](https://discuss.elastic.co/u/Elena_Kochetkova)\ **Replies:** 1\ **Last updated:** [April 29, 2019, 6:41pm UTC](https://discuss.elastic.co/t/logstash-and-index-recreation/178975 "2019-04-29T18:41:00Z")
We had elasticsearch upgraded elasticsearch cluster and accidentally pointed it to the old data directories and ever since we were getting tons of dangling indices log entries. We built a new elasticsearch cluster and my… --- ## [Logstash GROK pattern not accepted](https://discuss.elastic.co/t/logstash-grok-pattern-not-accepted/178924)
**Author:** [@weed101](https://discuss.elastic.co/u/weed101)\ **Replies:** 4\ **Last updated:** [April 29, 2019, 5:36pm UTC](https://discuss.elastic.co/t/logstash-grok-pattern-not-accepted/178924 "2019-04-29T17:36:48Z")
Hello, I'm having a problem with getting my grok pattern to work within my filter. I've created a test.conf file to simplify and narrow down my problem and I cannot seem to get this to work. I have tried the syntax an… --- ## [Change Logging Level of Filter](https://discuss.elastic.co/t/change-logging-level-of-filter/178964)
**Author:** [@wwalker](https://discuss.elastic.co/u/wwalker)\ **Replies:** 3\ **Last updated:** [April 29, 2019, 5:34pm UTC](https://discuss.elastic.co/t/change-logging-level-of-filter/178964 "2019-04-29T17:34:09Z")
I am using the DNS filter which logs a WARN message when it fails to resolve an IP to a hostname. My logstash logs are filling up with these events, which I don't care about. Currently, there are about 300k lines of re… --- ## [Search documents based on '\_version'](https://discuss.elastic.co/t/search-documents-based-on-version/178959)
**Author:** [@RaTheDev](https://discuss.elastic.co/u/RaTheDev)\ **Replies:** 0\ **Last updated:** [April 29, 2019, 3:28pm UTC](https://discuss.elastic.co/t/search-documents-based-on-version/178959 "2019-04-29T15:28:57Z")
Looking for querying on documents using '\_version' field. Out of my 20M records in Elastic Search and few records missed the latest updates. I realized those records have '\_version' value as 1, Can some help me to que… --- ## [Can't separated elasticsearch index using logstash](https://discuss.elastic.co/t/cant-separated-elasticsearch-index-using-logstash/178661)
**Author:** [@LockyZ](https://discuss.elastic.co/u/LockyZ)\ **Replies:** 2\ **Last updated:** [April 29, 2019, 3:13pm UTC](https://discuss.elastic.co/t/cant-separated-elasticsearch-index-using-logstash/178661 "2019-04-29T15:13:51Z")
Hello Guys, I'm trying to create separated indexes using logstash 6.7, basically I have a server running filebeat which sends two logs to a logstash which sends to a elasticsearch, below you may find my config files. f… --- ## [Twitter-input-plugin not installed by default in docker image 7.0.0](https://discuss.elastic.co/t/twitter-input-plugin-not-installed-by-default-in-docker-image-7-0-0/178947)
**Author:** [@Sebastian\_Treu](https://discuss.elastic.co/u/Sebastian_Treu)\ **Replies:** 0\ **Last updated:** [April 29, 2019, 1:52pm UTC](https://discuss.elastic.co/t/twitter-input-plugin-not-installed-by-default-in-docker-image-7-0-0/178947 "2019-04-29T13:52:51Z")
Hi, I found that twitter-input-plugin has been removed from default in major version 7. root@logstash1:~# docker run -it docker.elastic.co/logstash/logstash:7.0.0 ./bin/logstash-plugin list | grep twitter root@logstash1… --- ## [Calculate time difference between two dates ( start date and end date of an log file )](https://discuss.elastic.co/t/calculate-time-difference-between-two-dates-start-date-and-end-date-of-an-log-file/178916)
**Author:** [@Badem](https://discuss.elastic.co/u/Badem)\ **Replies:** 3\ **Last updated:** [April 29, 2019, 1:24pm UTC](https://discuss.elastic.co/t/calculate-time-difference-between-two-dates-start-date-and-end-date-of-an-log-file/178916 "2019-04-29T13:24:56Z")
Hello, everyone. I'm trying without success to calculate the difference between the start date and the end date of my log file in order to get the time duration ( in second ) of an script. Here is my input : 17/04/201… --- ## [Ignore this ,this topic is not valid any more](https://discuss.elastic.co/t/ignore-this-this-topic-is-not-valid-any-more/178829)
**Author:** [@cheriemilk](https://discuss.elastic.co/u/cheriemilk)\ **Replies:** 1\ **Last updated:** [April 29, 2019, 1:02pm UTC](https://discuss.elastic.co/t/ignore-this-this-topic-is-not-valid-any-more/178829 "2019-04-29T13:02:29Z")
Hi Team, Logstash configuraiton run as below input { file{ path =\> "C:/elkstack/elasticsearch-6.5.1/logs/userscenario.csv" start\_position =\> "beginning" sincedb\_path =\> "C:/elkstack/elasticsearch-6.5.1/sincedb/sinc… --- ## [QUESTION ABOUT "LOGSTASH GROK MATCH" WITH ARRAY](https://discuss.elastic.co/t/question-about-logstash-grok-match-with-array/178827)
**Author:** [@GGUERRA](https://discuss.elastic.co/u/GGUERRA)\ **Replies:** 3\ **Last updated:** [April 29, 2019, 12:35pm UTC](https://discuss.elastic.co/t/question-about-logstash-grok-match-with-array/178827 "2019-04-29T12:35:38Z")
Hello everyone, I have the following question regarding the logstash filters. I have logs that have fields that sometimes appear and sometimes do not, for example "ip" = "1.1.1.1", "mac" = "a1: a1: a1: a1: a1: a1", "by… --- ## [Update document without id](https://discuss.elastic.co/t/update-document-without-id/178654)
**Author:** [@dililili](https://discuss.elastic.co/u/dililili)\ **Replies:** 4\ **Last updated:** [April 29, 2019, 11:56am UTC](https://discuss.elastic.co/t/update-document-without-id/178654 "2019-04-29T11:56:17Z")
I'm pretty new to logstash. I have an index called person in Kibana. The index includes firstname, lastname, email, address... Now I need to update person's address by using a table in database. My table only have emails… --- ## [How to add date in email subject triggering through watcher](https://discuss.elastic.co/t/how-to-add-date-in-email-subject-triggering-through-watcher/178892)
**Author:** [@Mohanraj](https://discuss.elastic.co/u/Mohanraj)\ **Replies:** 0\ **Last updated:** [April 29, 2019, 10:01am UTC](https://discuss.elastic.co/t/how-to-add-date-in-email-subject-triggering-through-watcher/178892 "2019-04-29T10:01:20Z")
Hi, we are getting the count details and sending the details through email using teh watcher option. Do we have any option to send the date in the email subject ? For example, my email subject should be like this Subj… --- ## [Single pipeline to monitor multiple java applications (jmx)](https://discuss.elastic.co/t/single-pipeline-to-monitor-multiple-java-applications-jmx/178870)
**Author:** [@jasony](https://discuss.elastic.co/u/jasony)\ **Replies:** 0\ **Last updated:** [April 29, 2019, 8:53am UTC](https://discuss.elastic.co/t/single-pipeline-to-monitor-multiple-java-applications-jmx/178870 "2019-04-29T08:53:44Z")
hello, i am trying to use logstash with jmx input plugin, and considering below use case. if "A" host runs 3 java applications. how can i get jmx metrics from each java application? as far as i know, i need to run 3 p… --- ## [Grok filtering not happening](https://discuss.elastic.co/t/grok-filtering-not-happening/178863)
**Author:** [@ltom](https://discuss.elastic.co/u/ltom)\ **Replies:** 0\ **Last updated:** [April 29, 2019, 8:30am UTC](https://discuss.elastic.co/t/grok-filtering-not-happening/178863 "2019-04-29T08:30:12Z")
I have two docker containers running, docker.elastic.co/beats/filebeat:7.0.0 running Filebeat, and sebp/elk:latest running ELK stack. The goal is to have my nginx logs in Kibana. The Filebeat container is sending the ng… --- ## [GROK Pattern to split string to multiple values](https://discuss.elastic.co/t/grok-pattern-to-split-string-to-multiple-values/178636)
**Author:** [@Mohanraj](https://discuss.elastic.co/u/Mohanraj)\ **Replies:** 4\ **Last updated:** [April 29, 2019, 6:07am UTC](https://discuss.elastic.co/t/grok-pattern-to-split-string-to-multiple-values/178636 "2019-04-29T06:07:38Z")
I am getting a string ".." in the logs and would like to split this using grok pattern. is there any method to do so ? example: input log string : java-app.ABCD.win-service-test Output should be app\_name: java-app o… --- ## [Logstash exists with unexpected error](https://discuss.elastic.co/t/logstash-exists-with-unexpected-error/178695)
**Author:** [@yangjiajun2014](https://discuss.elastic.co/u/yangjiajun2014)\ **Replies:** 1\ **Last updated:** [April 29, 2019, 5:05am UTC](https://discuss.elastic.co/t/logstash-exists-with-unexpected-error/178695 "2019-04-29T05:05:02Z")
I use version 5.6.2 and 6.2.3 in my production env.My logstash exists frequently because of fllowing error: \[2019-03-05T08:49:19,668\]\[FATAL\]\[logstash.runner \] An unexpected error occurred! {:error=\>#\ **Author:** [@gitika88](https://discuss.elastic.co/u/gitika88)\ **Replies:** 1\ **Last updated:** [April 29, 2019, 3:01am UTC](https://discuss.elastic.co/t/logstash-permission-access-denied-error-for-stack-org-jruby-rubyfile-javain-utime/178016 "2019-04-29T03:01:15Z") Hi, I am new to Data Analytics and trying to do a POC for one of our clients using ELK. My Kibana and Elastic Search servers are working fine and using the command bin/logstash -r -f /Users/gitika.dua/Documents/Data\_Anal… --- ## [Logstasth 7 cannot create geoip.location](https://discuss.elastic.co/t/logstasth-7-cannot-create-geoip-location/178792)
**Author:** [@zozo6015](https://discuss.elastic.co/u/zozo6015)\ **Replies:** 0\ **Last updated:** [April 28, 2019, 9:49am UTC](https://discuss.elastic.co/t/logstasth-7-cannot-create-geoip-location/178792 "2019-04-28T09:49:56Z")
Hello, I am parsing the haproxy logs and I would like to create geoip mapping on all the request IP addresses. I have added the following to the logstash configuration: geoip { database =\> "/etc/logstash/GeoLite2-Ci… [Previous page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=555) [Next page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=557)