# Logstash **URL:** https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=364 [Latest](https://discuss.elastic.co/latest.md) · [Categories](https://discuss.elastic.co/categories.md) · [Tags](https://discuss.elastic.co/tags.md) **Page:** 365 --- ## [Translate private ip to geo location using external dictionary](https://discuss.elastic.co/t/translate-private-ip-to-geo-location-using-external-dictionary/230710)
**Author:** [@ptamba](https://discuss.elastic.co/u/ptamba)\ **Replies:** 5\ **Last updated:** [May 2, 2020, 10:20pm UTC](https://discuss.elastic.co/t/translate-private-ip-to-geo-location-using-external-dictionary/230710 "2020-05-02T22:20:54Z")
Hi all, I'm trying to follow example in this link https://discuss.elastic.co/t/private-ip-geoip-from-dictionary/156763, however I can't seem to make it work. Basically I want to translate private ip address using exter… --- ## [Http\_poller input with basic authentification](https://discuss.elastic.co/t/http-poller-input-with-basic-authentification/230811)
**Author:** [@hiba](https://discuss.elastic.co/u/hiba)\ **Replies:** 1\ **Last updated:** [May 2, 2020, 5:48pm UTC](https://discuss.elastic.co/t/http-poller-input-with-basic-authentification/230811 "2020-05-02T17:48:59Z")
Hi , i can't get data from this api: this is my logstash file.conf: input { http\_poller { urls =\> { url =\> "XXXXXXXX" auth =\> { user =\> "bbc" password =\> "bbc" } } request\_timeout =\> 60 schedule =\> { every =\>… --- ## [Latitude/Longitude values - zero's get removed](https://discuss.elastic.co/t/latitude-longitude-values-zeros-get-removed/230789)
**Author:** [@Sketchy](https://discuss.elastic.co/u/Sketchy)\ **Replies:** 1\ **Last updated:** [May 2, 2020, 12:42pm UTC](https://discuss.elastic.co/t/latitude-longitude-values-zeros-get-removed/230789 "2020-05-02T12:42:49Z")
I am recieving json that contains Latitude and Longitude, in the message they look like this: "Latitude": -37.000000, "Longitude": 145.000000 In the output they look like this: "Latitude" =\> -37.0, "Longitude" =\> 145.… --- ## [How calculate time différence between two rows in csv file?](https://discuss.elastic.co/t/how-calculate-time-difference-between-two-rows-in-csv-file/230192)
**Author:** [@Emna1](https://discuss.elastic.co/u/Emna1)\ **Replies:** 5\ **Last updated:** [April 29, 2020, 2:14pm UTC](https://discuss.elastic.co/t/how-calculate-time-difference-between-two-rows-in-csv-file/230192 "2020-04-29T14:14:12Z")
Hi, Please i need to calculate différence of time between two rows in my file My csv file it looks like this ent\_sort , Date , car enter, 2020-04-28 05:04:02 , c1 … --- ## [Does a Logstash pipeline created in Kibana automatically start in Elastic cloud?](https://discuss.elastic.co/t/does-a-logstash-pipeline-created-in-kibana-automatically-start-in-elastic-cloud/230750)
**Author:** [@cristianplaytreks](https://discuss.elastic.co/u/cristianplaytreks)\ **Replies:** 3\ **Last updated:** [May 2, 2020, 11:10am UTC](https://discuss.elastic.co/t/does-a-logstash-pipeline-created-in-kibana-automatically-start-in-elastic-cloud/230750 "2020-05-02T11:10:08Z")
Hi! TL;DR How do I check the status of the Logstash pipeline in Elastic cloud? I created a new deployment in Elastic cloud and then I created a new Logstash pipeline. However, I'm not sure how to check any information … --- ## [Parsing artifactory logs using 7.6.2](https://discuss.elastic.co/t/parsing-artifactory-logs-using-7-6-2/230786)
**Author:** [@nsrushi](https://discuss.elastic.co/u/nsrushi)\ **Replies:** 0\ **Last updated:** [May 2, 2020, 2:02am UTC](https://discuss.elastic.co/t/parsing-artifactory-logs-using-7-6-2/230786 "2020-05-02T02:02:49Z")
Hi there I setup filebeat to send JFROG artifactory logs and it is not showing any logs i the filebeat Kibana. here is my filebeat configuraion input\_type: log Paths that should be crawled and fetched. Glob based … --- ## [Grokparsefailure with nginx logs](https://discuss.elastic.co/t/grokparsefailure-with-nginx-logs/230753)
**Author:** [@cloudbadmin](https://discuss.elastic.co/u/cloudbadmin)\ **Replies:** 1\ **Last updated:** [May 2, 2020, 12:40am UTC](https://discuss.elastic.co/t/grokparsefailure-with-nginx-logs/230753 "2020-05-02T00:40:35Z")
we have a custom nginx log that looks like this 172.16.23.132 - \[172.16.23.132\] - - \[30/Apr/2020:20:49:22 +0000\] \\"GET /health HTTP/2.0\\" 200 7 \\"-\\" \\"curl/7.58.0\\" 52 0.013 \[foobar-apiservice-443\] 172.24.12.90:9001 7 … --- ## [How to write mutate for nested field as array in config file?](https://discuss.elastic.co/t/how-to-write-mutate-for-nested-field-as-array-in-config-file/230655)
**Author:** [@apremgeorge](https://discuss.elastic.co/u/apremgeorge)\ **Replies:** 1\ **Last updated:** [May 1, 2020, 1:26pm UTC](https://discuss.elastic.co/t/how-to-write-mutate-for-nested-field-as-array-in-config-file/230655 "2020-05-01T13:26:16Z")
I have written the logstash config file to upload a csv, filter { csv { separator =\> "," skip\_header =\> true columns =\> \[LoanID,Applicant\_Income1,Occupation1,Time\_At\_Work1,Date\_Of\_Join1,Gender,LoanAmount,Marital\_… --- ## [Logstash parse JSON into individual events](https://discuss.elastic.co/t/logstash-parse-json-into-individual-events/230661)
**Author:** [@hiba](https://discuss.elastic.co/u/hiba)\ **Replies:** 2\ **Last updated:** [May 1, 2020, 1:15pm UTC](https://discuss.elastic.co/t/logstash-parse-json-into-individual-events/230661 "2020-05-01T13:15:07Z")
Hi, Im trying an HTTP Poller which returns me the response in the below format: {"term":"hamma","suggestions":{"cities":\[{"id":101509,"name":"Hammaliya","type":"city","region":"Sri Lanka","country":"Sri Lanka"},{"id":8… --- ## [Elasticseach as input in logstash? - From the beginning](https://discuss.elastic.co/t/elasticseach-as-input-in-logstash-from-the-beginning/230671)
**Author:** [@Rysiu](https://discuss.elastic.co/u/Rysiu)\ **Replies:** 1\ **Last updated:** [May 1, 2020, 12:27pm UTC](https://discuss.elastic.co/t/elasticseach-as-input-in-logstash-from-the-beginning/230671 "2020-05-01T12:27:01Z")
Hi, I have a big problem. Is there any way to configure Logstash input (elasticsearch) so that after rebooting Logstash does not load the whole elasticsearch index (input) from the beginning? So that it will load the … --- ## [Remove fields with empty value](https://discuss.elastic.co/t/remove-fields-with-empty-value/230470)
**Author:** [@vikramaddagulla](https://discuss.elastic.co/u/vikramaddagulla)\ **Replies:** 5\ **Last updated:** [May 1, 2020, 11:41am UTC](https://discuss.elastic.co/t/remove-fields-with-empty-value/230470 "2020-05-01T11:41:09Z")
Hello, I am receiving the data in logstash and I can see that at times, some of the fields do not have any values. Please see below for : flowid compositedetails causedby I would like to remove these fields if th… --- ## [Logstash Combine two fields of different documents based on another field (without an ID)](https://discuss.elastic.co/t/logstash-combine-two-fields-of-different-documents-based-on-another-field-without-an-id/230461)
**Author:** [@Nani\_20](https://discuss.elastic.co/u/Nani_20)\ **Replies:** 2\ **Last updated:** [May 1, 2020, 5:35am UTC](https://discuss.elastic.co/t/logstash-combine-two-fields-of-different-documents-based-on-another-field-without-an-id/230461 "2020-05-01T05:35:12Z")
I am using the latest version of logstash(7.6.2). I am trying to add two fields based on values in an another field. Find below my sample data with fields TimeStamp, Value and Event. Also, find below my desired output. S… --- ## [Stream Data from Elastic Search to CSV](https://discuss.elastic.co/t/stream-data-from-elastic-search-to-csv/230664)
**Author:** [@rsuper\_6616](https://discuss.elastic.co/u/rsuper_6616)\ **Replies:** 0\ **Last updated:** [May 1, 2020, 5:01am UTC](https://discuss.elastic.co/t/stream-data-from-elastic-search-to-csv/230664 "2020-05-01T05:01:47Z")
I am trying to find a way to stream data from Elastic Search to csv using Logstash. --- ## [Getting \[esaggs\] \> "field" is a required parameter when pipelining with logstash](https://discuss.elastic.co/t/getting-esaggs-field-is-a-required-parameter-when-pipelining-with-logstash/230652)
**Author:** [@tbhaxor](https://discuss.elastic.co/u/tbhaxor)\ **Replies:** 0\ **Last updated:** [May 1, 2020, 1:04am UTC](https://discuss.elastic.co/t/getting-esaggs-field-is-a-required-parameter-when-pipelining-with-logstash/230652 "2020-05-01T01:04:52Z")
Output without plugin, with logstash root@ubuntu-s-2vcpu-4gb-blr1-01:~# curl localhost:9200/\_cat/indices yellow open filebeat-2020.04.30 qKQlYmodS4uEEjkPIA4r4w 1 1 8868 0 4.3mb 4.3mb green open .kibana\_task\_m… --- ## [Secure the access to logstash APi Endpoints](https://discuss.elastic.co/t/secure-the-access-to-logstash-api-endpoints/230648)
**Author:** [@tlejmi](https://discuss.elastic.co/u/tlejmi)\ **Replies:** 0\ **Last updated:** [May 1, 2020, 12:05am UTC](https://discuss.elastic.co/t/secure-the-access-to-logstash-api-endpoints/230648 "2020-05-01T00:05:15Z")
I want to secure the access to logstash APi Endpoints 0.0.0.0: 9600/\_node Logstash : "version": "7.6.2" . I followed : https://www.elastic.co/guide/en/logstash/current/ls-security.html but it still without authenti… --- ## [How to create time variable in input plugin](https://discuss.elastic.co/t/how-to-create-time-variable-in-input-plugin/230647)
**Author:** [@alxsss](https://discuss.elastic.co/u/alxsss)\ **Replies:** 1\ **Last updated:** [May 1, 2020, 12:36am UTC](https://discuss.elastic.co/t/how-to-create-time-variable-in-input-plugin/230647 "2020-05-01T00:36:37Z")
Hello, I use http\_poller plugin as an input plugin. I need to pass current hour and previous hour as parameter values to its url. I wondered how this could be done, since did not find any documentation on this. --- ## [How to increase pipeline workers when running logstash on kubernetes](https://discuss.elastic.co/t/how-to-increase-pipeline-workers-when-running-logstash-on-kubernetes/230643)
**Author:** [@user2416](https://discuss.elastic.co/u/user2416)\ **Replies:** 0\ **Last updated:** [April 30, 2020, 10:03pm UTC](https://discuss.elastic.co/t/how-to-increase-pipeline-workers-when-running-logstash-on-kubernetes/230643 "2020-04-30T22:03:46Z")
Hi, we are running logstash on kubernetes using stable helm chart. How can i increase pipeline.workers ? is it possible to set in values.yaml. --- ## [Can Logstash manage Beats ILM and Templates?](https://discuss.elastic.co/t/can-logstash-manage-beats-ilm-and-templates/230447)
**Author:** [@jerrac](https://discuss.elastic.co/u/jerrac)\ **Replies:** 1\ **Last updated:** [April 30, 2020, 9:34pm UTC](https://discuss.elastic.co/t/can-logstash-manage-beats-ilm-and-templates/230447 "2020-04-30T21:34:59Z")
For various reasons I can't have my Beats talking directly to Elasticsearch on all the networks I will be monitoring. So I need to use Logstash as a proxy. Beats will ship to Logstash, Logstash will ship to Elasticsearch… --- ## [Can i add field with the contents of current year?](https://discuss.elastic.co/t/can-i-add-field-with-the-contents-of-current-year/230439)
**Author:** [@mtudisco](https://discuss.elastic.co/u/mtudisco)\ **Replies:** 3\ **Last updated:** [April 30, 2020, 7:22pm UTC](https://discuss.elastic.co/t/can-i-add-field-with-the-contents-of-current-year/230439 "2020-04-30T19:22:23Z")
Hi, I've seen the notation in the output part of logstash to put %{+YYYY.MM.dd} to name the index, for instance: output { elasticsearch { hosts =\> \["host1:9200"\] index =\> "prefix-%{+YYYY.MM.dd}" user =\> e… --- ## [Best input/output for logstash on aws](https://discuss.elastic.co/t/best-input-output-for-logstash-on-aws/230601)
**Author:** [@thenewguy](https://discuss.elastic.co/u/thenewguy)\ **Replies:** 0\ **Last updated:** [April 30, 2020, 5:25pm UTC](https://discuss.elastic.co/t/best-input-output-for-logstash-on-aws/230601 "2020-04-30T17:25:11Z")
I am running an app on elastic beanstalk. I deploy it with a logstash. I would like to have logstash put the logs somewhere that they can be retrieved safely without going missing. There are several aws options. Shoul… --- ## [Aggregate filter works in Logstash 6.8 but not in 7.4](https://discuss.elastic.co/t/aggregate-filter-works-in-logstash-6-8-but-not-in-7-4/230531)
**Author:** [@Criss](https://discuss.elastic.co/u/Criss)\ **Replies:** 1\ **Last updated:** [April 30, 2020, 3:10pm UTC](https://discuss.elastic.co/t/aggregate-filter-works-in-logstash-6-8-but-not-in-7-4/230531 "2020-04-30T15:10:06Z")
Hello Team, I have a pipeline with aggregate filter ( with "start event" and "end event"). This pipeline works perfect in logstash version 6.8 and the output contains a map with my values Output in Logstash 6.8: "Che… --- ## [Logstash event- Failed to Parse field](https://discuss.elastic.co/t/logstash-event-failed-to-parse-field/230351)
**Author:** [@Miguel\_Alvarez](https://discuss.elastic.co/u/Miguel_Alvarez)\ **Replies:** 5\ **Last updated:** [April 30, 2020, 3:05pm UTC](https://discuss.elastic.co/t/logstash-event-failed-to-parse-field/230351 "2020-04-30T15:05:16Z")
Hi. I have this error in my logstash: \[2020-04-28T14:58:13,744\]\[WARN \]\[logstash.outputs.elasticsearch\] Could not index event to Elasticsearch. {:status=\>400, :action=\>\["index", {:\_id=\>nil, :\_index=\>"nonfood\_pro-2020.04… --- ## [Failed to execute action error](https://discuss.elastic.co/t/failed-to-execute-action-error/230467)
**Author:** [@vinayavvaru](https://discuss.elastic.co/u/vinayavvaru)\ **Replies:** 1\ **Last updated:** [April 30, 2020, 3:00pm UTC](https://discuss.elastic.co/t/failed-to-execute-action-error/230467 "2020-04-30T15:00:49Z")
Hi, After running conf file from command prompt we are getting below error, please help us Error Details : C:\\Users\\VAVVARU\\Desktop\\ELK\_Softwares\\logstash-7.6.2\\bin\>logstash -f pipeline.conf Sending Logstash logs to … --- ## [Using Logstash for deduplication of Elasticsearch documents](https://discuss.elastic.co/t/using-logstash-for-deduplication-of-elasticsearch-documents/230548)
**Author:** [@Youssef\_SBAI](https://discuss.elastic.co/u/Youssef_SBAI)\ **Replies:** 2\ **Last updated:** [April 30, 2020, 2:32pm UTC](https://discuss.elastic.co/t/using-logstash-for-deduplication-of-elasticsearch-documents/230548 "2020-04-30T14:32:17Z")
input { elasticsearch { "http://localhost:9200" index =\> "stocks" query =\> '{ "sort": \[ "\_doc" \] }' } } filter { fingerprint { key =\> "1234ABCD" method =\> "SHA256" source =\> \[… --- ## [Multiple logstash nodes gathering log by http\_poller prevent duplicates](https://discuss.elastic.co/t/multiple-logstash-nodes-gathering-log-by-http-poller-prevent-duplicates/230568)
**Author:** [@mfielt](https://discuss.elastic.co/u/mfielt)\ **Replies:** 1\ **Last updated:** [April 30, 2020, 2:26pm UTC](https://discuss.elastic.co/t/multiple-logstash-nodes-gathering-log-by-http-poller-prevent-duplicates/230568 "2020-04-30T14:26:37Z")
Hi, We are using the ELK stack for a couple of years now and are on version 7.6.2 For the first time we are using the http\_poller plugin. Each logstash has the same configuration and therefor the nodes are getting the … --- ## [Issue time message when shown not correctly in elastic](https://discuss.elastic.co/t/issue-time-message-when-shown-not-correctly-in-elastic/230569)
**Author:** [@seeds17](https://discuss.elastic.co/u/seeds17)\ **Replies:** 1\ **Last updated:** [April 30, 2020, 2:21pm UTC](https://discuss.elastic.co/t/issue-time-message-when-shown-not-correctly-in-elastic/230569 "2020-04-30T14:21:05Z")
I'm new one. I had some issue about time message. when i got log from firewall. time logged 's shown correctly. but in the raw message , the time changed by plus +7 in timezone without doing anything. as shown in pict… --- ## [Twitter Input Errors](https://discuss.elastic.co/t/twitter-input-errors/230115)
**Author:** [@xennn](https://discuss.elastic.co/u/xennn)\ **Replies:** 1\ **Last updated:** [April 30, 2020, 2:01pm UTC](https://discuss.elastic.co/t/twitter-input-errors/230115 "2020-04-30T14:01:48Z")
Hello, i have a problem with my twitter Input. I try to add a proxy but no success. Is there a way to debug this? I get the following error: \[2020-04-28T11:03:41,962\]\[WARN \]\[org.logstash.execution.ShutdownWatcherExt\] … --- ## [Parsing measurement xml data using logstash](https://discuss.elastic.co/t/parsing-measurement-xml-data-using-logstash/226897)
**Author:** [@yagoza](https://discuss.elastic.co/u/yagoza)\ **Replies:** 3\ **Last updated:** [April 30, 2020, 1:40pm UTC](https://discuss.elastic.co/t/parsing-measurement-xml-data-using-logstash/226897 "2020-04-30T13:40:41Z")
\ \ \ \ **Author:** [@cpiment](https://discuss.elastic.co/u/cpiment)\ **Replies:** 3\ **Last updated:** [April 30, 2020, 1:27pm UTC](https://discuss.elastic.co/t/monitoring-logstash-with-metricbeat/230336 "2020-04-30T13:27:01Z") Hi, I'm configuring the monitoring feature of the Elastic Stack using metricbeat 7.6.2. I have configured a metricbeat with this configuration: - module: logstash metricsets: - node - node\_sta… --- ## [Issue with logstah and caracter encoding](https://discuss.elastic.co/t/issue-with-logstah-and-caracter-encoding/230533)
**Author:** [@mick](https://discuss.elastic.co/u/mick)\ **Replies:** 0\ **Last updated:** [April 30, 2020, 11:54am UTC](https://discuss.elastic.co/t/issue-with-logstah-and-caracter-encoding/230533 "2020-04-30T11:54:49Z")
Hello, I have a logstash input: input { syslog { host =\> "127.0.0.1" port =\> 10515 type =\> "nats" } } it's comming from a custom app. If I tcpdump the udp traffic on that port, here is what I get in pay… [Previous page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=363) [Next page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=365)