# Logstash not sending data to Elasticsearch **URL:** \ **Category:** Logstash\ **Tags:** docker\ **Created:** [December 20, 2023, 3:10pm UTC](https://discuss.elastic.co/t/logstash-not-sending-data-to-elasticsearch/349736 "2023-12-20T15:10:42Z")\ **Posts on this page:** 7\ **Page:** 1
**Author:** ![gtartjr](https://avatars.discourse-cdn.com/v4/letter/g/e9a140/32.png) [@gtartjr](https://discuss.elastic.co/u/gtartjr)\ **Post date:** [December 20, 2023, 3:10pm UTC](https://discuss.elastic.co/t/logstash-not-sending-data-to-elasticsearch/349736/1 "2023-12-20T15:10:42Z")
I am unable to get Logstash to read data and send to Elasticsearch index. My Elastcistac is 8.11.2, under a Docker for Windows platform. I have 2 jsonl formatted files that I need to index into Elasticsearch by a unique key. My logstash.conf is: ```auto file { path => "/usr/share/logstash/companies.jsonl" mode => "read" start_position => "beginning" sincedb_path => "/dev/null" codec => json_lines file_chunk_size => 524288 } } output { elasticsearch { hosts => "${ELASTIC_HOSTS}" user => "${ELASTIC_USER}" password => "${ELASTIC_PASSWORD}" ssl_certificate_authorities => ["/usr/share/logstash/certs/ca/ca.crt"] index => "my-elasticsearch-index" document_id => "%{field-from-my-input-file}" action => "update" doc_as_upsert => true id => "Bulk-Filings-Index-Upload" } } ``` Logstash initializes with no errors and I can see the input data in the container. However, there is no indication of any attempt to send data to Elasticsearch. No messages appear in the Logstash log anfter the message: ```auto 2023-12-20 08:42:11 [2023-12-20T13:42:11,425][INFO][logstash.agent] Pipelines running {:count=>1, :running_pipelines=>[:main], :non_running_pipelines=>[]} ``` No relevant messages appear in the Elasticsearch log. I have turned on debug and can see my data in the log which appears properly formatted and the record key is as expected. Adding `stdout { codec => rubydebug }` to my logstash output produces no output. This is a development environment, thus the "/dev/null" for the sincedb\_path. What am I missing? ---
**Author:** ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)\ **Post date:** [December 20, 2023, 3:48pm UTC](https://discuss.elastic.co/t/logstash-not-sending-data-to-elasticsearch/349736/2 "2023-12-20T15:48:24Z")
Hi @gtartjr welcome to the community > [@gtartjr](#): > > `codec => json_lines` What does a few sample lines of the source look like? Pretty sure you need to take that out that's made for a streaming case, not reading from a file From the [docs](https://www.elastic.co/guide/en/logstash/current/plugins-codecs-json_lines.html) > This codec will decode streamed JSON that is newline delimited. Encoding will emit a single JSON string ending in a @delimiter NOTE: Do not use this codec if your source input is line-oriented JSON, for example, redis or file inputs. Rather, use the json codec. More info: This codec is expecting to receive a stream (string) of newline terminated lines. The file input will produce a line string without a newline. Therefore this codec cannot work with line oriented inputs. ---
**Author:** ![gtartjr](https://avatars.discourse-cdn.com/v4/letter/g/e9a140/32.png) [@gtartjr](https://discuss.elastic.co/u/gtartjr)\ **Post date:** [December 20, 2023, 4:20pm UTC](https://discuss.elastic.co/t/logstash-not-sending-data-to-elasticsearch/349736/3 "2023-12-20T16:20:05Z")
This is a sample record, but you're right, I failed to see that part of the doc and I see why this may not work as expected. Each record does terminate with a " \n ", new line (ASCII 10). ```auto {"cik": "0000876684", "entityType": "other", "sic": "", "sicDescription": "", "insiderTransactionForOwnerExists": 0, "insiderTransactionForIssuerExists": 0, "name": "ALLIED FINANCIAL CORP II", "tickers": [], "exchanges": [], "ein": "521689359", "description": "", "website": "", "investorWebsite": "", "category": "", "fiscalYearEnd": null, "stateOfIncorporation": "", "stateOfIncorporationDescription": "", "addresses": {"mailing": {"street1": "1919 PENNSYLVANIA AVE", "street2": null, "city": "WASHINGTON", "stateOrCountry": "DC", "zipCode": "20006", "stateOrCountryDescription": "DC"}, "business": {"street1": "1919 PENNSYLVANIA AVE", "street2": null, "city": "WASHINGTON", "stateOrCountry": "DC", "zipCode": "20006", "stateOrCountryDescription": "DC"}}, "phone": "2023311112", "flags": "", "formerNames": []} ``` ---
**Author:** ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)\ **Post date:** [December 20, 2023, 4:25pm UTC](https://discuss.elastic.co/t/logstash-not-sending-data-to-elasticsearch/349736/4 "2023-12-20T16:25:11Z")
So that is precisely not the case for the for `json_lines` Try `codec => "json"` ---
**Author:** ![gtartjr](https://avatars.discourse-cdn.com/v4/letter/g/e9a140/32.png) [@gtartjr](https://discuss.elastic.co/u/gtartjr)\ **Post date:** [December 26, 2023, 3:40pm UTC](https://discuss.elastic.co/t/logstash-not-sending-data-to-elasticsearch/349736/5 "2023-12-26T15:40:57Z")
You were absolutely right. Solved my problem immediately in this case. Is there some nuance about how ````auto ``` sincedb_path => "/dev/null" ```` ```auto works? After completing my first test, I changed the sincedb_path to point to my custom sincedb_path, then back to "/dev/null" again. Now I'm not getting data to read from either configuration, even with new data introduced, ``` ---
**Author:** ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)\ **Post date:** [December 26, 2023, 4:06pm UTC](https://discuss.elastic.co/t/logstash-not-sending-data-to-elasticsearch/349736/6 "2023-12-26T16:06:06Z")
Two things come to mind one. You should probably read in detail. How since\_db works > **[File input plugin | Logstash Reference \[8.11\] | Elastic](https://www.elastic.co/guide/en/logstash/current/plugins-inputs-file.html#plugins-inputs-file-sincedb_path)** Second, I see you're creating your own document ID. Tou would probably want to look closely at that. We often see folks somehow Just overriding the same documents over and over again... Unless you have a very specific reason to use your own document ID that generated document, I use her very foolproof. So make sure you know what you're doing there ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [January 23, 2024, 4:06pm UTC](https://discuss.elastic.co/t/logstash-not-sending-data-to-elasticsearch/349736/7 "2024-01-23T16:06:19Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.