Logstash: Input from SQS does not get inserted to InfluxDB

Elastic Stack Logstash
1

I am trying to insert data to InfluxDB using Logstash with input method SQS: I send message to Sqs and Logstash reads the messages from Sqs and tries to put data to InfluxDB.

I have set logstash like below, however, the attempt to insert data to influxDB keeps making an error saying [WARN ][logstash.outputs.influxdb] Non recoverable exception while writing to InfluxDB {:exception=>#<Errno::EPIPE: Broken pipe - No message available>} . As you can see, the output of stdout seems fine. What am I doing wrong?

Any advice or suggestion would be very appreciated.

Thank you in advance.

stdout value

{
      "userId" => 1,
         "lng" => 127,
    "@version" => "1",
  "@timestamp" => 2019-04-02T12:46:33.006Z,
         "lat" => 36
}

logstash.conf

input {
    sqs {
        codec => "json"
        queue => "logstash-test"        
    }
}

output {
    stdout {}
    influxdb {
        measurement => "location"
        send_as_tags => ["userId"]
        data_points => {
            "userId" => "%{[userId]}"
            "lat" => "%{[lat]]}"
            "lng" => "%{[lng]]}"
        }
        host => "localhost"
        port => "xxxx"
        db => "db_name"
        codec => json
    }
}

Used logstash plugins:
logstash-input-sqs
logstash-output-influxdb

2

In v 5.0.0 we switched to using the influx db ruby client instead of a general http client.

There is a report of the same error you are seeing in the pre 5.0.0 version but nothing since.

That said, the current release of the logstash-output-influxdb is constrained to use version 0.3.X of the influxdb client. Since then the infuxdb folks have released 0.4.X -> 0.7.X of their client.

What versions do you have?

3

I found that im using influxdb-0.7.0.gem. So I have removed 0.7 and installed influxdb-0.3.17.gem and still does not work :frowning: (still the same error)

Logstash:.6.7.0
InfluxDB: v1.7.5
Influx Output Plugin: v5.0.5

4

Do you get any data into influxdb?

5

no I do not get any data.

6

I have to presume that other users can use this output with v1.7.5 of influxdb.

The last "problem" issue was logged in the plugin repo in October last year indicated an issue with interpolation of the db setting and not a general communications issue, so presumably it mostly worked for this user.

I will do a local test myself.

7

I appreciate your help. If the plugin does not work, I will have to use http output plugin and influxdb's http api.

8

Happy to help. I need to become familiar with this plugin anyway.

Hold off on trying http as this plugin obviously has benefits.

9

Yes, definitely. I also want to utilize this plugin if possible. I wil try to find the way to work it out, too.

10

Any instructions on the InfluxDB setup would help me recreate what you have? I want to compare apples to apples not oranges :slight_smile:

11

I have followed the instructions on https://portal.influxdata.com/downloads/(install via Homebrew) which is like below

brew update
brew install influxdb
12

any config changes?

13

I haven't touched any configuration after installation.

14

Did you create the user and password etc

With InfluxDB running, visit localhost:9999.
Click Get Started

Enter a Username for your initial user.
Enter a Password and Confirm Password for your user.
Enter your initial Organization Name.
Enter your initial Bucket Name.
15

I did not create any user and set up any password for the user (since I didnt create any user). No organization nor bucket.

16

Seems that bucket = database

17

I have created database using CLI command.

18

Perhaps you need to create a organization, user, password and database (bucket) and then update the influxdb output to use those values (except organization, there is no setting for this).

19

I will try to create user and password.

20

After adding user and password, I was able to make it work. I think the doc should be updated accordingly.