Send sensor data to ElasticSearch/Kibana on Raspberry Pi LAN?

Elastic Stack Elasticsearch
1

Is it possible to send sensor data from an ESP32 to a Raspberry Pi running ELK, all on a LAN? I'm also wondering if we can then send data from the ELK interface back to that ESP32. I've seen videos where were sensor data is sent to the kibana interface, but it requires having a current timestamp, which we won't have on a LAN. We could add real time clocks, I assume that would fix the issue?

I'm not incredibly well versed on ELK, and I've done a bit of googling to see if this exact set up has been done and have yet to find it. We're also wanting to add a couple different buttons that, when pressed, will push data to the ESP32.

I would certainly prefer doing this over building my own website.

Thanks

2

It's possible, but it might be a bit slow due to the resources of the RPi.

It usually uses $now as defined by the system time, so it should be fine.

3

Were not necessarily locked into using a pi. It was convenient for building a website on a LAN, but I've recently discovered elasticsearch and its amazing. My question is, how should data be transferred between the devices on this LAN? Were using an ESP32 with 4 water sensors and an ultrasonic sensor. We also want to be able to send data from the local website (ELK) to the ESP32.

I saw something called logstash-gelf on this website. Due to my inexperience, I'm not really sure if it will work on a LAN.

Also, I'm assuming the system time of the RPi can be altered by the user? Like, if we have a real time clock on the Pi and ask the user to enter the time and date on start up type of thing. We would most likely try it out on a RPi, and if that doesn't work, then use an actual PC.

Just trying to get a confirmation that "Yes, you can get X to work with Y" so I can dive in and know that if i figure everything out, it will eventually work out, rather than spending a lot of time on something, only to figure out it won't work because I'm on a LAN.

4

That connector seems good, I guess it depends on what you end up coding in.

I know a lot of dev on the ESPs get done on Python, so there's also GitHub - cmanaha/python-elasticsearch-logger: Python Elasticsearch handler for the standard python logging framework.

On a personal note, I'm super keen to hear your progress on this. I'm looking at doing something similar myself :smiley:

5

Thank you in advance warkolm, I really appreciate the advice.

It's extremely overwhelming, as I literally just learned about elasticsearch today and I don't understand what 99% of the terms used are lol

We were planning on programming the ESP32 on the arduino IDE since there are libraries for everything we need to do. As I'm typing this, I realize the link I sent you is actually in python....

So what I should be researching is a connector right?

Edit: Cleared some things up

6

We're here to help :slight_smile:

Yeah. Just something to send the data out as an event. If it's via TCP then you can use Logstash to receive and then send to Elasticsearch. If it can be done as JSON and you can send if over HTTP, then you can send . it to Elasticsearch directly.

7

So the guy on this site was able to publish it via HTTP, but his problem generating an Elasticsearch compatible timestamp. He used a NTP unix timestamp to solve to problem, but we cant use NTP since we won't be connected to the internet. Is publishing directly to Elasticsearch via HTTP really that difficult?

8

In that link Isaac is just using NTP to make things as accurate as possible. If you set the system time on the device then that should be fine.

So just use datetime (in python), or whatever the equivalent is for your language when you build the json message.

9

FWIW, I have a bunch of these things at home and I've fiddled with a few different options. If it's of any help to you, I found https://github.com/marcoskirsch/nodemcu-httpserver to be reasonably easy to install on the device and then I just polled them periodically from the RPi. Below is the lua server file for a dht11 sensor I dropped in.

return function (connection, req, args)
  dofile("httpserver-header.lc")(connection, 200, 'json')

  pin = 4
  gnd = 3
  high = nil

  if gnd ~= nil then
    gpio.mode(gnd, gpio.OUTPUT)
    gpio.write(gnd, gpio.LOW)
  end
  if high ~= nil then
    gpio.mode(high, gpio.OUTPUT)
    gpio.write(high, gpio.HIGH)
  end
  gpio.mode(pin, gpio.INPUT)

  status, temp, humidity, temp_dec, humi_dec = dht.read(pin)
  if status == dht.OK then
    connection:send('{"error":0, "temp":'..temp..', "humidity":'..humidity..'}')
  elseif status == dht.ERROR_CHECKSUM then
    connection:send('{"error":1, "error_message":"Checksum error occurred when checking values"}')
  elseif status == dht.ERROR_TIMEOUT then
    connection:send('{"error":1, "error_message":"Timeout occurred when checking values"}')
  end
end

This allowed me to have much more complicated logic for shipping the data in than I was willing to write on the edge device

1 Like
10

Oh that's great, accuracy is not a concern in this application fortunately!

So on another note, can I send data from the website to the ESP32? We want to be able to click a button on the web page and have it send a command to the ESP32.

It will also have an option for the user to input a time (for example, 6:00 PM) and a number, it will then send that number to the ESP32 when it has reach that time.

Sorry, I know these questions are stupid, but finding specific information like this is difficult lol

11

I'd avoid situations where arbitrary/user data can be sent to these little devices. They're super low power and very easy to overwhelm.

12

Even if it's small data?

The reason being were going to eventually need to calibrate these sensors. We also have our ESP32 connected to a feeding motor that we want activated at certain times of the day.

So basically, were taking measurements from 4 probes and an ultrasonic sensor. Were then sending that data to a RPi (or computer) to display on a local webserver. We then want to be able to control sensor calibration/feeding from that webserver. Originally, I intended on building a website from scratch, transferring data to and from using MQTT.

13

I mean, I wouldn't trust them for anything that could have any significant load, but I've never tried to build out something exactly like this. I'd personally think about it as how much processing I can move off these edge devices and onto a central processing component. If the calibration only needs to be done once or rarely, I'd probably look at how I can do that without having to do the calibration as part of a bootstrapping vs an ongoing/open re-calibration endpoint/server.

1 Like
14

Yeah, most of these operations are rarely done. Calibration is done once every couple years, and the user will only have the ESP32 turn on the feeding motor once or twice per day.

15

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.