How to get selected log from filebeat index by python

i am using python elasticsearch module to search filebeat log by using using keyword. same as i do in "Kibana Discover" but my search does not return anything.

# make sure ES is up and running
import requests
res = requests.get('http://xxxxx:9200')
print(res.content)


# Import Elasticsearch package 
from elasticsearch import Elasticsearch 
# Connect to the elastic cluster

es=Elasticsearch([{'host':'xxxxx','port':9200}],timeout=30)
es

es.search(index='*',  body={
  'query': {
    'match': {
      'message': 'SocketTimeoutException',
     }
  }
})

es

it should give me log file like this but it does not.

message:14:20:01,387 ERROR [com.yuma.jca.sockets.concox] (default-threads - 37) MessageWork SocketTimeoutException for imei: 0358735075610732-> Read timed out @timestamp:Aug 8, 2019 @ 15:20:01.559 ecs.version:1.0.0

but i get like this, only the print connection

b'{\n "name" : "kibana",\n "cluster_name" : "elasticsearch",\n "cluster_uuid" : "pspfiiegRre8OOFSsLWIhQ",\n "version" : {\n "number" : "7.2.0",\n "build_flavor" : "default",\n "build_type" : "deb",\n "build_hash" : "508c38a",\n "build_date" : "2019-06-20T15:54:18.811730Z",\n "build_snapshot" : false,\n "lucene_version" : "8.0.0",\n "minimum_wire_compatibility_version" : "6.8.0",\n "minimum_index_compatibility_version" : "6.0.0-beta1"\n },\n "tagline" : "You Know, for Search"\n}\n'

Hi @Fosiul_Alam,

return a result, here you print es which is the connection...

res = es.search(index='*', body={
print('result:', res)

will return the result of the query, i.e a dict with ["hits"]["hits"]...

For information to avoid importing/instaling libs only for this :

# make sure ES is up and running
import requests

you can use ping, already included in elasticsearch-py

https://elasticsearch-py.readthedocs.io/en/master/api.html?highlight=ping#elasticsearch.Elasticsearch.ping

Timeout is huge why you need a 30 second timeout there's maybe other problem that can be solved in your configuration??

es.search(index='*',  body={
  'query': {
    'match': {
      'message': 'SocketTimeoutException',
     }
  }

  
})

print('result:', res)

This gives me out put as
result: <Response [200]>

but it should give me the out put of the search result right ? as you mentioned .
what i am doing wrong then ?

es.search(index='*', body={"query": {"match_all": {}}})
#print(res['message'])
print('result:', res['hits'])

The Error I get is : -

Traceback (most recent call last):
File ".\test.py", line 25, in
print('result:', res['hits'])
TypeError: 'Response' object is not subscriptable

Hi,

You need to assign like in the code I wrote in the previous post.

res = es.search...
print(res)

Your problem is the res that you print is the one you assign in your first line.

Thanks,
this is what i get now with bellow code

data = es.search(index='*', body={
'query': {
'match': {
'message': 'SocketTimeoutException',
}
}
})
print(data['hits'])

{'total': {'value': 10000, 'relation': 'gte'}, 'max_score': 12.30989, 'hits': [{'_index': 'filebeat-7.2.0-2019.07.27-000001', '_type': 'file': {'path': '/444/TEST13/standalone/log/222.log'}}, 'message': '04:57:08,075 ERROR [com.yuma.jca.sockets.222] (default-threads - 12) MessageWork SocketTimeoutException for imei: 44444444444-> Read timed out', 'input': {'type': 'log'}, 'agent': {'id': 'e47d8400-1077-4fd1-a56b-68e69e1b129e', 'version': '7.2.0', 'type': 'filebeat', 'ephemeral_id': 'fdcfb726-af0c-4de6-acea-f20627caed57', 'hostname': 'TEST'}, 'ecs': {'version': '1.0.0'}, 'host': {'id': '982b7b8d1795445e9b8a9a794550123f', 'containerized': False, 'name': 'TEST', 'hostname': 'TEST', 'architecture': 'x86_64', 'os': {'platform': 'ubuntu', 'version': '18.04.2 LTS (Bionic Beaver)', 'family': 'debian', 'name': 'Ubuntu', 'kernel': '4.15.0-45-generic', 'codename': 'bionic'}}, 'cloud': {'instance': {'id': '135683766'}, 'region': 'lon1', 'provider': 'TEST1'}}}, {'_index': 'filebeat-7.2.0-2019.07.27-000001', '_type': '_doc', '_id': 'xuQXPGwBUSUoRIJvSPfY', '_score': 12.30989, '_source': {'@timestamp': '2019-07-29T04:57:34.540Z', 'ecs': {'version': '1.0.0'}, 'host': {'architecture': 'x86_64', 'os': {'name': 'Ubuntu', 'kernel':
'4.15.0-45-generic', ', 'log': {'offset': 2249855, 'file': {'path': '/444/TEST13/standalone/log/222.log'}}, 'message': '04:57:34,107 ERROR [com.yuma.jca.sockets.222] (default-threads - 13) MessageWork
SocketTimeoutException for imei: 352621109470648-> Read timed out', 'input': {'type': 'log'}}}, {'_index': 'filebeat-7.2.0-2019.07.27-000001',
'2019-07-29T05:06:17.742Z', 'input': {'type': 'log'}, 'log': {'offset': 2605382, 'file': {'path': '/444/TEST13/standalone/log/222.log'}}, 'message': '05:06:15,988 ERROR [com.yuma.jca.sockets.222] (default-threads - 17) MessageWork SocketTimeoutException for imei: 44444444444-> Read timed out'}}, {'_index': 'filebeat-7.2.0-2019.07.27-000001', '_type': '_doc', '_id': '5eYpPGwBUSUoRIJvN7Wg', '_score': 12.30989, '_source': {'@timestamp': '2019-07-29T05:17:08.791Z', 'input': {'type': 'log'}, 'ecs': {'version': '1.0.0'}, 'host': {'hostname': 'TEST', 'name': 'TEST', 'architecture': 'x86_64', ,e6-acea-f20627caed57', 'hostname': 'TEST'}, 'cloud': {'instance': {'id': '135683766'}, 'region': 'lon1', 'provider': 'TEST1'}, 'message': '05:17:08,787 ERROR [com.yuma.jca.sockets.vt600] (default-threads - 27) MessageWork SocketTimeoutException for imei: 121190262627-> Read timed out', 'log': {'offset': 2643432, 'file': {'path': '/444/TEST13/standalone/log/VT600.log'}}}}, {'_index': 'filebeat-7.2.0-2019.07.27-000001', '_type': '_doc', '_id': 'euYpPGwBUSUoRIJvAbB2', '_score': 12.30989, '_source': {'@timestamp': '2019-07-29T05:16:54.996Z', 'ecs': {'version': '1.0.0'}, 'host': {'os': {'kernel': '4.15.0-45-generic', 'codename': 'bionic', 'platform': 'ubuntu', 'version': '18.04.2 LTS (Bionic Beaver)', 'family': 'debian', 'name': 'Ubuntu'}, 'id': '982b7b8d1795445e9b8a9a794550123f', 'log': {'offset': 2983707, 'file': {'path': '/444/TEST13/standalone/log/222.log'}}, 'message': '05:16:54,976 ERROR [com.yuma.jca.sockets.222] (default-threads - 29) MessageWork SocketTimeoutException for imei: 44444444444-> Read timed out', 'input': {'type': 'log'}, 'agent': {'version': '7.2.0', 'type': 'filebeat', 'ephemeral_id': 'fdcfb726-af0c-4de6-acea-f20627caed57', 'hostname': 'TEST', 'id': 'e47d8400-1077-4fd1-a56b-68e69e1b129e'}}},
{'_index': 'filebeat-7.2.0-2019.07.27-000001', 'provider': 'TEST1'}, 'log': {'offset': 2989504, 'file': {'path': '/444/TEST13/standalone/log/222.log'}}, 'message': '05:17:03,820 ERROR [com.yuma.jca.sockets.222] (default-threads - 8) MessageWork SocketTimeoutException for imei: 352621109471182-> Read timed out', 'input': {'type': 'log'}}}, {'_index': 'filebeat-7.2.0-2019.07.27-000001', '_type': '_doc', '_id': 'V-lGPGwBUSUoRIJvsX-1', '_score': 12.30989, '_source': {'@timestamp': '2019-07-29T05:49:21.499Z', 'message': '05:49:21,269 ERROR [com.yuma.jca.sockets.222] (default-threads - 37) MessageWork SocketTimeoutException for imei: 352621109470648-> Read timed out', 'input': {'type': 'log'}, 'ecs': {'version': '1.0.0'}, 'host': {'name': 'TEST', 'os': {'family': 'debian', 'name': 'Ubuntu', 'kernel': '4.15.0-45-generic', 'codename': 'bionic', 'platform': 'ubuntu', 'version': '18.04.2 LTS (Bionic Beaver)'}, 'id': '982b7b8d1795445e9b8a9a794550123f', 'containerized': False, 'hostname': 'TEST', 'architecture': 'x86_64'}, 'agent': {'ephemeral_id': 'fdcfb726-af0c-4de6-acea-f20627caed57', 'hostname': 'TEST', 'id': 'e47d8400-1077-4fd1-a56b-68e69e1b129e', 'version': '7.2.0', 'type': 'filebeat'}, 'cloud': {'instance': {'id': '135683766'}, 'region': 'lon1', 'provider': 'TEST1'}, 'log': {'file': {'path': '/444/TEST13/standalone/log/222.log'}, 'offset': 4044045}}}, {'_index': 'filebeat-7.2.0-2019.07.27-000001', '_type': '_doc', '_id': 'pecvPGwBUSUoRIJvWkmU', '_score': 12.30989, '_source': {'@timestamp': '2019-07-29T05:23:51.119Z', 'log': {'offset': 3228117, 'file': {'path': '/444/TEST13/standalone/log/222.log'}}, 'message': '05:23:50,399 ERROR [com.yuma.jca.sockets.222] (default-threads - 17) MessageWork SocketTimeoutException for imei: 44444444444-> Read timed out', {'id': 'e47d8400-1077-4fd1-a56b-68e69e1b129e', 'version': '7.2.0', 'type': 'filebeat', 'ephemeral_id': 'fdcfb726-af0c-4de6-acea-f20627caed57', 'hostname': 'TEST'}, 'cloud': {'instance': {'id': '135683766'}, 'region': 'lon1', 'provider': 'TEST1'}}}, {'_index': 'filebeat-7.2.0-2019.07.27-000001', '_type': '_doc', '_id': 'K-g9PGwBUSUoRIJvLJ-Z', '_score': 12.30989, '_source': {'@timestamp': '2019-07-29T05:38:57.377Z', 'agent': {'type': 'filebeat', 'ephemeral_id': 'fdcfb726-af0c-4de6-acea-f20627caed57', : '135683766'}}, 'log': {'offset': 3782293, 'file': {'path': '/444/TEST13/standalone/log/222.log'}}, 'message': '05:38:56,520 ERROR [com.yuma.jca.sockets.222] (default-threads - 7) MessageWork SocketTimeoutException for imei: 352621109470648-> Read timed out', 'input': {'type': 'log'}}}, {'_index': 'filebeat-7.2.0-2019.07.27-000001', '_type': '_doc', '_id': 'JOg9PGwBUSUoRIJvFZ0f', '_score': 12.30989, '_source': {'@timestamp': '2019-07-29T05:38:51.372Z', 'log': {'offset': 3771978, 'file': {'path': '/444/TEST13/standalone/log/222.log'}}, 'message': '05:38:50,356 ERROR [com.yuma.jca.sockets.222] (default-threads - 40) MessageWork SocketTimeoutException for imei: 352121088182330-> Read timed out', 'input':
{'type': 'log'}, }}]}

if i want only this line which start with 'message' , how shall i pass ?

'message': '04:57:08,075 ERROR [com.yuma.jca.sockets.222] (default-threads - 12) MessageWork SocketTimeoutException for imei: 44444444444-> Read timed out'

Hi,

You need to use a for loop.
https://www.w3schools.com/python/python_for_loops.asp

for one_line in data['hits']['hits']:
    print('one message content: ', one_line.get('message'))

It will print for each line the content of the message key of the dictionary.

You can also check about elasticsearch-dsl it may be more easy to start with.

Hi
Thanks

thsi is i have done

data = es.search(index='filebeat-7.2.0-2019.07.27-000001', body={
'query': {
'match': {
'message': 'SocketTimeoutException',
}
}
})
#print(data['hits']['total']['value'])
print(data['hits']['hits'])
print(" getting one line")
for one_line in data['hits']['hits']:
print('one message content: ', one_line.get('message'))

out put for
print(data['hits']['hits'])

[{'_index': 'filebeat-7.2.0-2019.07.27-000001', '_type': '_doc', '_id': 'ROQWPGwBUSUoRIJv5O0L', '_score': 12.32211, '_source': {'@timestamp': '2019-07-29T04:57:08.532Z', 'log': {'offset': 2225916, 'file': {'path': '/root/TEST/standalone/log/DEVICE.log'}}, 'message': '04:57:08,075 ERROR [com.TEST.jca.sockets.DEVICE] (default-threads - 12) MessageWork SocketTimeoutException for zzzz: 11222-> Read timed out', 'input': {'type': 'log'}, 'agent': {'id': 'e47d8400-1077-4fd1-a56b-68e69e1b129e', 'version': '7.2.0', 'type': 'filebeat', 'ephemeral_id': 'fdcfb726-af0c-4de6-acea-f20627caed57', 'hostname': 'SERVER'}, 'ecs': {'version': '1.0.0'}, 'host': {'id': '982b7b8d1795445e9b8a9a794550123f', 'containerized': False, 'name': 'SERVER', 'hostname': 'SERVER', 'architecture': 'x86_64', 'os': {'platform': 'ubuntu', 'version': '18.04.2 LTS (Bionic Beaver)', 'family': 'debian', 'name': 'Ubuntu', 'kernel': '4.15.0-45-generic', 'codename': 'bionic'}}, 'cloud': {'instance': {'id': '135683766'}, 'region': 'lon1', 'provider': 'TEST'}}},
{'_index': 'filebeat-7.2.0-2019.07.27-000001', '_type': '_doc', '_id': 'xuQXPGwBUSUoRIJvSPfY', '_score': 12.32211, '_source': {'@timestamp': '2019-07-29T04:57:34.540Z', 'ecs': {'version': '1.0.0'}, 'host': {'architecture': 'x86_64', 'os': {'name': 'Ubuntu', 'kernel': '4.15.0-45-generic', 'codename': 'bionic', 'platform': 'ubuntu', 'version': '18.04.2 LTS (Bionic Beaver)', 'family': 'debian'}, 'id': '982b7b8d1795445e9b8a9a794550123f', 'containerized': False, 'name': 'SERVER', 'hostname': 'SERVER'}, 'agent': {'ephemeral_id': 'fdcfb726-af0c-4de6-acea-f20627caed57', 'hostname': 'SERVER', 'id': 'e47d8400-1077-4fd1-a56b-68e69e1b129e', 'version': '7.2.0', 'type': 'filebeat'}, 'cloud': {'provider': 'TEST', 'instance': {'id': '135683766'}, 'region': 'lon1'}, 'log': {'offset': 2249855, 'file': {'path': '/root/TEST/standalone/log/DEVICE.log'}}, 'message': '04:57:34,107 ERROR [com.TEST.jca.sockets.DEVICE] (default-threads - 13) MessageWork SocketTimeoutException for zzzz: 1111-> Read timed out', 'input': {'type': 'log'}}},
{'_index': 'filebeat-7.2.0-2019.07.27-000001', '_type': '_doc', '_id': 'BeUfPGwBUSUoRIJvRr5w', '_score': 12.32211, '_source': {'@timestamp': '2019-07-29T05:06:17.742Z', 'input': {'type': 'log'}, 'host': {'hostname': 'SERVER', 'architecture': 'x86_64', 'name': 'SERVER', 'os': {'codename': 'bionic', 'platform': 'ubuntu', 'version': '18.04.2 LTS (Bionic Beaver)', 'family': 'debian', 'name': 'Ubuntu', 'kernel': '4.15.0-45-generic'}, 'id': '982b7b8d1795445e9b8a9a794550123f', 'containerized': False}, 'agent': {'type': 'filebeat', 'ephemeral_id': 'fdcfb726-af0c-4de6-acea-f20627caed57', 'hostname': 'SERVER', 'id': 'e47d8400-1077-4fd1-a56b-68e69e1b129e', 'version': '7.2.0'}, 'ecs': {'version': '1.0.0'}, 'cloud': {'instance': {'id': '135683766'}, 'region': 'lon1', 'provider': 'TEST'}, 'log': {'offset': 2605382, 'file': {'path': '/root/TEST/standalone/log/DEVICE.log'}}, 'message': '05:06:15,988 ERROR [com.TEST.jca.sockets.DEVICE] (default-threads - 17) MessageWork SocketTimeoutException for zzzz: 11222-> Read timed out'}},
{'_index': 'filebeat-7.2.0-2019.07.27-000001', '_type': '_doc', '_id': 'JOg9PGwBUSUoRIJvFZ0f', '_score': 12.32211, '_source': {'@timestamp': '2019-07-29T05:38:51.372Z', 'log': {'offset': 3771978, 'file': {'path': '/root/TEST/standalone/log/DEVICE.log'}}, 'message': '05:38:50,356 ERROR [com.TEST.jca.sockets.DEVICE] (default-threads - 40) MessageWork SocketTimeoutException for zzzz: 333333-> Read timed out', 'input': {'type': 'log'}, 'host': {'hostname': 'SERVER', 'architecture': 'x86_64', 'name': 'SERVER', 'os': {'family': 'debian', 'name': 'Ubuntu', 'kernel': '4.15.0-45-generic', 'codename': 'bionic', 'platform': 'ubuntu', 'version': '18.04.2 LTS (Bionic Beaver)'}, 'id': '982b7b8d1795445e9b8a9a794550123f', 'containerized': False}, 'agent': {'type': 'filebeat', 'ephemeral_id': 'fdcfb726-af0c-4de6-acea-f20627caed57', 'hostname': 'SERVER', 'id': 'e47d8400-1077-4fd1-a56b-68e69e1b129e', 'version': '7.2.0'}, 'ecs': {'version': '1.0.0'}, 'cloud': {'instance': {'id': '135683766'}, 'region': 'lon1', 'provider': 'TEST'}}}]

but for
print('one message content: ', one_line.get('message'))
i get

one message content:  None
one message content:  None
one message content:  None
one message content:  None

What i am doing wrong ?
Thanks for all your help

Try to debug with pdb to check the structure of the result.

https://docs.python.org/3/library/pdb.html

You need to understand the structure of "one_line" it suppose to be a dict then you can access the key.

https://docs.python.org/3/tutorial/datastructures.html?highlight=dictionary#dictionaries

this code works to get the log file starts with 'message' by iterating a List , the Dictionary

data = es.search(index='filebeat-7.2.0-2019.07.27-000001', body={
'query': {
'match': {
'message': 'SocketTimeoutException',
}
}
})
length= len(data['hits']['hits'])
for i in range(length):
print (data['hits']['hits'][i]['_source']['message'])