How to search for a data in elasticsearch?

Elastic Stack Elasticsearch
1

Recently I followed a tutorial provided in the elasticsearch site to push data to elastic cloud and I have successfully been able to do that.

I am also able to post a sample document and search for a term in it.But I can't do the same for an apache log.

Below is the query I posted to elastic cloud

curl -H 'Content-Type: application/json' -k -u elastic:changeme -XPUT https://5a482b9559fc4a59b6b*************.ap-southeast-1.aws.found.io:9243/test-data/cities/21 -d '{ "rank": 21, "city": "Bangalore", "state": "Karnataka"}'

When I search for a term called Bangalore, I use the below query

curl -H 'Content-Type: application/json' -k -u elastic:changeme -XGET https://5a482b9559fc4a5****************.ap-southeast-1.aws.found.io:9243/test-data/cities/_search?pretty=true -d '{"query": {"query_string": {"query": "bangalore"}}}'

I successfully get the following result

{
  "took" : 4,
  "timed_out" : false,
  "_shards" : {
    "total" : 5,
    "successful" : 5,
    "skipped" : 0,
    "failed" : 0
  },
  "hits" : {
    "total" : 1,
    "max_score" : 0.2876821,
    "hits" : [
      {
        "_index" : "test-data",
        "_type" : "cities",
        "_id" : "21",
        "_score" : 0.2876821,
        "_source" : {
          "rank" : 21,
          "city" : "Bangalore",
          "state" : "Karnataka"
        }
      }
    ]
  }
}

Now for the sample apache logs I posted to elastic cloud, if I run a query for index .monitoring-kibana-6-2018.03.16 and type doc, it gives me back all the information for the particular log

Query

curl -H 'Content-Type: application/json' -k -u elastic:changeme -XGET https://5a482b9559fc4a5************.ap-southeast-1.aws.found.io:9243/.monitoring-kibana-6-2018.03.19/doc/_search?pretty=true

The information sent back

{
  "took" : 1,
  "timed_out" : false,
  "_shards" : {
    "total" : 1,
    "successful" : 1,
    "skipped" : 0,
    "failed" : 0
  },
  "hits" : {
    "total" : 4935,
    "max_score" : 1.0,
    "hits" : [
      {
        "_index" : ".monitoring-kibana-6-2018.03.19",
        "_type" : "doc",
        "_id" : "zQyPO2IBFM2Pr3qzOtYp",
        "_score" : 1.0,
        "_source" : {
          "cluster_uuid" : "-fcvGIrYRpaQInvIaV2yxA",
          "timestamp" : "2018-03-19T00:01:05.061Z",
          "interval_ms" : 10000,
          "type" : "kibana_stats",
          "source_node" : {
            "uuid" : "QdV84OHQQHK2cqfcosLYOg",
            "host" : "172.29.73.212",
            "transport_address" : "172.29.73.212:19974",
            "ip" : "172.29.73.212",
            "name" : "instance-0000000002",
            "timestamp" : "2018-03-19T00:01:05.062Z"
          },
          "kibana_stats" : {
            "cloud" : {
              "name" : "aws",
              "id" : "ec2dd6c8-6f45-fabe-5bf4-8184ef1b3d63"
            },
            "concurrent_connections" : 66357,
            "os" : {
              "load" : {
                "1m" : 0.29833984375,
                "5m" : 0.470703125,
                "15m" : 0.5126953125
              },
              "memory" : {
                "total_in_bytes" : 32168431616,
                "free_in_bytes" : 1181507584,
                "used_in_bytes" : 30986924032
              },
              "uptime_in_millis" : 5260700000
            },
            "process" : {
              "event_loop_delay" : 90620.95409584045,
              "memory" : {
                "heap" : {
                  "total_in_bytes" : 153329664,
                  "used_in_bytes" : 137886824,
                  "size_limit" : 872415232
                },
                "resident_set_size_in_bytes" : 209448960
              },
              "uptime_in_millis" : 306763138
            },
            "requests" : {
              "disconnects" : 0,
              "total" : 63356,
              "status_codes" : {
                "200" : 1437,
                "302" : 61069,
                "304" : 839,
                "404" : 11
              }
            },
            "response_times" : {
              "average" : 2447.6666666666665,
              "max" : 3864
            },
            "timestamp" : "2018-03-19T00:01:00.092Z",
            "kibana" : {
              "uuid" : "0f753e6b-a648-433a-8b35-36de5ce4a3a5",
              "name" : "ba105a5a75ee",
              "index" : ".kibana",
              "host" : "ba105a5a75ee",
              "transport_address" : "0.0.0.0:18472",
              "version" : "6.2.2",
              "snapshot" : false,
              "status" : "green"
            },
            . . . . . . . . . . . . . . . . . . .
            . . . . . . . . . . . . . . . . . . .

It returns a very big output, so I am only showing a part of it.Now If I try to run a query to search for a term say response_times

Query

curl -H 'Content-Type: application/json' -k -u elastic:QbKxXTTXwxnV4kQ0UA48Kedz -XGET https://5a482b9559fc4a5****************.ap-southeast-1.aws.found.io:9243/.monitoring-kibana-6-2018.03.19/doc/_search?pretty=true -d '{"query": {"query_string": {"query": "response_times"}}}'

I get the below output

{
  "took" : 3,
  "timed_out" : false,
  "_shards" : {
    "total" : 1,
    "successful" : 1,
    "skipped" : 0,
    "failed" : 0
  },
  "hits" : {
    "total" : 0,
    "max_score" : null,
    "hits" : [ ]
  }
}

As you can see, I want the value for a field.So why doesn't the same search query work for apache logs?

2

R u using kibana or curl for searching

3

I am using curl

4

@souvik_ray

try this command in curl:

curl -XGET "localhost:9200/indexname" -d '{
query:{
match :{
city : "Banglore"
}
}
}'

you will get the record which contains city name banglore.

5

As I answered on SOF:

Depending on what you want to do but you can run a Range Query for example on that response_times field:

curl -XGET 'localhost:9200/_search?pretty' -H 'Content-Type: application/json' -d'
{
    "query": {
        "range" : {
            "response_times" : {
                "gte" : 3800,
                "lte" : 3900
            }
        }
    }
}
'
7

So ideally my curl command should be something like

curl -H 'Content-Type: application/json' -k -u elastic:changeme -XGET https://5a482b9559fc4a59***************.ap-southeast-1.aws.found.io:9243/.monitoring-kibana-6-2018.03.19/doc/_search?pretty=true -d '{"query": {"range" : {"response_times" : {"gte" : 3800,"lte" : 3900}}}}'

But it doesn't match anything.

8

Ha sorry. It should be I think:

curl -XGET 'localhost:9200/_search?pretty' -H 'Content-Type: application/json' -d'
{
    "query": {
        "range" : {
            "kibana_stats.response_times" : {
                "gte" : 3800,
                "lte" : 3900
            }
        }
    }
}
'
9

I am sorry but it still doesn't work.I don't get any hits.

10

have u installed kibana?

11

@Shaik_Thowsif Yes I have

12

then open kibana and search using console

13

Could you provide a full recreation script as described in About the Elasticsearch category. It will help to better understand what you are doing. Please, try to keep the example as simple as possible.

A full reproduction script will help readers to understand, reproduce and if needed fix your problem. It will also most likely help to get a faster answer.

14

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.