Querying only logs coming from source x

Elastic Stack Elasticsearch
1

Hi,

I am trying to search only logs that come from a certain source.

Here is what I tried to do:

GET filebeat-*/log/_search
{
"query": {
"bool":{
"must": [
{ "match": { "source": "/opt/modulo/smsc/cdr/traffic.cdr" } }
]
}
}
}

From some reason this retrieves me all logs also logs that doent come from this exact path.

How can I retrieve only logs from this path?

2

I could not reproduce this using a vanilla setup. Maybe the template is wrong? Please post the output of GET _template/filebeat.

3

Hi I get an empty reply, what is this query?

4

There should have been a template for your index. Probably the absence of it is the reason why you are querying and more is coming. What about GET _template? Can you post the output?

1 Like
5

Ummm... from some reason also this gives me a blank message

6

I think I know what's going on, but I need one last confirmation. Expand the filebeat-* index pattern into an existing index, for instance, filebeat-2017.06.22 (you can get a list of existing indices with GET _cat/indices) and then get the output of GET filebeat-2027.06.22/_mapping. Post this output here please.

7

Hi,

Sorry about the delays,

The query is:

GET filebeat-2017.06.14/_mapping

The output is pretty long. Since the the reply is limited to 7000 chars I will split the output:

{
  "filebeat-2017.06.14": {
    "mappings": {
      "log": {
        "properties": {
          "@timestamp": {
            "type": "date"
          },
          "@version": {
            "type": "text",
            "fields": {
              "keyword": {
                "type": "keyword",
                "ignore_above": 256
              }
            }
          },
          "Jms_Lai_TS": {
            "type": "text",
            "fields": {
              "keyword": {
                "type": "keyword",
                "ignore_above": 256
              }
            }
          },
          "apiMedTimeGot": {
            "type": "text",
            "fields": {
              "keyword": {
                "type": "keyword",
                "ignore_above": 256
              }
            }
          },
          "apiMedTimeSent": {
            "type": "text",
            "fields": {
              "keyword": {
                "type": "keyword",
                "ignore_above": 256
              }
            }
          },
          "authCBGot": {
            "type": "text",
            "fields": {
              "keyword": {
                "type": "keyword",
                "ignore_above": 256
              }
            }
          },
          "authCBSent": {
            "type": "text",
            "fields": {
              "keyword": {
                "type": "keyword",
                "ignore_above": 256
              }
            }
          },
          "beat": {
            "properties": {
              "hostname": {
                "type": "text",
                "fields": {
                  "keyword": {
                    "type": "keyword",
                    "ignore_above": 256
                  }
                }
              },
              "name": {
                "type": "text",
                "fields": {
                  "keyword": {
                    "type": "keyword",
                    "ignore_above": 256
                  }
                }
              },
              "version": {
                "type": "text",
                "fields": {
                  "keyword": {
                    "type": "keyword",
                    "ignore_above": 256
                  }
                }
              }
            }
          },
          "billable": {
            "type": "text",
            "fields": {
              "keyword": {
                "type": "keyword",
                "ignore_above": 256
              }
            }
          },
          "blackListCBGot": {
            "type": "text",
            "fields": {
              "keyword": {
                "type": "keyword",
                "ignore_above": 256
              }
            }
          },
          "blackListCBSent": {
            "type": "text",
            "fields": {
              "keyword": {
                "type": "keyword",
                "ignore_above": 256
              }
            }
          },
8

Second part (probably not an useful part:

      "calledImsi": {
        "type": "text",
        "fields": {
          "keyword": {
            "type": "keyword",
            "ignore_above": 256
          }
        }
      },
      "calledMsc": {
        "type": "text",
        "fields": {
          "keyword": {
            "type": "keyword",
            "ignore_above": 256
          }
        }
      },
      "calledMscNpi": {
        "type": "text",
        "fields": {
          "keyword": {
            "type": "keyword",
            "ignore_above": 256
          }
        }
      },
      "calledMscTon": {
        "type": "text",
        "fields": {
          "keyword": {
            "type": "keyword",
            "ignore_above": 256
          }
        }
      },
      "calledNumber": {
        "type": "text",
        "fields": {
          "keyword": {
            "type": "keyword",
            "ignore_above": 256
          }
        }
      },
      "callingImsi": {
        "type": "text",
        "fields": {
          "keyword": {
            "type": "keyword",
            "ignore_above": 256
          }
        }
      },
      "callingMsc": {
        "type": "text",
        "fields": {
          "keyword": {
            "type": "keyword",
            "ignore_above": 256
          }
        }
      },
      "callingNumber": {
        "type": "text",
        "fields": {
          "keyword": {
            "type": "keyword",
            "ignore_above": 256
          }
        }
      },
      "campignId": {
        "type": "text",
        "fields": {
          "keyword": {
            "type": "keyword",
            "ignore_above": 256
          }
        }
      },
      "cdrType": {
        "type": "text",
        "fields": {
          "keyword": {
            "type": "keyword",
            "ignore_above": 256
          }
        }
      },
      "channel": {
        "type": "text",
        "fields": {
          "keyword": {
            "type": "keyword",
            "ignore_above": 256
          }
        }
      },
      "clientId": {
        "type": "text",
        "fields": {
          "keyword": {
            "type": "keyword",
            "ignore_above": 256
          }
        }
      },
      "concatenated": {
        "type": "boolean"
      },
      "concatenatedFrom": {
        "type": "text",
        "fields": {
          "keyword": {
            "type": "keyword",
            "ignore_above": 256
          }
        }
      },
      "deferred": {
        "type": "text",
        "fields": {
          "keyword": {
            "type": "keyword",
            "ignore_above": 256
          }
        }
      },
      "destinationProtocol": {
        "type": "text",
        "fields": {
          "keyword": {
            "type": "keyword",
            "ignore_above": 256
          }
        }
      },
      "gmt1": {
        "type": "text",
        "fields": {
          "keyword": {
            "type": "keyword",
            "ignore_above": 256
          }
        }
      },
      "gmt2": {
        "type": "text",
        "fields": {
          "keyword": {
            "type": "keyword",
            "ignore_above": 256
          }
        }
      },
      "host": {
        "type": "text",
        "fields": {
          "keyword": {
            "type": "keyword",
            "ignore_above": 256
          }
        }
      },
      "input_type": {
        "type": "text",
        "fields": {
          "keyword": {
            "type": "keyword",
            "ignore_above": 256
          }
        }
      },
9

The issue here is that the proper filebeat template was not created. If you are not indexing directly in elasticsearch from filebeat then you need to load the index template before indexing any data.

10

Hi,

Thanks, I didnt completely understand what does this mean, but I tried to fix any way.

since I am sending from Filebeat to Logstash to ES I understood I have to do this manually. To this manually in Centos I understood I wrote something like this:

curl -H 'Content-Type: application/json' -XPUT 'http://192.168.1.116:9200/_template/filebeat' -d@/etc/filebeat/filebeat.template.json

Is this right?

I still get for the query:

GET filebeat-*/log/_search
{
"query": {
"bool":{ 
"must": [
{ "match": { "source": "\/path_to_files" } }
]
}
}
}

all the logs and not only the logs in the path I wanted.

On the other I dont get now for:
GET _template

an empty answer.

11

Maybe there is another way all together to obtain logs only from a certain source?
By the way I have control to put the logs on different directories if this can help

12

The command is wrong, there is a missing space, it should be:

curl -H 'Content-Type: application/json' -XPUT 'http://192.168.1.116:9200/_template/filebeat' -d @/etc/filebeat/filebeat.template.json

You need to get it working, this is crucial. You may verify it worked by running GET _template and verifying that it is not an empty response.

Lastly, after you properly create the template, for the template to be effective, you will need to delete all filebeat-* indices and reindex everything again.

Note: if you can not reindex externally, then let me know, since there is a way to ask Elasticsearch to reindex internally

13

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.