Duplicate entries in Kibana
Description
Attachments
- 10 Feb 2014, 10:03 AM
- 10 Feb 2014, 10:00 AM
- 10 Feb 2014, 09:57 AM
- 10 Feb 2014, 09:57 AM
Gliffy Diagrams
Activity
Sergey Zemlyanoy February 13, 2014 at 2:19 PM
Thanks guys! Your advice helped
Sergey Zemlyanoy February 13, 2014 at 9:46 AM
Wow! Preliminary I see that it woks for me. I will test for some time and leave feedback here. Thanks
l February 13, 2014 at 9:31 AM
I believe it's linked to: https://groups.google.com/forum/#!topic/logstash-users/ZJ08bJAKixY
hi,
This was happening owing to the manner in which I was introducing new lines inside the monitored log file. In order to familiarize myself with the filter capabilities of logstash, I used the vi editor to open and edit (add new lines) the monitored log file. This somehow would mess with logstash's ability to accurately track the last read position in the log file. The ".sincedb" (strangely) file would have multiple entries in it. I dont really understand why that should happen, though.
But, If I would introduce a new line into the log file by doing an - echo "hello" >> log_file - things would work as expected.
Sergey Zemlyanoy February 12, 2014 at 2:42 PM
when I append someerrorstring1 -> apache_error.log
in Kibana see:
someerrorstring1
when I append someerrorstring2 -> apache_error.log
in Kibana I see new entries:
someerrorstring1
someerrorstring2
So it ends up with duplicates
someerrorstring1
someerrorstring1
someerrorstring2
Philippe Weber February 12, 2014 at 2:35 PM
Sorry but it's not fully clear for me, could you post an example
Dear all,
I have the logstash+elasticsearch+Kibana environment and the problem which I encountered is that I get duplicate messages on Kibana which has a just a unique filed _id, all the rest are similar.
F.i.
{
"_index": "logstash-2014.02.07",
"_type": "apacheerror",
"_id": "uJ4lHB0ASpWqWx683tcdMw",
"_score": null,
"_source": {
"message": "[Fri Feb 07 16:58:47 2014] [error] [client ::1] Action \"/index\" does not exist.",
"@version": "1",
"@timestamp": "2014-02-07T15:58:48.062+00:00",
"type": "apacheerror",
"host": "web2.improve",
"path": "/var/log/httpd/error_log",
"timestamp": "[Fri Feb 07 16:58:47 2014]",
"severity": "error",
"client_ip": "::1",
"info": "Action \"/index\" does not exist.",
"tags": [
"apacheerror_groked"
]
},
"sort": [
1391788728062,
1391788728062
]
}
{
"_index": "logstash-2014.02.07",
"_type": "apacheerror",
"_id": "azfX_OPuR2S4N2cmzsplDA",
"_score": null,
"_source": {
"message": "[Fri Feb 07 16:58:47 2014] [error] [client ::1] Action \"/index\" does not exist.",
"@version": "1",
"@timestamp": "2014-02-07T15:58:48.062+00:00",
"type": "apacheerror",
"host": "web2.improve",
"path": "/var/log/httpd/error_log",
"timestamp": "[Fri Feb 07 16:58:47 2014]",
"severity": "error",
"client_ip": "::1",
"info": "Action \"/index\" does not exist.",
"tags": [
"apacheerror_groked"
]
},
"sort": [
1391788728062,
1391788728062
]
}
Is this behaviour expected in ES/Kibana or how can I adjust not to receive duplicates?
Thanks in advance