Duplicated documents elasticsearch embedded

Description

In my Windows Server, I've NxLog configured to send Setup eventviewer logs to logstash through JSON. Setup eventviewer has a total of 1991 logs.

Logstash is configured to send that logs to two different destinations: File and elastic search.

Here is my config:

logstash.conf

All the 1991 logs is sent to the file, perfectly. But they are duplicated when inserted into elasticsearch. The duplicated events has diferent _id

See example bellow:

I've search for a single event, registered as a RecordNumber of 1991 (my last event). RecordNumber is an eventlog unique identifier incremented by Windows every time an event is generated by the system in a particular log scope (in my case Setup).

/elasticsearch_data/json.log

In elasticsearch, I have did this query:

elasticsearch query

An this is the result:

elastic search query results

Is you see, the _id is different for each one.

I already cleaned up entire logstash with the bellow comand, purged elasticsearch index directory (/elasticsearch_data/*) and restarted logstash, but no sucess...

clean elasticsearch

Any idea what might be happening?

Status

Assignee

Logstash Developers

Reporter

Bruno Galindro da Costa

Affects versions

Configure