Input Elasticsearch / Output Elasticsearch state
Description
Gliffy Diagrams
Activity
Show:
Philippe Weber August 1, 2014 at 12:08 PM
Closing in favor of PR
Thomas De Smedt October 18, 2013 at 2:08 PM
added a clean pull request on https://github.com/logstash/logstash/pull/730
Thomas De Smedt October 18, 2013 at 12:53 PMEdited
i have a possible fix at https://github.com/Syntigo-nv/logstash/commit/754df6d7ff91917927850b955906013262903366 but need to cleanup this repository a bit first before issuing a pull request (i'm still learning to branch correctly)
It allows you to do:
input {
elasticsearch {
host => "localhost"
index => "mydata-2018.09.*"
query => "data_version:3.0"
scroll_size => 500
include_meta => true
meta_field => "es_orig"
}
}
filter {
mutate { replace => ["data_version","3.1"] }
}
output {
elasticsearch_http {
host => "localhost"
index => "%{[es_orig][_index]}"
index_type => "%{[es_orig][_type]}"
document_id => "%{[es_orig][_id]}"
}
}Chris Denneen September 25, 2013 at 1:54 AM
Can you grab id from input and use on output as key?
Richard Pijnenburg September 13, 2013 at 9:33 AM
We will have to brainstorm about this.
If I read indice from input and send to output this seems to work (when no indice errors occur). Problem is if I run this configuration again it will duplicate the destination indice. Need a way to keep track of what it's either grabbed already from input or probably better what it has in output indice. If we track state upon input we might miss something on second run if never indexed successful to output. Maybe a check upon post if this entry exists? If so either just update or skip.