Elasticsearch: introduce Routing parameter to save space

Description

Elastic search allows to set a routing parameter when creating documents to have related documents in the same shard:
http://www.elasticsearch.org/guide/reference/api/index_/
This makes sense for log messages as well: e.g. if you have extracted the session id from the message, it would be sensible to store all entries to that session in the same shard.

So I propose to introduce an additional configuration parameter "routing" similar to the existing parameters "index" and "index_type" that sets this parameter. If you have, for instance, stored the session id in the field sid, you might use

elasticsearch { ... routing => "%{sid}" ...}

I expect this to save space since session specific terms like user IDs and so forth are stored only in one shard.

Environment

None

Status

Assignee

Logstash Developers

Reporter

Hans-Peter Störr

Fix versions

Affects versions

Priority

Configure