We're updating the issue view to help you get more done. 

Elasticsearch input, elasticsearch close connection due to scroll_id too long

Description

Hello,
I'm trying to output to S3 6 month of data coming from an elasticsearch.
I have a backlog of 1 week for an index named logstash-* and a backlog of 6 month for an other index.

If I configure my elasticearch input for logstash-* it works but for another-* it doesn't.

We can see errors on elasticsearch with this info :
[2013-10-09 17:43:12,075][WARN ][http.netty ] [i-338c3651] Caught exception while handling client http traffic, closing connection [id: 0x9c6588c8, /10.45.10.177:40916 => /10.45.10.92:9200]
org.elasticsearch.common.netty.handler.codec.frame.TooLongFrameException: An HTTP line is larger than 4096 bytes.

If I do a tcpdump I can see the scroll_id in the request is huge which could explain the 4096 bytes limit.

Is there anyway to post the scroll_id rather than to set it in the URL.
https://github.com/logstash/logstash/blob/master/lib/logstash/inputs/elasticsearch.rb#L122

My problem seems similar to that one :
https://groups.google.com/forum/#!msg/elasticsearch/jXKLdCrrIHM/gG_y--hOzlcJ

What do you think?

I guess it can be a big problem for people having a lot of data to migrate from Logstash 1.1 to 1.2 as the logic will be the same (use elasticsearch input)

Environment

None

Status

Assignee

Logstash Developers

Reporter

jujugrrr

Labels

Affects versions

1.2.1

Priority