Elastic search index
Hi all , Has anyone worked on large scale es indices greater than 500 gb of data. need to ask few questions Thanks
InbornBattle
Stealth
6 months ago
Wanted to know if the data is not time specific like logs so how do others maintain shard size upto 40 gb. Initially our index had 5 primary shards and we had to create a new index with 10 primary shards to reduce size of each shard .now the same problem is occurring wherein each of 10 shards are upto 50 gb each and we are finding ways to solve this recurring problem.we cannot delete any data from the index as it fuels our critical search operations. Moreover the data will continue to grow as we expand our products.
Do you manage your own cluster or do you use a managed cluster?
IIRC, we can use the rollover API in an ILM policy for your index based on a condition on the index size. You might have to use index aliases though to ensure queries redirect to the new index instead of the old one.
InbornBattle
Stealth
6 months ago
We have AWS managed service. Buth problem with ilm is that it writes on only the current index only . We index our data on the basis of I'd. So if we need to update a particular document having an I'd we donot know on which index it will be present.so indexing the documents becomes a problem as we donot want to have duplicates in our indices.
Please frame and ask them here in comments. This way others will benefit from the answers too.
What do you want to know?
RealOldmonk
Stealth
6 months ago
Never checked the size of the cluster, share your questions
