Hi all , Has anyone worked on large scale es indices greater than 500 gb of data. need to ask few questions Thanks
Wanted to know if the data is not time specific like logs so how do others maintain shard size upto 40 gb. Initially our index had 5 primary shards and we had to create a new index with 10 primary shards to reduce size of each shard .now the same problem is occurring wherein each of 10 shards are upto 50 gb each and we are finding ways to solve this recurring problem.we cannot delete any data from the index as it fuels our critical search operations. Moreover the data will continue to grow as we expand our products.
Do you manage your own cluster or do you use a managed cluster?
IIRC, we can use the rollover API in an ILM policy for your index based on a condition on the index size. You might have to use index aliases though to ensure queries redirect to the new index instead of the old one.
We have AWS managed service. Buth problem with ilm is that it writes on only the current index only . We index our data on the basis of I'd. So if we need to update a particular document having an I'd we donot know on which index it will be present.so indexing the documents becomes a problem as we donot want to have duplicates in our indices.
Please frame and ask them here in comments. This way others will benefit from the answers too.
What do you want to know?
Never checked the size of the cluster, share your questions
I am trying to make a DaC approach for maintaining elastic rule through GitHub and facing kibana issues "where it says I'm unauthorised evn though having a super access"
Hi all, currently working at startup and we have started to face scaling issues. Example Database tables have started increasing rows. So how do you handle this kind of data? How to think in the direction of improving and writing queries...
Hi all,
Wanted to know the size of database in your respective orgs and choice of db. Also pls mention any challanges faced while scaling it
Mine Main db around 3 terra bytes ( no sharding so on a single node with replication) Choice ...
I have made a search for my project using elastic search in which i used synonyms and and fuzziness and basic matching. I want it to work as users feeling, what users want , can you guys help me how to implement it in elastic search, i t...
What is the best response time for computing over 1M rows in AWS Redshift for the first load ? Comment below 👇