Dynamodb. Close to 9 TB total across all partitions. 
  
 Biggest problem is hot partitions for sure.

Software Engineer

Software Developer

Hi all,  Wanted to know the size of database in your respective orgs and choice of db. Also pls mention any challanges faced while scaling it  Mine  Main db around 3 terra bytes ( no sharding so on...

Welcome to IT Company Discussion's Private Community

Hi all,

Wanted to know the size of database in your respective orgs and choice of db. Also pls mention any challanges faced while scaling it

Mine 
Main db around 3 terra bytes ( no sharding so on a single node with replication)
Choice of db maria db

Size of database and choice of databse

Senior Software Engineer

Size of database and choice of databse- Grapevine

Welcome to Software Engineers's Private Community

Hi all, currently working at startup and we have started to face scaling issues. Example Database tables have started increasing rows. So how do you handle this kind of data? How to think in the direction of improving and writing queries on billions of records to perform optimal?
Can people with experience help me with any resources or personal experience they have gone through when going from 1 to 100?
Minor of the minor suggestions would help.
Grateful and thanks to all in advance.🙌

Suggestions for tackling scaling issues

Going through the job posting I've seen this sentence pop up in the requirements multiple times:
"Have delivered, extended, and maintained large scale distributed systems."
I'm having a hard time understanding what this means in a job context. Can anybody explain with like a simple example of delivering a distributed system that would make them eligible for this requirement? If I'm working on a webapp that uses tech like kaftka and zookeeper, can I consider myself as meeting this requirement?

What are distributed systems in Job context?

Hi all ,
Has anyone worked on large scale es indices greater than 500 gb of data. need to ask few questions
Thanks

Elastic search index

Welcome to Office Gossip's Private Community

I work for a growing startup in Bangalore. The data team size is very small with no data lead. Not a single query we hit the DB gives any result. Owing to cost cutting my firm has cut down the instance size and they're planning to downgrade the DB too. We data analysts are relying on the data engineering team to fetch us results for our queries which takes a long time to get a single table. This is highly inefficient. The data size is growing at a massive scale however the analytics efforts are trickling down due to poor infrastructure. 

Is this happening in your data team too? How can I solve this problem given there are no experienced data leaders in our firm to voice out.

Rant of a Data Analyst

Data Analyst

Actual job: export these 1894 user emails into an excel sheet for marketing by EoD please

Interview: In 15m design a 100k rps platform

We have our database and data in MySQL local machine. Monthly data storage and fetch is <1M rows. would like to migrate this to AWS. how do we do this? Would like to explore free credits service provider until we exhaust and explore. Intent to maintain a centralised database and running queries.

Migrating database from local machine to Cloud

Business Analyst

Before you go about how "stack isn't important, skill is", the purpose of this post to understand what the current market trend is for backend. 
A bit about me, I'm a backend dev with experience in Node.js, MySQL and MongoDB. Around 4 YoE. 
My goal is to work in a company that serves customers at scale, and to learn how to engineer at scale. If you work in such a company / know someone who does, what stacks are you/they using and how are they using it? I'm seeking answers in terms of language, DB, cloud providers, etc.

Backend stack to prepare for

Should I learn Sprig Data JPA? Is this still relevant? What are the latest DB technologies I should learn as a Software Developer?

Spring Data JPA, still relevant?

Personally for me? Shifting from pandas to polars for data related tasks changed my life. Code remained similar in syntax but the systems became more performant at extracting efficiency out of the same machine. 

Using connectorx for interfacing with the postgres servers was also a gamechanging moment instead of the OG pd.read_sql_query.

Across both of these minor changes, on moderate to large datasets, we achieved a near 10x improvement in querying data + doing operations on them.

As a Software Engineer, which library or framework was a gamechanger for you?

Robotics Engineer

Welcome to Indian Startups's Private Community

Would love to know more about what's happening in different companies like Zoho. Seeing a few opportunities in the space esp with AI integrations. 

What scale are you operating at? What are the challenges you're facing?

HR and HRtech/SaaS companies in the house?

Product Owner

Hi everyone,

Can anyone guide me about OIC middleware technology? Not sure what is the future..

OIC - Oracle Integration Cloud

Hi everyone, 
I'm  having 2.4 years experience  of Oracle application DBA as well as PLSQL Developer  at INFOSYS . I'm currently looking for a  new opportunity to advance my caree. 

If you know of any openings or can refer me , please please please get in touch.

 Thank you🙏!

Having around 2.4 years experience with infosys. Need to switch.

Database Administrator

I've been an Oracle Database Administrator handling databases and exadata, data guard, performance tuning (not so great at it), little bit of Oracle Cloud Infrastructure as well and typical DBA tasks including tablespace reorganization, migrations, etc. for the last 3 years. 

Now I'm thinking about transitioning into a Data Engineer/Data Analyst roles as it seems it is having a more lucrative future ahead.

I'm good at doing DBA things but I'm doubtful about this considering the future scope of this as there isn't much a DBA has to do in databases as most of the work is handled by Database itself but I have an interest in it and getting calls from other companies as well for the same role.

Is DBA a really good career option considering the future and the compensation?

Or exploring or gaining skills in data engineering/analyst has more benefit?

Also, what would be the best among Data Engineer/Data Analyst/AI,ML Engineer?

Need advice on Career Transition

Associate Consultant

Welcome to Product Managers's Private Community

One who is entering into Product role in B2B company. What should they be focusing on initial days to get well versed with the Product and what's is the scope of PMs in India in this domain for next decade?

Size of database and choice of databse

Discover More

Software Engineers on

by Dezi Lee

Stealth

Suggestions for tackling scaling issues

Software Engineers on

by Isaiah Carmden

Oracle

What are distributed systems in Job context?

IT Company Discussion on

by Kendall Lee

Stealth

Elastic search index

Office Gossip on

by Karilyn Dean

Stealth

Rant of a Data Analyst

Software Engineers on

by Jordon Hyrum

Fintech Startup

Interview: In 15m design a 100k rps platform

Software Engineers on

by Blair Denver

Piqyu

Migrating database from local machine to Cloud

Software Engineers on

by Aaron Gabriel

Stealth

Backend stack to prepare for

Software Engineers on

by Anise Carmden

Stealth

Spring Data JPA, still relevant?

Software Engineers on

by Kendall Carmden

Stealth

As a Software Engineer, which library or framework was a gamechanger for you?

Indian Startups on

by Jordon Carmden

Stealth

HR and HRtech/SaaS companies in the house?

Software Engineers on

by Kendall Gabriel

Capgemini

OIC - Oracle Integration Cloud

IT Company Discussion on

by Isaiah Hyrum

Infosys

Having around 2.4 years experience with infosys. Need to switch.

IT Company Discussion on

by Dezi Carmden

Infosys

Need advice on Career Transition

Product Managers on

by Coy Taye

Stealth

Transitiining to PM -B2B SAAS