How so you end up managing tracking event stream of customer data?

I work for a growing startup in Bangalore. The data team size is very small with no data lead. Not a single query we hit the DB gives any result. Owing to cost cutting my firm has cut down the instance size and they're planning to downgrade the DB too. We data analysts are relying on the data engineering team to fetch us results for our queries which takes a long time to get a single table. This is highly inefficient. The data size is growing at a massive scale however the analytics efforts are trickling down due to poor infrastructure. Is this happening in your data team too? How can I solve this problem given there are no experienced data leaders in our firm to voice out.

Hi community, Context: In my current company, we have a data-pipeline, which in short works like this: we get raw events from Kafka dumped in S3. We run a batch job (Airflow), this job essentially picks up the raw jsons in s3, enforces data parser logic (we have a service written in python where we explicitly define what attributed we want from raw json, these attributes are accordingly parsed), the parsed data is then converted to CSV/parquet formats and dumped in s3 in another folder and later loaded into tables, which is used for analytics, etc. Problem: Today for every new event we generate, we have to write a parser logic from scratch, if the event structure is different. In case of small changes we can update attributes we want to parse in code itself. But post that we have to deploy changes which takes time. Is there a smarter way of doing it? For example, having a UI interface, where we select the attributes from json (that could include nested attributes), and that is parsed and dumped in s3, later loading happens. And if we want to update parser, we can do so from UI itself than going into code updating things, deploying, etc. Do we have any open source alternatives here? Or any good engineering blogs which has covered such/similar scenario?

AMA. How does a day to day pan out for a data scientist, Who is data science not really for, Does FOMO really affect my career in DS

As the title says I want to learn how and which tools do product people in startups use to query data, make dashboards, do A/B testing and all that. I'm working at a startup where none of this is practiced, so I want to know how other folks do it as most startups in the growth stage and above do this and expect the candidate to be aware of the tools, setup etc.

How do you end up getting insights from your data? Do you’ve data analyst/ business analyst? or do you query data yourself? or do you use mixpanel/ or some product analytic tool which is properly configured?

I'm always doubtful, even after doing rigorous customer validation and analyzing user data. Happy to learn from folks here.

I’ve worked for different companies and my experience varied a lot. One my previous employer didn’t want to invest in mixpanel / product analytics tool and we ended up configuring each event manually and passed it on to redshift. Due to this we just had too much of dependency on analytics. Compare to it now I only take there help when I want to get some correlation or major exploration done. How has been your experience so far?

Share your progress - the journey. - Where did you begin and where are you now? - What motivated you to go the extra mile? - Where do you see yourself in the next few years? - What did you do differently then the rest of the pack? This can serve as a motivation to those of us who wants to improve and as a appreciation for some of us who made themselves a stellar career. Eager to hear your stories. 🔥

Everybody wants to join a good team where you have positivity and your learning is great. But after going through the interview, before joining a company, how would you know the team you are going to join is a good team in that company?