Year by year I have noticed a constant decline in database skills amongst developers. 20 years ago, database design and implementation could consume 60% of the total development time spent in a software engineering project. Why? because effective data storage and retrieval is a huge and fundamental part of data-driven applications.

This seems like news nowadays. Many developers use the database as a simple bucket. Trust me, it's not a bucket. You can treat it as such, but you won't end up with a good product.

Object-relational mapping

I think it all started with the rise of ORM frameworks. Data access on the client-side is tedious and time-consuming and the initial motivation was to speed up the process. In the end, it moved developers away from the fundamentals. SQL is now generated and skills in how to optimize database querying were lost. What we got was generated generalized sub-optimal generated SQL. In those days I often found myself spending time on "cheating" the framework a lot to do what I want. In short, abusing the high-level ORM QL to do what I want on the SQL level.

Another thing that happened was that knowledge of database integrity constraints was lost, leading to more cases of data corruption. Indexing know-how was also lost. I'm pretty certain most developers today can't even explain how a b-tree works.

NoSQL

When NoSQL databases started to appear, many ran to them without a real plan with the main motivation to quickly solve performance issues, making database access easier and lessen the burden on the developers. NoSQL databases have its uses but it is not a replacement for relational databases. As they became more widely used the knowledge of ACID(Atomic, Consistency, Integrity, Durability) also began to fade.

DevOps

The most recent major factor playing a part in the problem is DevOps. Why not let the developers just do it all? Who needs specialized roles like DBA and Ops experts. We'll just run it all in the cloud with auto-scaling virtualization software!

Conclusion

To sum up, ORM, NoSQL, and DevOps are all progress. They are tools but it does not replace the need for a fundamental understanding of what you are doing. The ever-growing demand for quick fixes is the essence of the problem.

I will leave you with this amazingly well thought out database design by the engineering team behind Instagram. There are links in the post to their approach to scaling etc. It's just a great example of a cool design with the fundamentals always present and written in 2012 but it still holds.

That's my CTO rant of the day 😂 I hope it sparked some thought. Feel free to disagree in the comments.

About the top image: Yeah that's my ragdoll Idefix. The confused look is due to me putting him in a bucket 😊