Skip to main content

Scalability And Performance Split Your Data and Simplify

Introduction

Here are a few guidelines for supporting scalability and performance in your systems.
  1. Simplify — Simplify your code and design, you will gain from it an easier to understand and a scalable system, your life will be scalable, the more complex it is the less it’s possible to scale it out and the more complex your life is. Of Course if its not possible to simplify do not we are sane people, but many times we only think its not possible to simplify while it’s possible, so do ourself a favour and put some effort on this.
2. X Axis Duplicate Data Create multiple read only db’s or clones for your data and thus scale your reads. You can then use a read query to read across multiple copies of your data thus less strain on your servers.
3. Y Axis Split on business your data like microservices also in db level not only in service level, different roles, different db’s. Do you sell both underwear and have another line of business for atomic energy manufacturing? what do you think of having 2 db’s or more, or just splitting your data you could still have a single db, just make sure you split things.
4. Z Axis Split on same - If you have multiple customers split on customer id, you can put smaller customers on same shard and larger one on different. Be sure you split on categories that makes sense for an even split and not for example on location, in this case if 80% of your customers are from a certain place you didn’t split ok.
5. 3 Data centers rules — If you had only 2 data centers you need 200% capacity if one goes down in order to serve 100% capacity, if you have 3 data centers you need total of 150% so two of them would make 100% capacity if one of them goes down.
6. Remember storage alternatives — Remember you have different storages such as file storage such as `ceph` (don’t forget the file system), nosql, wide column storage such as cassandra and relational. Column storage usually provide automatic row sharding and asynchronous replication with eventual consistency, column split requires more of manual intervention.
7. Consistency - If you increase consistency for example on nosql then operations such as `getSomething` would require to contact all nodes to make sure they return the recent and greatest version.
8. Firewalls are like locks — You lock your main door but you don’t lock internal doors is that right? Credit card request through lock but not image request. Don’t overuse your firewall, it’s complex enough without it.
9. Really need a transaction? When you pass money from one customer to another do you really need a transaction? Consider all options, usually when you start considering event sourcing you see you can compromise without transactions.
10. Dont read validate your write Have you just wrote something to disk/cache/db? don’t reread it in order to validate it your servers have more useful things to do.

Resources



Comments

Popular posts from this blog

API Design Paper Summary and Review

API Design Paper Summary Introduction Is building API a solvable question, how far can we get into having good API’s and what is a good API at all? these are all very hard questions, usually you know the answers once you designed multiple APIs and got experience and then reviewed the decisions you have taken. Fortunately there are papers dealing with this problem exactly, for complex API’s used by a huge amount of people, the Qt API for example a very populate framework for desktop GUI building, and today we are going to go through a summary of that paper.

“The Little Manual of API Design” is a very nice paper written by Jasmin Blanchette has released a paper while working in trolltech, which is a Nokia company. I found it to be very clear and concise, and reassuring what we think of API design. It’s a difficult task that includes both artistic, social, programming and scientific skills. We are going to summarize this paper for you.

When you write an API you combine a set of symb…

Dev OnCall Patterns

IntroductionBeing On-Call is not easy. So does writing software. Being On-Call is not just a magic solution, anyone who has been On-Call can tell you that, it's a stressful, you could be woken up at the middle of the night, and be undress stress, there are way's to mitigate that. White having software developers as On-Calls has its benefits, in order to preserve the benefits you should take special measurements in order to mitigate the stress and lack of sleep missing work-life balance that comes along with it. Many software developers can tell you that even if they were not being contacted the thought of being available 24/7 had its toll on them. But on the contrary a software developer who is an On-Call's gains many insights into troubleshooting, responsibility and deeper understanding of the code that he and his peers wrote. Being an On-Call all has become a natural part of software development. Please note I do not call software development software engineering because …

Recursion Trees Primer

Recursion trees.

Controlling the fundamentals stands at the cornerstone of controlling a topic.  In our case in order to be a good developer its not enough or even not at all important to control the latest Java/JavaScript/big data technology but what's really important is the basics.  And the basics in computer science are maths, stats, algorithms and computer structure.

Steve wosniak the co-founder of apple said the same, what gave him his relative advantage was his deep understanding of programming and computer structure, this is what gave him the ability to create computer's which are less costly than the competitors (not that there were many) and by the way there were 3 founders to apple company one responsible for the technical side, one for the product and sales (Steve Jobs) and the third responsible for the company structure and growth, each of the three extremely important, it was not only the two Steve's but that's a topic for another episode.

And with that l…