Resilient Engineering: Designing Your System to Survive Pressure

So how do we work around this cognitive limitation to design and stably operate a system full of such complexity? To ensure PandaDoc's components are resistant to unexpected issues such as load spikes and infrastructure failures, we use a set of engineering practices and design patterns called “resilient engineering.” This article describes some of them on the entire scale of software design and engineering.

Explicit limits in SQL queries

How much harm an SQL query like this can do?

SELECT id, name, date FROM tbl WHERE parent_id = 5;

The answer here depends on the amount of data within this table, in addition to the type of hardware running the database and its load. This query may run perfectly fine on your development laptop, within your staging environment, or even within your test suite, where the dataset is small.

But in an actual production environment where the table may contain millions of rows that match the “WHERE” condition, this query may return millions of rows. Simply pulling them from the disk and putting them into the database cache can be challenging enough. Transferring them over the network into your application process, then placing them into memory can also lead to memory overflow. Each of these actions present risks that could adversely affect your application and disrupt your service.

To make this query safe, make one small change:

SELECT id, name, date FROM tbl WHERE parent_id = 5 LIMIT 10;

Adding this “LIMIT” clause places an upper limit on the number of rows the database will look for and return. So in this example, no more than 10 rows can be returned. The exact number of the row limit depends on the use case at hand. For example, 10 may be a reasonable number if your interface displays 10 items per page.

Sometimes people even add a "LIMIT 1" clause to queries that are expected to return just one row by primary key, but you could call this a bit paranoid and not go so far. On the other hand, if you imagine that your database could someday start using another column for the primary key, it might be better to be safe than sorry.

Another idea that takes this practice even further is to add an automated check that will write a log notice when you run a query without an explicit “LIMIT” clause. This way, you'll be sure you won't miss it.

Graceful degradation

When you work with a distributed system, chances are high that one component will fail, due simply to the number of separate components. At PandaDoc, we acknowledge this by planning for it. Designing communication between components should take failure modes into account.

So, how should a system behave if a component fails?

The “graceful degradation” approach advocates for being prepared for a major component’s failure, then having a fallback plan in place that will allow users to continue their work, even with limited functionality.

One example at PandaDoc is our Add-On Store, which allows users to manage their own add-ons. This service stores the state of all enabled add-ons across accounts and users. If the Add-On Store goes down, users will be unable to manage their add-ons. However, our main application will remain intact because we store a cached version of the enabled add-ons’ state outside the service.

Another example is how we handle document editor components such as the quote block. In case our quote service becomes non-responsive, our document editor will render a placeholder while keeping all other blocks available for editing.