Maintainability, security, cost

Product maintenance spans from launch to end-of-life. It includes identifying bugs, fixing them, adding features, improving performance, expanding test coverage, and writing documentation.

Companies that prioritize maintainability conduct an operational readiness review before launch. This process draws from BFSA recommendations and builds a list of best practices for operational excellence.

The goal is to educate teams and uncover operational gaps in the system. Only after resolving all major issues is the product considered launch-ready.

The core challenge is integrating operational practices into the system design phase—rather than treating them as post-launch tasks.

During interviews, expect questions on these maintainability aspects:

• Failure modes and mitigations: What happens when a component fails? How does the system respond?
• Network partition handling: How does the system manage and recover from network partitions?
• Monitoring: How is system health tracked? How do we detect specific failures?
• Testing: How are individual components tested? How are end-to-end data flows validated?
• Deployment: How are regular changes deployed safely? How quickly can bad changes be rolled back?

This list is not exhaustive. But mastering these four areas enables meaningful discussions with engineering teams.

Each topic is vast—some could fill entire books. A pragmatic approach is needed.

In this course, we build a reference architecture for a distributed messaging system. We then apply the four maintainability topics to this real-world system.

For each topic, we focus on concepts commonly used in practice.

Secure systems rely on three fundamental attributes: confidentiality, integrity, and availability.

• Confidentiality: Data is protected from unauthorized access.
• Integrity: Data is not corrupted or lost; only authorized users can modify it.
• Availability: Authorized users can access resources when needed.

In practice, this means the system must defend against unauthorized access and security attacks.

Key security areas include:

• Identity and permissions management: Who can access the system? Who can access what? How are authentication and authorization implemented?
• Infrastructure protection: Is the system resilient to DDoS attacks? How is it protected from SQL injection or cross-site scripting?
• Use of security tools: Should a web application firewall or API gateway be deployed?

Data must be protected in two states:

• At rest: When stored on disks or databases.
• In transit: When transmitted over networks.

System cost consists of three parts:

• Engineering cost: The effort required to design, implement, test, and deploy the system.
• Maintenance cost: The effort needed to maintain the system over time.

Cost estimation is essential during design. This ensures financial feasibility and long-term sustainability.

These components form the foundation of cost optimization in system design.

In a system design interview, we should not expect questions about engineering cost.

This is because estimating engineering cost requires a detailed breakdown of tasks.

Such a breakdown is time-consuming and demands deep system understanding.

We lack both time and system-level detail during interviews.

Thus, engineering cost is not typically a focus in interviews.

Maintenance cost may be discussed in interviews.

Typical questions involve reducing maintenance cost.

This usually means automation.

Examples include automating:

• Monitoring
• Testing
• Deployments
• Configuration management
• Security patches

These topics will be covered separately when discussing maintainability.

We should expect questions about resource cost.

Resource cost includes both hardware and software.

Hardware resources include:

• Physical machines
• Hardware load balancers
• Networking devices

These are required to run the system.

Interviewers may ask for capacity estimation.

This involves determining how many machines are needed to process data.

Or how many disks are needed to store data.

Once we have the number, we multiply it by the cost per machine.

This gives an approximate total hardware cost.

Software resources include all commercial software used in the system.

Today, most such software is provided by public cloud services.

It does not matter whether the software comes from a public cloud provider or an in-house team.

From a cost perspective, both are equivalent.

In both cases, we pay.

For cloud providers: real money.

For in-house solutions: virtual money.

This virtual money is still a form of cost.

It contributes to the total system cost.

Therefore, we must include it in our estimation.

When using managed services, three main parameters affect cost:

• Number of requests made to the service
• Number of bytes transferred (in or out)
• Number of bytes stored

We can express total resource cost with this formula:

Total resource cost = hardware cost + request cost + transfer cost + storage cost

This is not an exact formula.

If we consider costs per hour—for example, virtual machines or load balancers—we should include those in hardware cost.

We could also include license costs.

But my goal is to show how system design influences cost.

This formula, though incomplete, effectively illustrates that design decisions impact cost.

When increasing system availability, we add redundant hardware.

This raises hardware cost.

Total resource cost increases.

To improve durability, we replicate data.

This increases storage.

Storage cost goes up.

There are cost-reduction techniques.

Elasticity allows releasing resources when not needed.

This reduces hardware cost.

We can use request batching and load polling to reduce the number of requests.

This lowers request cost.

These techniques directly affect total resource cost.

Thus, design choices directly influence cost efficiency.

Each decision—availability, durability, scalability—has a measurable cost impact.

Design must balance performance with financial constraints.

Cost is not a secondary concern.

It is a core outcome of system design decisions.

Data compression decreases the number of bytes transferred over the network, directly reducing bandwidth usage and improving response times.

To reduce storage costs, implement two-tiered storage:

• Hot storage: Holds frequently accessed data requiring fast retrieval.
  • Example: Use DynamoDB for low-latency access.
• Cold storage: Stores infrequently accessed or archived data with no need for fast access.
  • Example: Use S3 for long-term, infrequently accessed data.

Cold storage solutions are typically less expensive than hot storage.

Apply a pragmatic approach to cost optimization using real-world system designs.

Include back-of-the-envelope capacity estimation techniques.

Maintainability, security, and cost optimization rely on practical knowledge.

These include:

• Familiarity with proven design patterns
• Understanding of optimization techniques
• Experience with real system trade-offs

In interviews, avoid reinventing solutions.

Pre-learn established practices and techniques.

When asked, retrieve specific knowledge directly from memory.

This approach ensures efficient, accurate responses without overcomplicating the design.