[Wallet Messenger Service] Architecture

This post was migrated from Tistory. You can find the original here.

Message service architecture

Current state

Current state

Thinking through the problems that would arise as traffic grows in the current structure:

1. Resource usage from socket connections plus heavy traffic slows down pub/sub.

2. Pub/sub speed is fine, but DB processing becomes the bottleneck.

What if we add more servers?

Adding more servers

So would scaling servers based on channels solve the problem?

That would resolve the socket connection resource issue tied to the number of channels,

but if the real cause of traffic is a huge volume of messages coming from a handful of specific channels, it still doesn’t solve anything.

Conclusion

Conclusion

So instead of partitioning by channel, let’s consider a round-robin load-balancing approach based on the user.

In this structure, we don’t know in advance which pub/sub a given user will be looking at. That means a pub on any single server has to be propagated to the pub of every server. Let’s consider Kafka for that propagation role.

Let’s walk through a scenario.

Both users are watching the same channel 1:
User A, assigned to pub/sub server 1
User B, assigned to pub/sub server 2

Here’s how a sub reaches User B when User A publishes:

1. User A publishes to server 1.

2. Server 1 produces the message to Kafka.

3. Every server (1 and 2) runs the same consumer logic, and each publishes the same message to its own local server.

4. As a result, User A receives the sub from server 1, and User B receives the sub from server 2.

Given the real-time performance requirements of the service, the transactional work of persisting messages is handled in a separate consumer.

As long as the message consumer retains the timestamp at which the pub arrived from the client, there’s no constraint on the order of insert operations.