RFC(sharder): Scaling from the threads to the cores and beyond

[kyranet]

Preface: this RFC (Request For Comments) is meant to gather feedback and opinions for the feature set and some implementation details for @discordjs/sharder! While a lot of the things mentioned in this RFC will be opinionated from my side (like what I’d like to see implemented in the module), not everything might be added in the initial version or some extra things might be added as extras.

This RFC is split into several parts, described below. As an important note, through this entire specification, "shard" is not used to describe a gateway shard, those are handled by @discordjs/ws (see #8083). "Shard" is used to describe the client that the manager instantiates. Those could be (but are not limited to): workers, processes, or servers.

Before any work is done on this, I would like to ask them kindly to not take much if any inspiration from the current shard manager, and that this implementation needs to be as agnostic as possible so it can be used by bots made with discord.js's components (or even with older versions of discord.js).

Yes, I'm also aware this is over-engineered, but everything is designed with composability in mind, and has been thought with experience from implementing this in #7204. Truth be told, without some of those components, certain things would be significantly harder to implement.

---

ShardManager

Alternatively: Sharder, ProcessManager, ClientManager

This is the entry point in using this module. Its job is to handle the creation of shards, and is the central place where shards can intercommunicate. This manager also sets several parameters via environment variables (if available) to configure the clients.

As a result, a ShardManager has access to one or more ShardClients, and has a channel for each one of them.

API

// Error handling: TBD
// Lifecycle handlers: TBD

class ShardManager {
	public constructor(options: ShardManagerOptions);
}

interface ShardManagerOptions {
	strategy?: ChannelHandler | string;
}

Note: Shards will be configured in the strategy's options. The reasoning for this is simple, not all strategies have to be single-layer. There are also proxies which act as a second layer,

Strategies

This class also instantiates shards in one of the following four ways:

• Worker: using worker_threads, it has the benefit of faster IPC than process-based strategies.
• Fork: using child_process.fork, it's how the current shard manager creates the processes, and has the benefit that you can separate the manager from the worker.

  Note: If you open ports in the processes, you will have to open them at different ones, otherwise it will error.

• Cluster: using cluster.fork, unlike the fork mode, ports are shared and load-balanced.

  Note: You will have to handle the cluster-worker logic in the same file.

• Network: this is meant for cross-server sharding, and requires ShardManagerProxy. More information below.

Custom strategies will also be supported (although if you see something useful, please let us know and we'll consider it!).

Default: Fork?

Lifecycle Handlers

This class will emit events (not necessarily using EventEmitter) when:

• A shard has been created.
• A shard has been restarted.
• A shard has been destroyed.
• A shard has signalled a new status.
• A shard has signalled a "ready" status.
• A shard has signalled an "exit" status.
• A shard has signalled an "restarting" status.
• A shard has sent an invalid message. If unset, this will print the error to console.
• A shard has sent a ping.
• A shard has not sent a ping in the maximum time-frame specified. If unset, it will restart the shard.

Life checks

All of the manager's requests must automatically timeout within maximum ping time, so if the shards are configured to ping every 45 seconds, and have a maximum time of 60 seconds, then the request timeout must be 60 seconds. This timeout can also be configured globally (defaults to maximum ping time) and per-request (defaults to global value).

Similarly, if a shard hasn't sent a ping within the maximum time, the lifecycle handler is called.

---

ShardClient

Alternatively: ShardWorker, Worker, Client

This is a class that instantiates the channel with its respective ShardManager (or ShardManagerProxy) using the correct stream, and boils down to a wrapper of a duplex with extra management.

API

// Signals: TBD
// Error handling: TBD

class ShardClient {
	public constructor(options: ShardClientOptions);


	public static registerMessageHandler(name: string, op: () => MessageHandler): typeof ShardClient;
	public static registerMessageTransformer(name: string, op: () => MessageTransformer): typeof ShardClient;
}

interface ShardClientOptions {
	messageHandler?: MessageHandler | string;
	messageTransformer?: MessageTransformer | string;
}

Signals

• Starting: the shard is starting up and is not yet ready to respond to the manager's requests.
• Ready: the shard is ready and fully operative.
• Exit: the shard is shutting down and should not be restarted.
• Restarting: the shard is shutting down and should be restarted.

---

MessageHandler

Alternatively: MessageFormat, MessageBroker, MessageSerder

This is a class that defines how messages are serialized and deserialized.

API

type ChannelData = string | Buffer;

class MessageTransformer {
	public serialize(message: unknown, channel): Awaitable<ChannelData>;
	public deserialize(message: ChannelData, channel): Awaitable<unknown>;
}

Strategies

• JSON: data is serialized with JSON.parse and deserialized with JSON.stringify.
• V8: data is serialized with v8.serialize and deserialized with v8.deserialize.

Custom strategies will also be supported (custom format, ETF, YAML, TOML, XML, you name it).

Default: JSON?

---

MessageTransformer

Alternatively: MessageMiddleware, EdgeMiddleware, ChannelMiddleware

This is a class that defines what to do when reading a message, and what to do when writing one:

• When reading:

  graph LR
    A[Channel]
    A -->|Raw| B(MessageTransformer)
    B -->|Transformed| C(MessageHandler)
    B -->|Rejected| D[Channel]
    C -->|Deserialized| E[Application]
  

• When writing:

  graph LR
    A[Application]
    A -->|Raw output| B(MessageHandler)
    B -->|Serialized| C(MessageTransformer)
    C -->|Raw| D[Channel]
  

Note: This is optional but also composable (e.g. you can have two transformers) with insertion order of execution when writing (and inverse for reading), and may be used to transform data into a compressed, encrypted, decorated, or transformed in any way the developer desires them to be.

[Gzip, Aes256]

When writing, the data will go through Gzip and then to Aes256, while reading goes the opposite direction (Aes256, then Gzip).

API

type ChannelData = string | Buffer;

class MessageTransformer {
	public read(message: ChannelData, channel): Awaitable<ChannelData>;
	public write(message: ChannelData, channel): Awaitable<ChannelData>;
}

Strategies

It is unknown if we will ship any built-in strategy for encryption or transformation, since they're very application-defined, but we will at least provide two from Node.js:

• Gzip: uses zlib.gzip and zlib.gunzip.
• Brotli: uses zlib.brotliCompress and zlib.brotliDecompress.

Warning: We may change the functions used if we find a way to work with streams for better efficiency, but the algorithms may not change.

Custom strategies will also be supported (although if you see something useful, please let us know and we'll consider it!).

Default: []

---

ShardManagerProxy

This is a special class that operates as a shard for ShardManager and communicates to it via HTTPS, HTTP2, or HTTP3/QUIC, depending on the networking strategy used. Custom ones (such as gRPC) may be added. Encryption is recommended, so plain HTTP might not be available. There might also be a need to support SSH tunnels to bypass firewalls for greater security. A ShardManagerProxy is configured in a way similar to ShardManager, supporting all of its features, but also adds logic to communicate with the manager itself.

The proxy may also use a strategy so if a shard needs to send a message to another, which is available within the proxy, no request is made to the parent, otherwise it will send to the parent, which will try to find the shard among the different proxies.

---

Questions to answer

• Should we keep the names for the classes, or are any of the suggested alternative names better?
• Should we use Result<T, E> classes to avoid try/catch and enhance performance?
• Should we support async in MessageTransformer? Require streams only?
• Should we support have a raw data approach, or a command approach? If the latter, should we have eval?
  Raw data approach, commands can be implemented on a bot-level. Similarly to how Discord.js just emits events, people are free to build frameworks on top of @discordjs/sharder, so there's no need for us to force a command format, which would pose several limitations on how users can structure their payloads.
• Should messages be "repliable" (requires tracking as well as message tagging, etc)?
  • Always? Opt-in? Opt-out?
    Opt-in, not all broadcasts or requests need to be replied to.
  • Should we have timeouts?
    Yes, and tasks should be aborted and de-queued once the timeout is done.
  • Should we be able to cancel a request before/without timeout?
    I personally think this could be beneficial, but the implementation remains to be seen.
• What should be done when a message is sent from shard A to shard B, but B hasn't signalled "ready" (and is not "exit")? Should it wait for it to signal ready, or abort early? If the former, should we allow an option to "hint" how long a shard takes to ready?
• Is there a need for an error signal, or do we use exit for it?
• If a shard exits with process.exit() without signalling the manager (or proxy), should it try to restart indefinitely? Have a limit that resets on "restart"?
• If ShardManager dies, what should the shards do?
• If a ShardManagerProxy goes offline, what should its shards do? What should the manager do? Load-balance to other proxies?
• If a ShardManagerProxy is offline at spawn, what should the manager do? Load-balance to other proxies?
• If a ShardManagerProxy goes back online, should the manager load-balance its shards back to it?
• NEW: Should the HTTP-based channel strategies be pooling or open on demand?
  The WebSocket strategy obviously needs to be open at all times, and keeping a connection open makes it easier for messages to be sent in both ways, as well as they reduce latency since they need to do fewer network hops as they don't require a handshake on every request, but it'll also require an advanced system to properly read messages, and will also make aborting requests harder.
• NEW: Should different ShardManagerProxys be able to communicate directly to each other, similar to P2P? Or should it be centralized, requiring the ShardManager?
  I personally think there isn't really much a need for this, a centralized cache database (such as Redis) could store the information, same for RabbitMQ or alternatives. Is there a need for each proxy to know each other? One pro of having this is that it makes proxies more resilient to outages from the manager, but one downside is that it would need to send a lot of data between proxies and the manager to keep all the information synchronized. I believe it's best to leave this job to Redis and let the user decide how it should be used. After all, proxies are very configurable.
• NEW: Should we allow ShardManagerProxy to have multiple managers, or stick to a single one for simplicity?
  This way, managers can be mirrored, so if the main one is down, the proxies connect to a secondary one. How the data is shared between managers is up to the user, we will try to give the hooks required for the storage to be easy to store and persist.
• NEW: What should ShardManagerProxy if all the managers are offline? Should they just stay up?
  Their capabilities may be limited, since they can't load-balance by themselves, and they may not be able to communicate to each other without P2P communication or a central message broker system, but most of the operations should, in theory, be able to stay up. I believe we can make the behaviour customizable by the developer, with the default of keeping the work and try to reconnect once the managers are up.

---

Anything else missing

If you think this RFC is missing any critical/crucial bits of information, let me know. I will add it to the main issue body with a bolded EDIT tag to ensure everyone sees it (if they don't want to check the edited history).

Changelog

• Added answers to questions 4 and 5.
• Added four more questions (13, 14, 15, 16), all related to ShardManagerProxy.

[vladfrangu]

• Should we use Result<T, E> classes to avoid try/catch and enhance performance?

Personally I think yes.

---

• Should we support async in MessageTransformer? Require streams only?

We should most definitely support async too just in case I'd say...wouldn't make sense to be streams only

---

Should we support have a raw data approach, or a command approach?

• If the latter, should we have eval?

I think this is very bot specific, and if we allow bots to communicate with ease when sharding we shouldn't provide such things like commands by default (broadcastEval in d.js can be implemented at the library level)

---

Should messages be "repliable" (requires tracking as well as message tagging, etc)?

• Always? Opt-in? Opt-out?
• Should we have timeouts?
• Should we be able to cancel a request before/without timeout?

Opt-in sounds the best (as not all broadcasts need to be replied to). We could support timeouts for acks I suppose. Cancelling doesn't make much sense since if you sent it to 5 shards out of 10 then cancel it, you'd need to instruct the 5 shards that received it already to not run what they most likely already ran.

---

What should be done when a message is sent from shard A to shard B, but B hasn't signalled "ready" (and is not "exit")? Should it wait for it to signal ready, or abort early?

• If the former, should we allow an option to "hint" how long a shard takes to ready?

We should probably just enqueue it with a timeout, at least that's what I'd do. Might be more problematic for messages that expect a reply...

---

• Is there a need for an error signal, or do we use exit for it?

Could you provide examples of where such a signal could be emitted? 👀

---

• If a shard exits with process.exit() without signalling the manager (or proxy), should it try to restart indefinitely? Have a limit that resets on "restart"?

We should make this configurable by the users I'd say. But then we should also have a method that will be called when the connection fails multiple times, so users can setup alerts for it.

---

• If ShardManager dies, what should the shards do?

I guess this question is mostly for users that will use the proxy strategy... I'd say we leave this configurable by the users, but also have that aforementioned alert method that will be called. Alternatively we could just turn off the shards since the manager is gone, but users should be made aware somehow as to why it'd happen.

---

• If a ShardManagerProxy goes offline, what should its shards do? What should the manager do? Load-balance to other proxies?

Tricky question...at that point there's no communication between the shards and their manager...I'd argue in that case we would probably support load-balancing multiple proxies. If all die well...refer to previous two questions.

---

• If a ShardManagerProxy is offline at spawn, what should the manager do? Load-balance to other proxies?

Yep, alternatively exit early.

---

• If a ShardManagerProxy goes back online, should the manager load-balance its shards back to it?

That would mean there is always a bi-directional channel of communication between proxy and manager, which wouldn't be fully efficient unless we spread messages across all proxies...unless we want to do it that way we can probably keep using the current proxy till it goes 💀 .

---

Just as a reminder, these are my thoughts about these problems. These answers aren't meant to be taken as the rule!

[kyranet]

Should we support have a raw data approach, or a command approach?

• If the latter, should we have eval?

I think this is very bot specific, and if we allow bots to communicate with ease when sharding we shouldn't provide such things like commands by default (broadcastEval in d.js can be implemented at the library level)

Raw data approach it is.

And I believe we shouldn't make discord.js force either approach, as it limits the message format users can use for their applications. This is a major pain point I have with the current implementation, its wasteful and inefficient approach makes it not suitable for large-scale applications.

Should messages be "repliable" (requires tracking as well as message tagging, etc)?

• Always? Opt-in? Opt-out?
• Should we have timeouts?
• Should we be able to cancel a request before/without timeout?

Opt-in sounds the best (as not all broadcasts need to be replied to). We could support timeouts for acks I suppose. Cancelling doesn't make much sense since if you sent it to 5 shards out of 10 then cancel it, you'd need to instruct the 5 shards that received it already to not run what they most likely already ran.

Not just timeouts for acks, but for all the calls. Cancelling a request early would basically signal the manager to remove the message from the queue before sending them, and there's a chance we could make shards construct an AbortController so it can be used for, say, aborting HTTP requests mid-way. Some network-based channel strategies could provide a much more efficient alternative, but that remains to be tested.

If a request has been aborted and a few shards have already finished processing it, we could either ignore the results, send the partial results (useful for collecting data even when there's an outage), or allow users to specify either (we could achieve this by sending an enum value next to the command).

What should be done when a message is sent from shard A to shard B, but B hasn't signalled "ready" (and is not "exit")? Should it wait for it to signal ready, or abort early?

• If the former, should we allow an option to "hint" how long a shard takes to ready?

We should probably just enqueue it with a timeout, at least that's what I'd do. Might be more problematic for messages that expect a reply...

Got it, do we also want to add a "hint" for how long shards take to signal ready? This has some use-cases. If the developer configures said hint to be 60 seconds, and shard B has been initializing for 45 seconds (15 seconds for estimated), and the message timeout is 5 seconds, then we could make it abort early since it's far from margin of error (configurable, let's say 10%, which would make 60s±6s).

Said hint can also be used to determine when a shard takes way too long to signal ready and emit a warning for the developer.

We could also determine this time by "benchmarking" how long shards take to hit ready, after all, the first shard will display a rough estimate of how long the other shards will take to start up, and they can fine-tune this value as more shards spawn.

• Is there a need for an error signal, or do we use exit for it?

Could you provide examples of where such a signal could be emitted? 👀

Process exits with non-zero error code before signalling "ready". Also in worker mode, this would be when the worker cannot initialize.

• If ShardManager dies, what should the shards do?

I guess this question is mostly for users that will use the proxy strategy... I'd say we leave this configurable by the users, but also have that aforementioned alert method that will be called. Alternatively we could just turn off the shards since the manager is gone, but users should be made aware somehow as to why it'd happen.

Non-proxy strategies also would encounter this issue, specifically in fork and cluster modes.

I also want to provide a sensible default for this, since it can get complicated real quick, but we can ship without any and then implement one once real-life implementations come out.

• If a ShardManagerProxy goes offline, what should its shards do? What should the manager do? Load-balance to other proxies?

Tricky question...at that point there's no communication between the shards and their manager...I'd argue in that case we would probably support load-balancing multiple proxies. If all die well...refer to previous two questions.

My idea was that each proxy would be able to setup maximum resource usage, which boils down to how many shards they can manage. If load-balancing would result on all proxies to max out, then the missing shards would be queued for spawning once the outage is resolved and proxies start reconnecting to the manager. This would also apply for next question as well.

• If a ShardManagerProxy goes back online, should the manager load-balance its shards back to it?

That would mean there is always a bi-directional channel of communication between proxy and manager, which wouldn't be fully efficient unless we spread messages across all proxies...unless we want to do it that way we can probably keep using the current proxy till it goes 💀 .

Proxies and managers have bi-directional channel of communication, HTTP2, HTTP3/QUIC, gRPC (powered by HTTP2), and many other strategies support this kind of thing. HTTPS (1.1) also supports this with long-polling requests, and there's WS too, so yeah, it's definitely doable.

Nevertheless, don't forget that proxies don't communicate with the manager unless they have to, since they will try to always work with their local group as much as possible. Communication with the manager should be kept to load-balancing and status management.

At this point, developers should build a system that allows them to send messages between all processes by using a specialised message broker such as RabbitMQ. Spamming messages thru the network will only make load-balancing a lot slower.

[RealAlphabet]

• Should we use Result<T, E> classes to avoid try/catch and enhance performance?

Maybe I'm missing something here, but what do you mean by Result<T, E> ?

• If ShardManager dies, what should the shards do?

Don't die. If the manager dies, the children also die for worker_threads. It seems to me that it is the same for clusters given
that it is not possible to specify if the child processes will be detached from the parent process (to be checked).

Maybe I misunderstood but if a manager dies there will only be the other end of ShardManagerProxy that will have to wait for the manager to reconnect.

[didinele]

Maybe I'm missing something here, but what do you mean by Result<T, E>?

This refers to Rust's way of doing error handling, see: https://doc.rust-lang.org/std/result/

Some folks here have made a JS port of this API: https://www.npmjs.com/package/@sapphire/result

[SpaceEEC]

A bit of input from me:

• Should we keep the names for the classes, or are any of the suggested alternative names better?
  I'm fine with those names.
• Should we use Result<T, E> classes to avoid try/catch and enhance performance?
  I'm not sure if there is much gained by doing so, I lean towards no.
• Should we support async in MessageTransformer? Require streams only?
  Yes, supporting it sounds good.
  Streams are nice, but I don't think it would be too useful for us here.
• Should we support have a raw data approach, or a command approach? If the latter, should we have eval?
  If I understand this right: Raw data approach sounds better to me.
  discord.js could offer something (optional?) built-in, if we want to provide something developers can just use.
  I'd be against anything built-in eval either way.
• Should messages be "repliable" (requires tracking as well as message tagging, etc)?
  • Always? Opt-in? Opt-out?
    Opt-in
  • Should we have timeouts?
    Yes, avoid leaking requests if they never finish
  • Should we be able to cancel a request before/without timeout?
    This would be a "nice to have", but I doubt it's feasible to implement generic enough.
• What should be done when a message is sent from shard A to shard B, but B hasn't signalled "ready" (and is not "exit")? Should it wait for it to signal ready, or abort early? If the former, should we allow an option to "hint" how long a shard takes to ready?
  Hard to say, enqueueing it (for a limited amount of time) is probably better developer experience.
  Aborting early is probably less problematic, as there is less chance of overloading the shard, which could have been the cause of the shard to be in this not yet ready state in the first place.
• Is there a need for an error signal, or do we use exit for it?
  With the answer below, maybe not.
• If a shard exits with process.exit() without signalling the manager (or proxy), should it try to restart indefinitely? Have a limit that resets on "restart"?
  We could mimic how Erlang supervisors work. A not exacly directlink to the relevant documentation would be the Supervision Principles and then its subheading "Restart intensity and period", which I unfortunately can't directly link to.
• If ShardManager dies, what should the shards do?
  Die.
• If a ShardManagerProxy goes offline, what should its shards do? What should the manager do? Load-balance to other proxies?
  What does "offline" mean here? Dead? No communication with the manager possible? No communication from the manager to the proxy?
  This is a distributed programming question however, which is a bit out of my comfort zone either way.
• If a ShardManagerProxy is offline at spawn, what should the manager do? Load-balance to other proxies?
  Same as the answer above.
• If a ShardManagerProxy goes back online, should the manager load-balance its shards back to it?
  Same as the answer above.
• Should the HTTP-based channel strategies be pooling (polling?) or open on demand (pushing?)?
  Why would WebSockets require an advanced strategy to properly read messages and make aborting requests harder?
  I don't think I understand the complications enough, open on demand generally sounds better however.
• Should different ShardManagerProxys be able to communicate directly to each other, similar to P2P? Or should it be centralized, requiring the ShardManager?
  I vote "no". I don't think we want to implement anything distributed computing, as those things tend to get a bit more complex than one would like to. (Hello Paxos)
• What should ShardManagerProxy if all the managers are offline (do?)? Should they just stay up?
  All? I thought it's only communicating with just one.

[kyranet]

• What should be done when a message is sent from shard A to shard B, but B hasn't signalled "ready" (and is not "exit")? Should it wait for it to signal ready, or abort early? If the former, should we allow an option to "hint" how long a shard takes to ready?
  Hard to say, enqueueing it (for a limited amount of time) is probably better developer experience.
  Aborting early is probably less problematic, as there is less chance of overloading the shard, which could have been the cause of the shard to be in this not yet ready state in the first place.

The limited amount of time would be what the "hint" would do, it's basically a configurable timeout. If the messages are configured to take a reply within 5 seconds, and the shard takes 10 seconds and has just started, it won't enqueue unless it's been around 6 seconds (4 seconds remaining, gives 1 second before cancel).

• If a shard exits with process.exit() without signalling the manager (or proxy), should it try to restart indefinitely? Have a limit that resets on "restart"?
  We could mimic how Erlang supervisors work. A not exacly directlink to the relevant documentation would be the Supervision Principles and then its subheading "Restart intensity and period", which I unfortunately can't directly link to.

I appreciate that link, I'll have to take a look into it and update the RFC to implement the new information.

• If a ShardManagerProxy goes offline, what should its shards do? What should the manager do? Load-balance to other proxies?
  What does "offline" mean here? Dead? No communication with the manager possible? No communication from the manager to the proxy?
  This is a distributed programming question however, which is a bit out of my comfort zone either way.

Generally, anything that determines an outage: it's either not running or not accessible (the host cannot connect to the network, the port is not longer accessible, etc).

I understand that distributed programming can be out of your (and even mine) comfort zone — I myself have never used more than 1 machine to host a service, so I don't really have experience with horizontal scaling¹ and redundancy systems.

• Should the HTTP-based channel strategies be pooling (polling?) or open on demand (pushing?)?
  Why would WebSockets require an advanced strategy to properly read messages and make aborting requests harder?
  I don't think I understand the complications enough, open on demand generally sounds better however.

Seems I mixed up a few things while writing the paragraph, my apologies. WebSocket is just one of several strategies (the easiest but also the most limited and the least efficient). The paragraph was intended more for the raw approaches, doing HTTP calls (with support for HTTPS, HTTP2, and maybe QUIC in the future).

• Should different ShardManagerProxys be able to communicate directly to each other, similar to P2P? Or should it be centralized, requiring the ShardManager?
  I vote "no". I don't think we want to implement anything distributed computing, as those things tend to get a bit more complex than one would like to. (Hello Paxos²)

Noted, I was leaning towards telling users to use services such as Redis streams³ or RabbitMQ to distribute messages across shards more efficiently. This would also let us simplify the sharder by not reinventing the wheel.

Generally speaking, while the sharder supports message broadcasting and sending, developers should know that they should keep their messages to a minimum, as it can lead to message jamming, which can lead to the sharder's operations running slower due to the jamming's added latency.

• What should ShardManagerProxy if all the managers are offline (do?)? Should they just stay up?
  All? I thought it's only communicating with just one.

Related to the previous point in the issue's body, proxies would connect to several different managers, so if the manager that's managing them dies, another one takes over. It's related to redundancy. We can also make proxies connect to a single manager and tell users that for multi-manager setups, they should use a shared cache (e.g. Redis) so all managers share the state data.

Footnotes

1. https://en.wikipedia.org/wiki/Scalability#Horizontal_or_scale_out ↩

2. https://en.wikipedia.org/wiki/Paxos_(computer_science) ↩

3. https://redis.io/docs/data-types/streams/ ↩

[nc0fr]

Since you are planning on using core, threads etc, would it be plausible to add identification attributes to worker instances (PID, thread naming, ...) that could easily help end-users in observability of their clusters (not in the Node.js way).

As for questions:

• Should we use Result<T, E> classes to avoid try/catch and enhance performance?

Result does not seems very idiomatic to TypeScript and the Node community, and might be confusing for new contributors.

• Should we support have a raw data approach, or a command approach? If the latter, should we have eval?

The command approach seems better. Structured data sent across processes is a standard practice to provide thread-safety (channel-like communication) and make development easier (typed structures). An event pattern (rather than command or query) is the solution to use to provide loose coupling of workers and independent control over them (which is most likely to be a requirement for downstream frameworks).

• Should messages be "repliable" (requires tracking as well as message tagging, etc)?

They should be by default, letting downstream frameworks deciding if the reply is mandatory. The coupling of workers is a complex topic that should be left to downstream users as operators.

• What should be done when a message is sent from shard A to shard B, but B hasn't signalled "ready" (and is not "exit")? Should it wait for it to signal ready, or abort early? If the former, should we allow an option to "hint" how long a shard takes to ready?

At first it may be better to simply drop it and document the behaviour. In general, tight coupling requires both endpoints to be available and getting an inbox pattern (à la Erlang Virtual Machine's Actors) seems not to be a solution right here.

• If a shard exits with process.exit() without signalling the manager (or proxy), should it try to restart indefinitely? Have a limit that resets on "restart"?

ShardManagers should be started with some kind of policy/strategy, similarly to Erlang/OTP's Supervisor. Basically, it is a configuration set by downstream that declares the expected behaviour of the orchestrator (here it would be ShardManager) when one of it supervised child (here a ShardClient) fails.
The policies should however be well documented as they critically impact the system (infinite restart creates overhead since creating processes and OS thread is pretty extensive).

• If a ShardManagerProxy goes offline, what should its shards do? What should the manager do? Load-balance to other proxies?

This question is also downstream specific (CAP theorem), however to be opinionated, if a Parent dies, Children should safely exit ASAP since they often can not be saved.
Children should abort on network failure anyway, and worker rescue should not be a solution due to the statefulness of workers (shards).

However, it may be a better solution to also implements a strategy/policy pattern here and document the behaviour of each policies, since it is mostly a downstream problem.
For instance in Kubernetes, when the Parent (kubeapi) dies, children (kubelets) stay alive until they are required to interact with the Parent (which might have restarted before).
However this pattern is not recommended IMHO since Shards are stateful.

• If a ShardManagerProxy goes offline, what should its shards do? What should the manager do? Load-balance to other proxies?

IMHO, shards should immediately exit to avoid undefined behaviours since downstream may have rescheduled a proxy for clean shards.

• If a ShardManagerProxy is offline at spawn, what should the manager do? Load-balance to other proxies?

This is complex.
The easiest and safest solution in the majority of cases would be to let the manager accept more loads while other proxies are being rescheduled.
The manager should crash in case there is too much load.

• If a ShardManagerProxy goes back online, should the manager load-balance its shards back to it?

Ideally yes, as shards should be loosely dependant on the proxy.

• NEW: Should we allow ShardManagerProxy to have multiple managers, or stick to a single one for simplicity?

They should have multiple managers, for retry-policies.

• NEW: What should ShardManagerProxy if all the managers are offline? Should they just stay up?

Since proxies also serve as load-balancers for managers, I think it is better to let them scale down to zero yes. If all the managers failed, it is because there is a problem on downstream's cluster.

[nc0fr]

Yes, I'm also aware this is over-engineered, but everything is designed with composability in mind, and has been thought with experience from implementing this in #7204. Truth be told, without some of those components, certain things would be significantly harder to implement.

It is not that over engineered, I am very happy to see improvement in the scalability of the module as it is one of the most used (if not the most) Discord wrapper in the open-source community.

However the design seems to have some flaws, with each entity being too tied to others.
Is it because I am thinking at a bigger scale?

Since you are searching for composability, I am not sure about the goal of the module, wouldn't it be better to go to larger scale and adopts decomposition similarly to projets such as Nova or Nostrum with the Erlang Virtual Machine and thus using the Discord.js library as an abstraction between the scaling infrastructure and the business logic written by downstream.

I think the most confusing part of your proposal is the ShardManagerProxy, which looks like some sort of load-balancing where it is not required (thread-scale does not require that kind of arsenal).
You could also benefits from proposing a supercluster framework which manages clusters of shards (shards being here Gateway Shards)