Skip to content

Commit

Permalink
add Sharding (#64)
Browse files Browse the repository at this point in the history
implements #26

update dasgo
fix missing GUILD_MEMBERS_CHUNK intents
fix static check lint issues

* edit concept information
* fix enable intent functions
* refactor shardmanager interface
* implement SessionManager
* update Gateway Rate Limit Handling
* add Gateway tests for Session Manager, Rate Limits
* remove TestGatewayGlobalRateLimit test
* implement Shard Manager

implement InstanceShardManager

* add Shard integration test workflow
* generate Send Event Shard Manager functions
* update bundle
* fix shard test workflow
* fix data race in test
* remove app ID write on ready event

fix data race
  • Loading branch information
switchupcb committed Jul 8, 2023
1 parent 22302a7 commit 3da8b14
Show file tree
Hide file tree
Showing 45 changed files with 1,683 additions and 403 deletions.
23 changes: 22 additions & 1 deletion .github/workflows/shard.yml
Original file line number Diff line number Diff line change
Expand Up @@ -28,4 +28,25 @@ jobs:
uses: golangci/golangci-lint-action@v3
with:
version: v1.53.3
args: ./shard/...
args: ./shard/...

test-integration:
needs: test-unit
name: Integration Tests
environment: testing
runs-on: ubuntu-latest
steps:
- name: Checkout repository code
uses: actions/checkout@v3
- name: Set up Go
uses: actions/setup-go@v4
with:
go-version-file: ./shard/go.mod
- name: Run Integration Tests
env:
TOKEN: ${{ secrets.TOKEN }}
APPID: ${{ secrets.APPID }}
COVERAGE_TEST_TOKEN: ${{ secrets.COVERAGE_TEST_TOKEN }}
COVERAGE_TEST_GUILD: ${{ secrets.COVERAGE_TEST_GUILD }}
COVERAGE_TEST_CATEGORY: ${{ secrets.COVERAGE_TEST_CATEGORY }}
run: go test ./shard/tests/integration -race
84 changes: 53 additions & 31 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,13 @@ _This repository is STABLE. For more information, read the [roadmap](/_contribut

## A Next Generation Discord API Wrapper

High quality code merits easy development. Disgo uses developer operations to stay up-to-date with the ever-changing Discord API. Code generation is used to provide a clean implementation for every request and event. Data race detection is used with _an integration test that covers the entire Discord API_ in order to ensure that Disgo is safe for concurrent usage. In addition, **Disgo provides the following exclusive features**.
High-quality code merits easy development.

Disgo uses developer operations to stay up-to-date with the ever-changing Discord API.
- Code generation provides a clean implementation for every request and event.
- Data race detection is run on an integration test _that covers the entire Discord API_ to ensure that Disgo is safe for concurrent usage.

In addition, **Disgo provides the following exclusive features**.

- [EVERY Rate Limit (Global, Per Route, Per Resource, Custom, Gateway)](_contribution/concepts/REQUESTS.md#what-is-a-rate-limit)
- [Automatic Gateway Intent Calculation](_contribution/concepts/EVENTS.md#what-is-a-gateway-intent)
Expand All @@ -30,42 +36,52 @@ _Disgo uses [NO reflection or type assertion](_contribution/concepts/EVENTS.md#h

This breakdown provides you with a **full understanding** on how to use the API.

| Abstraction | Usecase | Example |
| :----------- | :------------------------------------------------------------------------------------------------------------------------------------------------ | :------------------------------------------------------------------ |
| **Resource** | A [Discord API Resource](https://discord.com/developers/docs/resources/application). | Guild Object. User Object. |
| **Event** | A [Discord API Event](https://discord.com/developers/docs/topics/gateway#commands-and-events-gateway-events). | A message is created. A user joins a channel. |
| **Client** | The Discord Bot [Application](https://discord.com/developers/docs/resources/application) that you program. One Bot = One Client. | Configure the bot settings. Set the token. |
| **Request** | Uses the Discord HTTP REST API to make one-time _requests_ for information. | Create an application command. Request guild information. |
| **Session** | Uses a Discord WebSocket Connection [(Gateway)](https://discord.com/developers/docs/topics/gateway) to receive _events_ that contain information. | Send a message when a command used or a user joins a voice channel. |
| Abstraction | Usecase | Example |
| :----------- | :------------------------------------------------------------------------------------------------------------------------------------------------ | :---------------------------------------------------------------------------------- |
| **Resource** | A [Discord API Resource](https://discord.com/developers/docs/resources/application). | Guild Object. <br> User Object. |
| **Event** | A [Discord API Event](https://discord.com/developers/docs/topics/gateway#commands-and-events-gateway-events). | A message is created. <br> A user joins a channel. |
| **Client** | The Discord Bot [Application](https://discord.com/developers/docs/resources/application) that you develop. <br><br> Bot = Client = Application. | Configure the bot settings. <br> <br> Set the token. |
| **Request** | Uses the Discord HTTP REST API to make one-time _requests_ for information. | Create an application command. <br> Request guild information. |
| **Session** | Uses a Discord WebSocket Connection [(Gateway)](https://discord.com/developers/docs/topics/gateway) to receive _events_ that contain information. | Send a message when an application command is used or a user joins a voice channel. |

You create a **Client** that calls for **Resources** using **Requests** and handles **Events** from **Sessions** using event handlers. For more information, please read [What is a Request?](/_contribution/concepts/REQUESTS.md) and [What is an Event?](/_contribution/concepts/EVENTS.md)
You create a **Client** that calls for **Resources** using **Requests** and handles **Events** from **Sessions** using event handlers.

_For more information, please read [What is a Request?](/_contribution/concepts/REQUESTS.md) and [What is an Event?](/_contribution/concepts/EVENTS.md)._

### Flags

A flag is a [flag](https://discord.com/developers/docs/resources/application#application-object-application-flags), [type](https://discord.com/developers/docs/resources/channel#embed-object-embed-types), [key](https://discord.com/developers/docs/resources/audit-log#audit-log-change-object-audit-log-change-key), [level](https://discord.com/developers/docs/resources/guild#guild-object-verification-level) or any other option that Discord provides. All flags are denoted by `Flag` in disgo: For example, `disgo.FlagUserSTAFF`, `disgo.FlagVerificationLevelHIGH`, `disgo.FlagPremiumTierNONE`, etc.
A flag is a [flag](https://discord.com/developers/docs/resources/application#application-object-application-flags), [type](https://discord.com/developers/docs/resources/channel#embed-object-embed-types), [key](https://discord.com/developers/docs/resources/audit-log#audit-log-change-object-audit-log-change-key), [level](https://discord.com/developers/docs/resources/guild#guild-object-verification-level) or any other option that Discord provides. All flags are denoted by `disgo.Flag` _(e.g., `disgo.FlagUserSTAFF`, `disgo.FlagVerificationLevelHIGH`, `disgo.FlagPremiumTierNONE`)_.

### Logging

Read [What is a Log](/_contribution/concepts/LOG.md) for a simple yet full understanding of logging. Disgo provides structured, leveled logging of the API Wrapper via the `disgo.Logger` global variable _(disabled by default)_. Enable the logger using `zerolog.SetGlobalLevel(zerolog.LEVEL)`.
Disgo provides structured, leveled logging of the API Wrapper via the `disgo.Logger` global variable _(disabled by default)_. Enable the logger using `zerolog.SetGlobalLevel(zerolog.LEVEL)`.

_Read [What is a Log](/_contribution/concepts/LOG.md) for a simple yet full understanding of logging._

### Sharding

Read [What is a Discord Shard](/_contribution/concepts/SHARD.md) for a simple yet full understanding of sharding on Discord. Using the [Shard Manager](/_contribution/concepts/SHARD.md#the-shard-manager) is **optional**. You can manually implement a shard manager through the `disgo.Client.Sessions` array.
Using the automatic [Shard Manager](/_contribution/concepts/SHARD.md#the-shard-manager) is **optional** and **customizable**.

_Read [What is a Discord Shard](/_contribution/concepts/SHARD.md) for a simple yet full understanding of sharding on Discord._

### Caching

Read [What is a Cache](/_contribution/concepts/CACHE.md) for a simple yet full understanding of the Disgo Cache. The [Disgo Cache](/_contribution/concepts/CACHE.md#the-disgo-cache) is **optional**. The **cache interface** allows you to replace the built-in cache with another store _(such as Redis or Memcached)_ and/or provide your own method of caching data.
The [Disgo Cache](/_contribution/concepts/CACHE.md#the-disgo-cache) is **optional** and **customizable**.

The **cache interface** allows you to replace the built-in cache with another store _(such as Redis or Memcached)_ or provide your own caching implementation.

_Read [What is a Cache](/_contribution/concepts/CACHE.md) for a simple yet full understanding of the Disgo Cache._

## Examples

| Example | Description |
| :----------------------------- | :--------------------------------------------------------- |
| main | Learn how to use `disgo`. |
| [command](/_examples/command/) | Create an application command and respond to interactions. |
| [message](/_examples/message/) | Send a message with text, emojis, files and/or components. |
| [message](/_examples/message/) | Send a message with text, emojis, files, and components. |
| [image](/_examples/image/) | Set the bot's avatar using an image. |

_Check out the [examples](/_examples/) directory for more._
_Check out the [examples](/_examples/) directory for more examples._

### Import

Expand All @@ -79,20 +95,22 @@ _Disgo branches are referenced by API version (i.e `v10`)._

### Configuration

**You must create a Discord Application in the [Discord Developer Portal](https://discord.com/developers/docs/getting-started#creating-an-app) to receive your Bot Token.**
_**You must create a Discord Application in the [Discord Developer Portal](https://discord.com/developers/docs/getting-started#creating-an-app) to receive your Bot Token.**_

Use the client to configure the bot's settings.

```go
bot := &disgo.Client{
ApplicationID: "APPID", // optional
Authentication: disgo.BotToken("TOKEN"), // or BearerToken("TOKEN")
Authorization: &disgo.Authorization{ ... },
Config: disgo.DefaultConfig(),
Handlers: new(disgo.Handlers),
Sessions: disgo.NewSessionManager()
}
```

_Need more information? Read the [bot example](/_examples/bot)._
_Need more information about configuration? Check out the [bot example](/_examples/bot)._

### Create a Command

Expand Down Expand Up @@ -131,7 +149,7 @@ _Disgo provides automatic [Gateway Intent](https://discord.com/developers/docs/t

### Output

Open a WebSocket **Session** to receive events.
Open a WebSocket **Session** to receive and send events.

```go
// Connect a new session to the Discord Gateway (WebSocket Connection).
Expand All @@ -146,7 +164,7 @@ if err := s.Connect(bot); err != nil {
The following message will be logged when a user creates an `InteractionCreate` event by using `/main` in a Direct Message with the Bot on Discord.

```
main called by SCB.
main called by switchupcb.
```

### Summary
Expand All @@ -168,6 +186,9 @@ disgo.<Endpoint>.Send()
disgo.Session.Connect()
disgo.Session.Disconnect()

// Use send events to send data to Discord's Gateway.
disgo.<Event>.SendEvent()

// Use event handlers to handle events from Discord's Gateway.
disgo.Client.Handle(<event>, <handler>)
disgo.Client.Remove(<event>, <index>)
Expand All @@ -179,26 +200,27 @@ disgo.Client.Authentication.<Settings>
disgo.Client.Authorization.<Settings>
disgo.Client.Config.Request.<Settings>
disgo.Client.Config.Gateway.<Settings>

// Use the client's shard manager to handle sharding automatically or manually.
disgo.Client.Shard.<Settings>
disgo.Client.Shard.<map[Session][]map[Shard][]GuildIDs>

// Use the client to manage the optional cache.
disgo.Client.Cache.<Settings>
disgo.Client.Cache.<Requests>
disgo.Client.Cache.<...>
```

## Features

### Why Go?

Go is a statically typed, compiled programming language _(with a garbage collector)_. As a result, it performs computationally better compared to _most_ languages that provide [Discord API Wrappers](https://discord.com/developers/docs/topics/community-resources#libraries). Go maintains superior asynchronous handling due to the use of [Goroutines](https://gobyexample.com/goroutines) and [Channels](https://gobyexample.com/channels). This is useful since **a Discord Bot is a server-side software**.
Go is a statically typed, compiled programming language _(with a garbage collector)_. So it performs computationally better compared to _most_ languages that provide [Discord API Wrappers](https://discord.com/developers/docs/topics/community-resources#libraries).

Go maintains superior asynchronous handling due to the use of [Goroutines](https://gobyexample.com/goroutines) and [Channels](https://gobyexample.com/channels): This is helpful since **a Discord Bot is server-side software**.

### Comparison

Disgo supports every feature in the Discord API and is **the most customizable Discord API Wrapper** due to its optional caching, shard management, rate limiting, and logging. **DiscordGo** is not feature-complete and **Disgord** is limiting. Look no further than the name. The word `disgo` contains 5 letters — while the others have 7+ — saving you precious keyboard strokes. Most important is Disgo's performance, which saves you money by reducing server costs. _Don't believe me?_ Check this out!
Disgo supports every feature in the Discord API and is **the most customizable Discord API Wrapper** due to its optional caching, shard management, rate limiting, and logging.
- **DiscordGo** is not feature-complete.
- **Disgord** is limiting.

Look no further than the name. The word `disgo` contains 5 letters — while the others have 7+ — saving you precious keyboard strokes.

Most important is Disgo's performance, which saves you money by reducing server costs.

_Don't believe me? Check this out!_

#### CPU

Expand All @@ -214,7 +236,7 @@ Disgo adds **5 MB** to a compiled binary.

### Contributing

Disgo is the easiest Discord Go API for developers to use and contribute to. You can contribute to this repository by viewing the [Project Structure, Code Specifications, and Roadmap](/_contribution/CONTRIBUTING.md).
Disgo is the easiest Discord Go API for developers to use and contribute to: You can contribute to this repository by viewing the [Project Structure, Code Specifications, and Roadmap](/_contribution/CONTRIBUTING.md).

## Ecosystem

Expand Down
6 changes: 3 additions & 3 deletions _contribution/CONTRIBUTING.md
Original file line number Diff line number Diff line change
Expand Up @@ -89,10 +89,10 @@ Use `go test` to run the tests in the current directory. Use `go test ./<dir>` t

# Roadmap

Disgo is **STABLE**. The following additional features are being implemented.
Disgo is **STABLE**.

The following additional features are being implemented:
1. Voice Connections ([UDP Decision](/_contribution/libraries/), [Audio Processing using Opus](https://discord.com/developers/docs/topics/voice-connections#encrypting-and-sending-voice))
2. [Sharding](https://github.com/switchupcb/disgo/issues/26)
3. [Cache](https://github.com/switchupcb/disgo/issues/39)
2. [Cache](https://github.com/switchupcb/disgo/issues/39)

[_Get assigned a feature or example now._](https://github.com/switchupcb/disgo/issues/45)
27 changes: 22 additions & 5 deletions _contribution/concepts/CACHE.md
Original file line number Diff line number Diff line change
@@ -1,15 +1,32 @@
# What is a Cache?

Disgo provides an **optional** cache along with a **cache interface** for your Discord Bot. Read [The Disgo Cache](/cache/README.md) for information about its implementation.
A [cache (computing)](https://en.wikipedia.org/wiki/Cache_(computing)) is a component that stores data for quick retrieval.

## Why Cache?

## Why Do We Cache?
A cache is an alternative to sending redundant requests — that take time to complete — to Discord: A cache stores data so that future requests for that data can be served immediately.

Using **Requests** and **Sessions** allow you to retrieve data from Discord, but it becomes redundant to receive the same data over and over again. For example, retrieving _the amount of users in a guild_ requires a network request to be sent _to_ and returned _from_ Discord. Without a cache, retrieving _the exact same information_ requires yet another network request; even when guild's condition remains unchanged. A cache is an alternative to sending redundant requests — that take time to complete — to Discord: A cache stores data so that future requests for that data can be served immediately.
For example, retrieving _the amount of users in a guild_ requires a network request to be sent to Discord, and another network request to be returned from Discord. Without a cache, retrieving _the exact same information_ again requires two more network requests, even when the guild's condition remains unchanged.

## When to Use a Cache?

Caches are useful for storing costly requests or calculations relevant to the lifetime of the application. In other words, a cache is **NOT** meant to be used for long-term storage. **If you need data to persist when your bot restarts, use a database.**
Caches are useful for storing costly requests or calculations relevant to the application's lifetime. In other words, a cache is **NOT** meant to be used for long-term storage.

**If you need data to persist when your bot restarts, use a database.**

## How Does a Cache Work?

Read [Caching Overview](https://aws.amazon.com/caching) for an in-depth explanation. A cache is typically stored in-memory which allows the application to store and retrieve data fast (with minimal latency). A cache that receives a request — for the amount of users in a guild — will store an entry for use later. When a request is made for _the exact same information_, the cache will use the in-memory entry instead of creating a costly network request. **Cache Invalidation** describes the process of replacing or removing cache entries. In the example above, we know to invalidate or update the stored value for the _amount of users in a guild_ when a user joins or leaves the server. For more information, read [Cache Invalidation](https://en.wikipedia.org/wiki/Cache_invalidation).
_Read [Caching Overview](https://aws.amazon.com/caching) for an in-depth explanation._

A cache is typically stored in-memory, which allows the application to store and retrieve data fast (with minimal latency).

A cache receiving a request — for the number of users in a guild — will store an entry for use later. When a request is made for _the exact same information_, the cache will use the in-memory entry instead of creating a costly network request.

**Cache Invalidation** describes the process of replacing or removing cache entries. In the above example, we know to invalidate or update the stored value for the _amount of users in a guild_ when a user joins or leaves the server.

_For more information, read [Cache Invalidation](https://en.wikipedia.org/wiki/Cache_invalidation)._


## How Do I Cache?

Disgo provides an **optional** cache along with a **cache interface** for your Discord Bot. Read [The Disgo Cache](/cache/README.md) for information about its implementation.

0 comments on commit 3da8b14

Please sign in to comment.