Skip to content

Commit

Permalink
feat(import): Switching graphql API to Rest API for auto imports (#739)
Browse files Browse the repository at this point in the history
This pr changes the Graphql API approach for the REST API on the auto
imports feature . This way we can import repos without a GitHub token
.There are some sync that won't work without proper validation, those
are :

1. PR Commits
2. Pr Reviews
3. Repo Issues
4. Repo metadata
5. Pull Requests
6. Repo Stars

In this content we also reintroduce stdout logging to the importer

To test this  you need to 

1. Log into mergestat
8. go to /repos page and click on  add repositories
9. Choose an user/org with public repos 
10. Those repos should be appearing  after a while

Closes #540
  • Loading branch information
ramiroJCB committed Jan 26, 2023
2 parents 0368ae6 + dcc4835 commit f9c9bd9
Show file tree
Hide file tree
Showing 11 changed files with 326 additions and 30 deletions.
17 changes: 9 additions & 8 deletions cmd/worker/main.go
Expand Up @@ -5,13 +5,6 @@ import (
"database/sql"
"errors"
"fmt"
"github.com/mergestat/mergestat/internal/cron"
"github.com/mergestat/mergestat/internal/jobs/repo"
"github.com/mergestat/mergestat/internal/syncer"
"github.com/mergestat/mergestat/internal/timeout"
"github.com/mergestat/sqlq"
"github.com/mergestat/sqlq/runtime/embed"
"github.com/mergestat/sqlq/schema"
"net/http"
_ "net/http/pprof"
"net/url"
Expand All @@ -22,6 +15,14 @@ import (
"syscall"
"time"

"github.com/mergestat/mergestat/internal/cron"
"github.com/mergestat/mergestat/internal/jobs/repo"
"github.com/mergestat/mergestat/internal/syncer"
"github.com/mergestat/mergestat/internal/timeout"
"github.com/mergestat/sqlq"
"github.com/mergestat/sqlq/runtime/embed"
"github.com/mergestat/sqlq/schema"

"github.com/go-git/go-git/v5"
"github.com/golang-migrate/migrate/v4"
_ "github.com/golang-migrate/migrate/v4/database/postgres"
Expand Down Expand Up @@ -247,7 +248,7 @@ func main() {
var worker, _ = embed.NewWorker(upstream, embed.WorkerConfig{Queues: queues})

// register job handlers for types implemented by this worker
_ = worker.Register("repos/auto-import", repo.AutoImport(pool, db))
_ = worker.Register("repos/auto-import", repo.AutoImport(&logger, pool))

// TODO all of the following "params" should be configurable
// either via the database/app or possibly with env vars
Expand Down
130 changes: 130 additions & 0 deletions internal/db/models.go

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

1 change: 1 addition & 0 deletions internal/db/querier.go

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

3 changes: 3 additions & 0 deletions internal/db/queries.sql
Expand Up @@ -86,6 +86,9 @@ INSERT INTO mergestat.repo_sync_logs (log_type, message, repo_sync_queue_id) VAL
-- name: SetSyncJobStatus :exec
SELECT mergestat.set_sync_job_status(@Status::TEXT, @ID::BIGINT);

-- name: FetchGitHubToken :one
SELECT pgp_sym_decrypt(credentials, $1) FROM mergestat.service_auth_credentials WHERE type = 'GITHUB_PAT' ORDER BY created_at DESC LIMIT 1;

-- We use a CTE here to retrieve all the repo_sync_jobs that were previously enqueued, to make sure that we *do not* re-enqueue anything new until the previously enqueued jobs are *completed*.
-- This allows us to make sure all repo syncs complete before we reschedule a new batch.
-- We have now also added a concept of type groups which allows us to apply this same logic but by each group type which is where the PARTITION BY clause comes into play
Expand Down
11 changes: 11 additions & 0 deletions internal/db/queries.sql.go

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

0 comments on commit f9c9bd9

Please sign in to comment.