Skip to content

Commit

Permalink
git: add push.refspec to push using a refspec
Browse files Browse the repository at this point in the history 
Add `.spec.git.push.refspec` to allow specifying a refspec to be used
for performing a push operation. If specified alongside
`.spec.git.push.branch`, two push operations, one for each specified
push configuration will be performed.

Signed-off-by: Sanskar Jaiswal <jaiswalsanskar078@gmail.com>
  • Loading branch information
@aryan9600
aryan9600 committed Aug 9, 2023
1 parent 09aa2ac commit 051844c
Show file tree
Hide file tree
Showing 6 changed files with 274 additions and 80 deletions.
12 changes: 10 additions & 2 deletions api/v1beta1/git.go
View file
Original file line number Diff line number Diff line change
Expand Up @@ -85,6 +85,14 @@ type PushSpec struct {
// Branch specifies that commits should be pushed to the branch
// named. The branch is created using `.spec.checkout.branch` as the
// starting point, if it doesn't already exist.
// +required
Branch string `json:"branch"`
// +optional
Branch string `json:"branch,omitempty"`

// Refspec specifies the Git Refspec to use for a push operation.
// If both Branch and Refspec are provided, then the commit is pushed
// to the branch and also using the specified refspec.
// For more details about Git Refspecs, see:
// https://git-scm.com/book/en/v2/Git-Internals-The-Refspec
// +optional
Refspec string `json:"refspec,omitempty"`
}
9 changes: 7 additions & 2 deletions config/crd/bases/image.toolkit.fluxcd.io_imageupdateautomations.yaml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -135,8 +135,13 @@ spec:
to the branch named. The branch is created using `.spec.checkout.branch`
as the starting point, if it doesn't already exist.
type: string
required:
- branch
refspec:
description: 'Refspec specifies the Git Refspec to use for
a push operation. If both Branch and Refspec are provided,
then the commit is pushed to the branch and also using the
specified refspec. For more details about Git Refspecs,
see: https://git-scm.com/book/en/v2/Git-Internals-The-Refspec'
type: string
type: object
required:
- commit
Expand Down
17 changes: 17 additions & 0 deletions docs/api/v1beta1/image-automation.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -638,11 +638,28 @@ string
</em>
</td>
<td>
<em>(Optional)</em>
<p>Branch specifies that commits should be pushed to the branch
named. The branch is created using <code>.spec.checkout.branch</code> as the
starting point, if it doesn&rsquo;t already exist.</p>
</td>
</tr>
<tr>
<td>
<code>refspec</code><br>
<em>
string
</em>
</td>
<td>
<em>(Optional)</em>
<p>Refspec specifies the Git Refspec to use for a push operation.
If both Branch and Refspec are provided, then the commit is pushed
to the branch and also using the specified refspec.
For more details about Git Refspecs, see:
<a href="https://git-scm.com/book/en/v2/Git-Internals-The-Refspec">https://git-scm.com/book/en/v2/Git-Internals-The-Refspec</a></p>
</td>
</tr>
</tbody>
</table>
</div>
Expand Down
36 changes: 33 additions & 3 deletions docs/spec/v1beta1/imageupdateautomations.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -408,8 +408,16 @@ type PushSpec struct {
// Branch specifies that commits should be pushed to the branch
// named. The branch is created using `.spec.checkout.branch` as the
// starting point, if it doesn't already exist.
// +required
Branch string `json:"branch"`
// +optional
Branch string `json:"branch,omitempty"`

// Refspec specifies the Git Refspec to use for a push operation.
// If both Branch and Refspec are provided, then the commit is pushed
// to the branch and also using the specified refspec.
// For more details about Git Refspecs, see:
// https://git-scm.com/book/en/v2/Git-Internals-The-Refspec
// +optional
Refspec string `json:"refspec,omitempty"`
}
```

Expand All @@ -418,7 +426,11 @@ pushed to the same branch at the origin. If `.spec.git.checkout` is not present,
to the branch given in the `GitRepository` referenced by `.spec.sourceRef`. If none of these yield a
branch name, the automation will fail.

When `push` is present, the `branch` field specifies a branch to push to at the origin. The branch
If `push.refspec` is present, the refspec specified is used to perform the push operation.
An example of a valid refspec is `refs/heads/branch:refs/heads/branch`. This allows users to
push to an arbitary destination reference.

If `push.branch` is present, the specified branch is pushed to at the origin. The branch
will be created locally if it does not already exist, starting from the checkout branch. If it does
already exist, it will be overwritten with the cloned version plus the changes made by the
controller. Alternatively, force push can be disabled by starting the controller with `--feature-gates=GitForcePushBranch=false`,
Expand All @@ -427,6 +439,9 @@ Note that without force push in push branches, if the target branch is stale, th
be able to conclude the operation and will consistently fail until the branch is either deleted or
refreshed.

If both `push.refspec` and `push.branch` are specified, then the reconciler will perform two push
operations, one to the specified branch and another using the specified refspec.

In the following snippet, updates will be pushed as commits to the branch `auto`, and when that
branch does not exist at the origin, it will be created locally starting from the branch `main`, and
pushed:
Expand All @@ -441,6 +456,21 @@ spec:
branch: auto
```

In the following snippet, updates and commits will be made on the `auto` branch locally.
The commits will be then pushed to the `auto` branch and then using the `refs/heads/auto:refs/heads/main`
refspec:

```yaml
spec:
git:
checkout:
ref:
branch: main
push:
branch: auto
refspec: refs/heads/auto:refs/heads/main
```

## Update strategy

The `.spec.update` field specifies how to carry out updates on the git repository. There is one
Expand Down
181 changes: 119 additions & 62 deletions internal/controller/imageupdateautomation_controller.go
View file
Original file line number Diff line number Diff line change
Expand Up @@ -216,30 +216,15 @@ func (r *ImageUpdateAutomationReconciler) Reconcile(ctx context.Context, req ctr
}

// validate the git spec and default any values needed later, before proceeding
var ref *sourcev1.GitRepositoryRef
var checkoutRef *sourcev1.GitRepositoryRef
if gitSpec.Checkout != nil {
ref = &gitSpec.Checkout.Reference
tracelog.Info("using git repository ref from .spec.git.checkout", "ref", ref)
checkoutRef = &gitSpec.Checkout.Reference
tracelog.Info("using git repository ref from .spec.git.checkout", "ref", checkoutRef)
} else if r := origin.Spec.Reference; r != nil {
ref = r
tracelog.Info("using git repository ref from GitRepository spec", "ref", ref)
checkoutRef = r
tracelog.Info("using git repository ref from GitRepository spec", "ref", checkoutRef)
} // else remain as `nil` and git.DefaultBranch will be used.

var pushBranch string
if gitSpec.Push != nil {
pushBranch = gitSpec.Push.Branch
tracelog.Info("using push branch from .spec.push.branch", "branch", pushBranch)
} else {
// Here's where it gets constrained. If there's no push branch
// given, then the checkout ref must include a branch, and
// that can be used.
if ref == nil || ref.Branch == "" {
return failWithError(fmt.Errorf("Push branch not given explicitly, and cannot be inferred from .spec.git.checkout.ref or GitRepository .spec.ref"))
}
pushBranch = ref.Branch
tracelog.Info("using push branch from $ref.branch", "branch", pushBranch)
}

tmp, err := os.MkdirTemp("", fmt.Sprintf("%s-%s", originName.Namespace, originName.Name))
if err != nil {
return failWithError(err)
Expand All @@ -250,42 +235,43 @@ func (r *ImageUpdateAutomationReconciler) Reconcile(ctx context.Context, req ctr
}
}()

debuglog.Info("attempting to clone git repository", "gitrepository", originName, "ref", ref, "working", tmp)

authOpts, err := r.getAuthOpts(ctx, &origin)
if err != nil {
return failWithError(err)
}

clientOpts := []gogit.ClientOption{gogit.WithDiskStorage()}
if authOpts.Transport == git.HTTP {
clientOpts = append(clientOpts, gogit.WithInsecureCredentialsOverHTTP())
var pushBranch string
var switchBranch bool
if gitSpec.Push != nil {
// We only need to switch branches when a branch has been specified in
// the push spec and it is different than the one in the checkout ref.
if gitSpec.Push.Branch != "" && gitSpec.Push.Branch != checkoutRef.Branch {
pushBranch = gitSpec.Push.Branch
switchBranch = true
tracelog.Info("using push branch from .spec.push.branch", "branch", pushBranch)
}
} else {
// Here's where it gets constrained. If there's no push branch
// given, then the checkout ref must include a branch, and
// that can be used.
if checkoutRef == nil || checkoutRef.Branch == "" {
return failWithError(
fmt.Errorf("Push spec not provided, and cannot be inferred from .spec.git.checkout.ref or GitRepository .spec.ref"),
)
}
pushBranch = checkoutRef.Branch
tracelog.Info("using push branch from $ref.branch", "branch", pushBranch)
}

// If the push branch is different from the checkout ref, we need to
// have all the references downloaded at clone time, to ensure that
// SwitchBranch will have access to the target branch state. fluxcd/flux2#3384
//
// To always overwrite the push branch, the feature gate
// GitAllBranchReferences can be set to false, which will cause
// the SwitchBranch operation to ignore the remote branch state.
allReferences := r.features[features.GitAllBranchReferences]
if pushBranch != ref.Branch {
clientOpts = append(clientOpts, gogit.WithSingleBranch(!allReferences))
}
debuglog.Info("attempting to clone git repository", "gitrepository", originName, "ref", checkoutRef, "working", tmp)

gitClient, err := gogit.NewClient(tmp, authOpts, clientOpts...)
gitClient, err := r.constructGitClient(ctx, &origin, tmp, switchBranch)
if err != nil {
return failWithError(err)
}
defer gitClient.Close()

opts := repository.CloneConfig{}
if ref != nil {
opts.Tag = ref.Tag
opts.SemVer = ref.SemVer
opts.Commit = ref.Commit
opts.Branch = ref.Branch
if checkoutRef != nil {
opts.Tag = checkoutRef.Tag
opts.SemVer = checkoutRef.SemVer
opts.Commit = checkoutRef.Commit
opts.Branch = checkoutRef.Branch
}

if enabled, _ := r.features[features.GitShallowClone]; enabled {
Expand All @@ -299,9 +285,9 @@ func (r *ImageUpdateAutomationReconciler) Reconcile(ctx context.Context, req ctr
return failWithError(err)
}

// When there's a push spec, the pushed-to branch is where commits
// When there's a push branch specified, the pushed-to branch is where commits
// shall be made
if gitSpec.Push != nil && !(ref != nil && ref.Branch == pushBranch) {
if switchBranch {
// Use the git operations timeout for the repo.
fetchCtx, cancel := context.WithTimeout(ctx, origin.Spec.Timeout.Duration)
defer cancel()
Expand Down Expand Up @@ -354,7 +340,6 @@ func (r *ImageUpdateAutomationReconciler) Reconcile(ctx context.Context, req ctr

debuglog.Info("ran updates to working dir", "working", tmp)

var statusMessage string
var signingEntity *openpgp.Entity
if gitSpec.Commit.SigningKey != nil {
if signingEntity, err = r.getSigningEntity(ctx, auto); err != nil {
Expand Down Expand Up @@ -388,40 +373,80 @@ func (r *ImageUpdateAutomationReconciler) Reconcile(ctx context.Context, req ctr
err = extgogit.ErrEmptyCommit
}

var statusMessage strings.Builder
if err != nil {
if !errors.Is(err, git.ErrNoStagedFiles) && !errors.Is(err, extgogit.ErrEmptyCommit) {
return failWithError(err)
}

log.Info("no changes made in working directory; no commit")
statusMessage = "no updates made"
statusMessage.WriteString("no updates made")

if auto.Status.LastPushTime != nil && len(auto.Status.LastPushCommit) >= 7 {
statusMessage = fmt.Sprintf("%s; last commit %s at %s", statusMessage, auto.Status.LastPushCommit[:7], auto.Status.LastPushTime.Format(time.RFC3339))
statusMessage.WriteString(fmt.Sprintf("; last commit %s at %s",
auto.Status.LastPushCommit[:7], auto.Status.LastPushTime.Format(time.RFC3339)))
}
} else {
// Use the git operations timeout for the repo.
pushCtx, cancel := context.WithTimeout(ctx, origin.Spec.Timeout.Duration)
defer cancel()
opts := repository.PushConfig{}
forcePush := r.features[features.GitForcePushBranch]
if forcePush && pushBranch != ref.Branch {
opts.Force = true

var pushToBranch bool
var pushWithRefspec bool
// If a refspec is specified, then we need to perform a push using
// that refspec.
if gitSpec.Push != nil && gitSpec.Push.Refspec != "" {
pushWithRefspec = true
}
if err := gitClient.Push(pushCtx, opts); err != nil {
return failWithError(err)
// We need to push the commit to the push branch if one was specified, or if
// no push config was specified, then we need to push to the branch we checked
// out to.
if (gitSpec.Push != nil && gitSpec.Push.Branch != "") || gitSpec.Push == nil {
pushToBranch = true
}

if pushToBranch {
// If the force push feature flag is true and we are pushing to a
// different branch than the one we checked out to, then force push
// these changes.
var pushConfig repository.PushConfig
forcePush := r.features[features.GitForcePushBranch]
if forcePush && switchBranch {
pushConfig.Force = true
}

if err := gitClient.Push(pushCtx, pushConfig); err != nil {
return failWithError(err)
}
log.Info("pushed commit to origin", "revision", rev, "branch", pushBranch)
statusMessage.WriteString(fmt.Sprintf("commited and pushed commit '%s' to branch '%s'", rev, pushBranch))
}

r.event(ctx, auto, eventv1.EventSeverityInfo, fmt.Sprintf("Committed and pushed change %s to %s\n%s", rev, pushBranch, message))
log.Info("pushed commit to origin", "revision", rev, "branch", pushBranch)
if pushWithRefspec {
pushConfig := repository.PushConfig{
Refspecs: []string{gitSpec.Push.Refspec},
}
if err := gitClient.Push(pushCtx, pushConfig); err != nil {
return failWithError(err)
}
log.Info("pushed commit to origin", "revision", rev, "refspec", gitSpec.Push.Refspec)

if pushToBranch {
statusMessage.WriteString(fmt.Sprintf(" and using refspec '%s'", gitSpec.Push.Refspec))
} else {
statusMessage.WriteString(fmt.Sprintf("committed and pushed commit '%s' using refspec '%s'", rev, gitSpec.Push.Refspec))
}
}

r.event(ctx, auto, eventv1.EventSeverityInfo, fmt.Sprintf("%s\n%s", statusMessage.String(), message))

auto.Status.LastPushCommit = rev
auto.Status.LastPushTime = &metav1.Time{Time: start}
statusMessage = "committed and pushed " + rev + " to " + pushBranch
}

// Getting to here is a successful run.
auto.Status.LastAutomationRunTime = &metav1.Time{Time: start}
imagev1.SetImageUpdateAutomationReadiness(&auto, metav1.ConditionTrue, imagev1.ReconciliationSucceededReason, statusMessage)
imagev1.SetImageUpdateAutomationReadiness(&auto, metav1.ConditionTrue, imagev1.ReconciliationSucceededReason, statusMessage.String())
if err := r.patchStatus(ctx, req, auto.Status); err != nil {
return ctrl.Result{Requeue: true}, err
}
Expand Down Expand Up @@ -547,6 +572,38 @@ func (r *ImageUpdateAutomationReconciler) getAuthOpts(ctx context.Context, repos
return opts, nil
}

// constructGitClient constructs and returns a new gogit client.
func (r *ImageUpdateAutomationReconciler) constructGitClient(ctx context.Context,
origin *sourcev1.GitRepository, repoDir string, switchBranch bool) (*gogit.Client, error) {
authOpts, err := r.getAuthOpts(ctx, origin)
if err != nil {
return nil, err
}

clientOpts := []gogit.ClientOption{gogit.WithDiskStorage()}
if authOpts.Transport == git.HTTP {
clientOpts = append(clientOpts, gogit.WithInsecureCredentialsOverHTTP())
}

// If the push branch is different from the checkout ref, we need to
// have all the references downloaded at clone time, to ensure that
// SwitchBranch will have access to the target branch state. fluxcd/flux2#3384
//
// To always overwrite the push branch, the feature gate
// GitAllBranchReferences can be set to false, which will cause
// the SwitchBranch operation to ignore the remote branch state.
allReferences := r.features[features.GitAllBranchReferences]
if switchBranch {
clientOpts = append(clientOpts, gogit.WithSingleBranch(!allReferences))
}

gitClient, err := gogit.NewClient(repoDir, authOpts, clientOpts...)
if err != nil {
return nil, err
}
return gitClient, nil
}

// getSigningEntity retrieves an OpenPGP entity referenced by the
// provided imagev1.ImageUpdateAutomation for git commit signing
func (r *ImageUpdateAutomationReconciler) getSigningEntity(ctx context.Context, auto imagev1.ImageUpdateAutomation) (*openpgp.Entity, error) {
Expand Down

0 comments on commit 051844c

Please sign in to comment.