/
README.Rmd
104 lines (78 loc) · 3.86 KB
/
README.Rmd
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
---
output: github_document
---
<!-- badges: start --> [](https://www.tidyverse.org/lifecycle/#experimental)
[](https://github.com/lorenzwalthert/touchstone/actions)
<!-- badges: end -->
```{r, include = FALSE}
knitr::opts_chunk$set(
collapse = TRUE,
comment = "#>",
fig.path = "man/figures/README-",
out.width = "100%",
eval = FALSE
)
```
# touchstone
{touchstone} is a developer tool for continuous benchmarking with a focus
on reliable relative measurement, uncertainty reporting and user convenience.
The results are directly reported as a comment in GitHub Pull Requests.
## Installation
You can install the package from CRAN:
``` r
install.packages("touchstone")
```
And the development version from [GitHub](https://github.com/lorenzwalthert/touchstone){target="_blank"} with:
``` r
# install.packages("devtools")
devtools::install_github("lorenzwalthert/touchstone")
```
## Getting Started
You can start using {touchstone} in your package repository with:
```{r, eval = FALSE}
touchstone::use_touchstone()
```
For a detailed explanation on how to configure and use {touchstone} see the
["Using touchstone"](https://lorenzwalthert.github.io/touchstone/articles/touchstone.html) vignette.
## Motivation
The motivation for touchstone is to provide accurate benchmarking results for
package developers. The following insights were the motivation:
- **Compute power in GitHub Action VMs varies too much for reliable isolated
benchmarks**: Experience with styler showed that a variation [around
30%](https://github.com/r-lib/styler/pull/679) for identical benchmarking code
is possible. The solution to this is to benchmark the two branches in one
CI/CD run and look at *relative difference* between branches. This matters in
particular when running one iteration of a benchmark takes long (>> a few
seconds) and speed implications are not huge.
- **Timelines of absolute changes are mostly noise:** Maintaining a timeline of
absolute benchmark times is of limited use because of the first bullet, at
least when benchmark results don't vary significantly (> 50%).
- **Dependencies should be identical across versions:** R and package versions
of dependencies must be fixed via [RSPM](http://packagemanager.rstudio.com) to
allow as much continuation as possible anyways. Changing the timestamp of RSPM
can happen in PRs that are only dedicated to dependency updates.
- **Pull requests are a natural unit for measuring speed:** Linking benchmarking
to pull requests make sense because you can easily benchmark any revision
against any other. Just open a pull request from one branch on another. You
can also easily keep track of how performance of your development version
evolves by opening a PR against a branch with the released version. No need to
ever merge these. Once you release a new version of you package, you can
change the target branch of the pull request to start anew. The pull request
comments will preserve the information.
## Conceptual
For your PR branch and the target branch, {touchstone} will:
* build the two versions of the package in isolated libraries.
* run the code you want to benchmark, many times, in [random
order](https://lorenzwalthert.github.io/touchstone/articles/inference.html#sampling).
This ensures the accurate measurement of relative differences between the
branches.
Once done with the measurements, it will
* comment the results of the benchmarking on the PR.
* create visualizations as Github Action artifacts that show how the
distribution of the timings for both branches.
## Status
This package is experimental. You can see an example usage in
[{styler}](https://github.com/r-lib/styler/pull/799).