Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

PMM-10231 Service Params Source #1110

Closed
wants to merge 60 commits into from
Closed
Show file tree
Hide file tree
Changes from 53 commits
Commits
Show all changes
60 commits
Select commit Hold shift + click to select a range
6a508a3
PMM-10231 Credentials source implementation. API + admin
pkadej Aug 1, 2022
9a15289
Merge branch 'main' into PMM-10231_credential-sources
pkadej Aug 2, 2022
cddfb8a
PMM-10231 Credentials source implementation. API + agent
pkadej Aug 2, 2022
336742f
Merge branch 'main' into PMM-10231_credential-sources
pkadej Aug 2, 2022
91ff03b
Merge branch 'main' into PMM-10231_credential-sources
pkadej Aug 3, 2022
a9049ab
PMM-10231 Credentials source implementation.
pkadej Aug 3, 2022
79a3865
Merge branch 'main' into PMM-10231_credential-sources
pkadej Aug 8, 2022
a0b3f85
PMM-10231 Credentials source implementation.
pkadej Aug 8, 2022
20e3173
Merge branch 'main' into PMM-10231_credential-sources
pkadej Aug 9, 2022
aee4176
PMM-10231 Credentials source implementation.
pkadej Aug 9, 2022
79b6193
PMM-10231 Credentials source implementation.
pkadej Aug 9, 2022
d8b291a
PMM-10231 Credentials source implementation. Fixing tests.
pkadej Aug 9, 2022
304e811
PMM-10231 Credentials source implementation. Fixing tests.
pkadej Aug 9, 2022
c469a5b
PMM-10231 Credentials source implementation. Change admin's defaults.
pkadej Aug 9, 2022
5e39a33
Merge branch 'main' into PMM-10231_credential-sources
pkadej Aug 10, 2022
7dfc5e7
Merge branch 'main' into PMM-10231_credential-sources
pkadej Aug 10, 2022
0e33ad5
Merge branch 'main' into PMM-10231_credential-sources
pkadej Aug 11, 2022
591c8c4
Merge branch 'main' into PMM-10231_credential-sources
pkadej Aug 11, 2022
8761747
Merge branch 'main' into PMM-10231_credential-sources
pkadej Aug 15, 2022
132d5c5
Merge branch 'main' into PMM-10231_credential-sources
pkadej Aug 16, 2022
637fdf2
Merge branch 'main' into PMM-10231_credential-sources
pkadej Aug 17, 2022
800e10a
Merge branch 'main' into PMM-10231_credential-sources
pkadej Aug 18, 2022
cdc336a
PMM-10231 Defaults set for username/passwords in case when credential…
pkadej Aug 18, 2022
fb675b6
PMM-10231 Fixing api-tests.
pkadej Aug 18, 2022
be9f8b0
Merge branch 'main' into PMM-10231_credential-sources
pkadej Aug 23, 2022
b02b19a
PMM-10231 Commands fix.
pkadej Aug 23, 2022
1477334
PMM-10231 Codereview fixes.
pkadej Aug 24, 2022
0e92c4c
Merge branch 'main' into PMM-10231_credential-sources
pkadej Aug 24, 2022
d1b5dea
Merge branch 'main' into PMM-10231_credential-sources
pkadej Aug 26, 2022
dec0d4c
PMM-10231 Codereview fixes.
pkadej Aug 26, 2022
17d442e
PMM-10231 Codereview fixes.
pkadej Aug 26, 2022
3d471f5
Merge branch 'main' into PMM-10231_credential-sources
pkadej Aug 27, 2022
0108205
Merge branch 'main' into PMM-10231_credential-sources
pkadej Aug 30, 2022
c982151
Merge branch 'main' into PMM-10231_credential-sources
pkadej Sep 5, 2022
e7911cf
PMM-10231 Updates for code review.
pkadej Sep 5, 2022
000f9c7
PMM-10231 Updates for code review.
pkadej Sep 5, 2022
04883e3
Merge branch 'main' into PMM-10231_credential-sources
pkadej Sep 7, 2022
87ecf15
Merge branch 'main' into PMM-10231_credential-sources
pkadej Sep 9, 2022
fe1abae
Merge branch 'main' into PMM-10231_credential-sources
pkadej Sep 9, 2022
8e5a3f1
PMM-10231 Updating descriptor.bin
pkadej Sep 9, 2022
81df2ce
Merge branch 'main' into PMM-10231_credential-sources
pkadej Sep 19, 2022
b699b67
Merge branch 'main' into PMM-10231_credential-sources
pkadej Oct 19, 2022
38e0f3d
PMM-10231 api regenerate
pkadej Oct 19, 2022
ab22a33
PMM-10231 api regenerate
pkadej Oct 19, 2022
826e2e7
PMM-10231 service-params-source renaming api+admin
pkadej Oct 19, 2022
9e11efd
PMM-10231 service-params-source renaming agent
pkadej Oct 19, 2022
8b2d481
PMM-10231 service-params-source renaming managed
pkadej Oct 19, 2022
d78199b
PMM-10231 api regenerate
pkadej Oct 19, 2022
850a268
PMM-10231 Name fixes.
pkadej Oct 19, 2022
8760e5b
Merge branch 'main' into PMM-10231_credential-sources
pkadej Oct 20, 2022
1eee71a
PMM-10464 kong updates
pkadej Oct 20, 2022
d8ba120
PMM-10464 kong default values updates
pkadej Oct 20, 2022
2816f80
Merge branch 'main' into PMM-10231_credential-sources
pkadej Oct 21, 2022
2587477
Merge branch 'main' into PMM-10231_credential-sources
pkadej Oct 24, 2022
e54513a
Merge branch 'main' into PMM-10231_credential-sources
pkadej Oct 24, 2022
eaadde0
PMM-10464 Adding TODO.
pkadej Oct 24, 2022
45cce4c
Merge branch 'main' into PMM-10231_credential-sources
pkadej Oct 25, 2022
d7b570f
PMM-10464 Bumping Kong + pointers for username/passwords in pmm-admin.
pkadej Oct 25, 2022
caea892
Merge branch 'main' into PMM-10231_credential-sources
pkadej Oct 26, 2022
8f254ae
Merge branch 'main' into PMM-10231_credential-sources
pkadej Oct 26, 2022
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
24 changes: 24 additions & 0 deletions admin/commands/management/add.go
Expand Up @@ -97,3 +97,27 @@ func processGlobalAddFlagsWithSocket(cmd connectionGetter, opts AddCommonFlags)

return serviceName, socket, host, uint16(portI), nil
}

// Determine if parameters were passed.
var (
usernameParameterSpecified = false
passwordParameterSpecified = false
)

// Types for username and password
type (
username string
password string
)

// AfterApply Change usernameParameterSpecified flag if argument was passed
func (u username) AfterApply() error {
usernameParameterSpecified = true
return nil
}

// AfterApply Change passwordParameterSpecified flag if argument was passed
func (p password) AfterApply() error {
passwordParameterSpecified = true
return nil
}
pkadej marked this conversation as resolved.
Show resolved Hide resolved
21 changes: 2 additions & 19 deletions admin/commands/management/add_external.go
Expand Up @@ -57,7 +57,7 @@ type AddExternalCommand struct {
RunsOnNodeID string `name:"agent-node-id" help:"Node ID where agent runs (default is autodetected)"`
Username string `help:"External username"`
Password string `help:"External password"`
CredentialsSource string `type:"existingfile" help:"Credentials provider"`
ServiceParamsSource string `help:"Path to file with service parameters"`
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Let's please update the help here so it's clear what is expected.
With the current help it's not clear what the file shall contain.

Scheme string `placeholder:"http or https" help:"Scheme to generate URI to exporter metrics endpoints"`
MetricsPath string `placeholder:"/metrics" help:"Path under which metrics are exposed, used to generate URI"`
ListenPort uint16 `placeholder:"port" required:"" help:"Listen port of external exporter for scraping metrics. (Required)"`
Expand All @@ -71,18 +71,6 @@ type AddExternalCommand struct {
SkipConnectionCheck bool `help:"Skip exporter connection checks"`
}

func (cmd *AddExternalCommand) GetCredentials() error {
creds, err := commands.ReadFromSource(cmd.CredentialsSource)
if err != nil {
return fmt.Errorf("%w", err)
}

cmd.Password = creds.Password
cmd.Username = creds.Username

return nil
}

func (cmd *AddExternalCommand) RunCmd() (commands.Result, error) {
customLabels := commands.ParseCustomLabels(cmd.CustomLabels)

Expand Down Expand Up @@ -110,12 +98,6 @@ func (cmd *AddExternalCommand) RunCmd() (commands.Result, error) {
cmd.MetricsPath = fmt.Sprintf("/%s", cmd.MetricsPath)
}

if cmd.CredentialsSource != "" {
if err := cmd.GetCredentials(); err != nil {
return nil, fmt.Errorf("failed to retrieve credentials from %s: %w", cmd.CredentialsSource, err)
}
}

params := &external.AddExternalParams{
Body: external.AddExternalBody{
RunsOnNodeID: cmd.RunsOnNodeID,
Expand All @@ -133,6 +115,7 @@ func (cmd *AddExternalCommand) RunCmd() (commands.Result, error) {
MetricsMode: pointer.ToString(strings.ToUpper(cmd.MetricsMode)),
Group: cmd.Group,
SkipConnectionCheck: cmd.SkipConnectionCheck,
ServiceParamsSource: cmd.ServiceParamsSource,
},
Context: commands.Ctx,
}
Expand Down
21 changes: 2 additions & 19 deletions admin/commands/management/add_external_serverless.go
Expand Up @@ -53,7 +53,7 @@ type AddExternalServerlessCommand struct {
Scheme string `placeholder:"https" help:"Scheme to generate URI to exporter metrics endpoints"`
Username string `help:"External username"`
Password string `help:"External password"`
CredentialsSource string `type:"existingfile" help:"Credentials provider"`
ServiceParamsSource string `help:"Path to file with service parameters"`
Address string `placeholder:"1.2.3.4:9000" help:"External exporter address and port"`
Host string `placeholder:"1.2.3.4" help:"External exporters hostname or IP address"`
ListenPort uint16 `placeholder:"9999" help:"Listen port of external exporter for scraping metrics"`
Expand Down Expand Up @@ -88,18 +88,6 @@ or even you can specify --address instead of host and port as individual paramet
`
}

func (cmd *AddExternalServerlessCommand) GetCredentials() error {
creds, err := commands.ReadFromSource(cmd.CredentialsSource)
if err != nil {
return fmt.Errorf("%w", err)
}

cmd.Password = creds.Password
cmd.Username = creds.Username

return nil
}

func (cmd *AddExternalServerlessCommand) RunCmd() (commands.Result, error) {
customLabels := commands.ParseCustomLabels(cmd.CustomLabels)

Expand All @@ -117,12 +105,6 @@ func (cmd *AddExternalServerlessCommand) RunCmd() (commands.Result, error) {
cmd.MetricsPath = fmt.Sprintf("/%s", cmd.MetricsPath)
}

if cmd.CredentialsSource != "" {
if err := cmd.GetCredentials(); err != nil {
return nil, fmt.Errorf("failed to retrieve credentials from %s: %w", cmd.CredentialsSource, err)
}
}

params := &external.AddExternalParams{
Body: external.AddExternalBody{
AddNode: &external.AddExternalParamsBodyAddNode{
Expand Down Expand Up @@ -151,6 +133,7 @@ func (cmd *AddExternalServerlessCommand) RunCmd() (commands.Result, error) {
MetricsMode: pointer.ToString(external.AddExternalBodyMetricsModePULL),
Group: cmd.Group,
SkipConnectionCheck: cmd.SkipConnectionCheck,
ServiceParamsSource: cmd.ServiceParamsSource,
},
Context: commands.Ctx,
}
Expand Down
21 changes: 2 additions & 19 deletions admin/commands/management/add_haproxy.go
Expand Up @@ -48,7 +48,7 @@ type AddHAProxyCommand struct {
ServiceName string `name:"name" arg:"" default:"${hostname}-haproxy" help:"Service name (autodetected default: ${hostname}-haproxy)"`
Username string `help:"HAProxy username"`
Password string `help:"HAProxy password"`
CredentialsSource string `type:"existingfile" help:"Credentials provider"`
ServiceParamsSource string `help:"Path to file with service parameters"`
Scheme string `placeholder:"http or https" help:"Scheme to generate URI to exporter metrics endpoints"`
MetricsPath string `placeholder:"/metrics" help:"Path under which metrics are exposed, used to generate URI"`
ListenPort uint16 `placeholder:"port" required:"" help:"Listen port of haproxy exposing the metrics for scraping metrics (Required)"`
Expand All @@ -61,18 +61,6 @@ type AddHAProxyCommand struct {
SkipConnectionCheck bool `help:"Skip connection check"`
}

func (cmd *AddHAProxyCommand) GetCredentials() error {
creds, err := commands.ReadFromSource(cmd.CredentialsSource)
if err != nil {
return fmt.Errorf("%w", err)
}

cmd.Password = creds.Password
cmd.Username = creds.Username

return nil
}

func (cmd *AddHAProxyCommand) RunCmd() (commands.Result, error) {
isSupported, err := helpers.IsHAProxySupported()
if !isSupported {
Expand All @@ -95,12 +83,6 @@ func (cmd *AddHAProxyCommand) RunCmd() (commands.Result, error) {
cmd.MetricsPath = fmt.Sprintf("/%s", cmd.MetricsPath)
}

if cmd.CredentialsSource != "" {
if err := cmd.GetCredentials(); err != nil {
return nil, fmt.Errorf("failed to retrieve credentials from %s: %w", cmd.CredentialsSource, err)
}
}

params := &ha_proxy.AddHAProxyParams{
Body: ha_proxy.AddHAProxyBody{
ServiceName: cmd.ServiceName,
Expand All @@ -116,6 +98,7 @@ func (cmd *AddHAProxyCommand) RunCmd() (commands.Result, error) {
CustomLabels: customLabels,
MetricsMode: pointer.ToString(strings.ToUpper(cmd.MetricsMode)),
SkipConnectionCheck: cmd.SkipConnectionCheck,
ServiceParamsSource: cmd.ServiceParamsSource,
},
Context: commands.Ctx,
}
Expand Down
63 changes: 22 additions & 41 deletions admin/commands/management/add_mongodb.go
Expand Up @@ -15,7 +15,6 @@
package management

import (
"fmt"
"strings"

"github.com/AlekSi/pointer"
Expand Down Expand Up @@ -51,15 +50,15 @@ func (res *addMongoDBResult) String() string {

// AddMongoDBCommand is used by Kong for CLI flags and commands.
type AddMongoDBCommand struct {
ServiceName string `name:"name" arg:"" default:"${hostname}-mongodb" help:"Service name (autodetected default: ${hostname}-mongodb)"`
Address string `arg:"" optional:"" help:"MongoDB address and port (default: 127.0.0.1:27017)"`
Socket string `help:"Path to socket"`
NodeID string `help:"Node ID (default is autodetected)"`
PMMAgentID string `help:"The pmm-agent identifier which runs this instance (default is autodetected)"`
Username string `help:"MongoDB username"`
Password string `help:"MongoDB password"`
AgentPassword string `help:"Custom password for /metrics endpoint"`
CredentialsSource string `type:"existingfile" help:"Credentials provider"`
ServiceName string `name:"name" arg:"" default:"${hostname}-mongodb" help:"Service name (autodetected default: ${hostname}-mongodb)"`
Address string `arg:"" optional:"" help:"MongoDB address and port (default: 127.0.0.1:27017)"`
Socket string `help:"Path to socket"`
NodeID string `help:"Node ID (default is autodetected)"`
PMMAgentID string `help:"The pmm-agent identifier which runs this instance (default is autodetected)"`
Username string `help:"MongoDB username"`
Password string `help:"MongoDB password"`
AgentPassword string `help:"Custom password for /metrics endpoint"`
ServiceParamsSource string `help:"Path to file with service parameters"`
// TODO add "auto"
QuerySource string `default:"${mongoDbQuerySourceDefault}" enum:"${mongoDbQuerySourcesEnum}" help:"Source of queries, one of: ${mongoDbQuerySourcesEnum} (default: ${mongoDbQuerySourceDefault})"`
Environment string `help:"Environment name"`
Expand Down Expand Up @@ -101,19 +100,6 @@ func (cmd *AddMongoDBCommand) GetSocket() string {
return cmd.Socket
}

func (cmd *AddMongoDBCommand) GetCredentials() error {
creds, err := commands.ReadFromSource(cmd.CredentialsSource)
if err != nil {
return fmt.Errorf("%w", err)
}

cmd.AgentPassword = creds.AgentPassword
cmd.Password = creds.Password
cmd.Username = creds.Username

return nil
}

func (cmd *AddMongoDBCommand) RunCmd() (commands.Result, error) {
customLabels := commands.ParseCustomLabels(cmd.CustomLabels)

Expand Down Expand Up @@ -144,26 +130,21 @@ func (cmd *AddMongoDBCommand) RunCmd() (commands.Result, error) {
return nil, err
}

if cmd.CredentialsSource != "" {
if err := cmd.GetCredentials(); err != nil {
return nil, fmt.Errorf("failed to retrieve credentials from %s: %w", cmd.CredentialsSource, err)
}
}

params := &mongodb.AddMongoDBParams{
Body: mongodb.AddMongoDBBody{
NodeID: cmd.NodeID,
ServiceName: serviceName,
Address: host,
Port: int64(port),
Socket: socket,
PMMAgentID: cmd.PMMAgentID,
Environment: cmd.Environment,
Cluster: cmd.Cluster,
ReplicationSet: cmd.ReplicationSet,
Username: cmd.Username,
Password: cmd.Password,
AgentPassword: cmd.AgentPassword,
NodeID: cmd.NodeID,
ServiceName: serviceName,
Address: host,
Port: int64(port),
Socket: socket,
PMMAgentID: cmd.PMMAgentID,
Environment: cmd.Environment,
Cluster: cmd.Cluster,
ReplicationSet: cmd.ReplicationSet,
Username: cmd.Username,
Password: cmd.Password,
AgentPassword: cmd.AgentPassword,
ServiceParamsSource: cmd.ServiceParamsSource,

QANMongodbProfiler: cmd.QuerySource == MongodbQuerySourceProfiler,

Expand Down
53 changes: 32 additions & 21 deletions admin/commands/management/add_mysql.go
Expand Up @@ -89,14 +89,15 @@ func (res *addMySQLResult) TablestatStatus() string {

// AddMySQLCommand is used by Kong for CLI flags and commands.
type AddMySQLCommand struct {
ServiceName string `name:"name" arg:"" default:"${hostname}-mysql" help:"Service name (autodetected default: ${hostname}-mysql)"`
Address string `arg:"" optional:"" help:"MySQL address and port (default: 127.0.0.1:3306)"`
Socket string `help:"Path to MySQL socket"`
NodeID string `help:"Node ID (default is autodetected)"`
PMMAgentID string `help:"The pmm-agent identifier which runs this instance (default is autodetected)"`
Username string `default:"root" help:"MySQL username"`
Password string `help:"MySQL password"`
AgentPassword string `help:"Custom password for /metrics endpoint"`
ServiceName string `name:"name" arg:"" default:"${hostname}-mysql" help:"Service name (autodetected default: ${hostname}-mysql)"`
Address string `arg:"" optional:"" help:"MySQL address and port (default: 127.0.0.1:3306)"`
Socket string `help:"Path to MySQL socket"`
NodeID string `help:"Node ID (default is autodetected)"`
PMMAgentID string `help:"The pmm-agent identifier which runs this instance (default is autodetected)"`
Username username `help:"MySQL username"`
Password string `help:"MySQL password"`
AgentPassword string `help:"Custom password for /metrics endpoint"`
ServiceParamsSource string `help:"Path to file with service parameters"`
// TODO add "auto", make it default
QuerySource string `default:"${mysqlQuerySourceDefault}" enum:"${mysqlQuerySourcesEnum}" help:"Source of SQL queries, one of: ${mysqlQuerySourcesEnum} (default: ${mysqlQuerySourceDefault})"`
MaxQueryLength int32 `placeholder:"NUMBER" help:"Limit query length in QAN (default: server-defined; -1: no limit)"`
Expand Down Expand Up @@ -126,6 +127,11 @@ func (cmd *AddMySQLCommand) GetServiceName() string {
return cmd.ServiceName
}

// GetDefaultUsername Default username for mysql connections.
func (cmd *AddMySQLCommand) GetDefaultUsername() string {
pkadej marked this conversation as resolved.
Show resolved Hide resolved
return "root"
}

func (cmd *AddMySQLCommand) GetAddress() string {
return cmd.Address
}
Expand Down Expand Up @@ -180,6 +186,10 @@ func (cmd *AddMySQLCommand) RunCmd() (commands.Result, error) {
}
}

if cmd.ServiceParamsSource == "" && !usernameParameterSpecified {
cmd.Username = username(cmd.GetDefaultUsername())
}

serviceName, socket, host, port, err := processGlobalAddFlagsWithSocket(cmd, cmd.AddCommonFlags)
if err != nil {
return nil, err
Expand All @@ -196,19 +206,20 @@ func (cmd *AddMySQLCommand) RunCmd() (commands.Result, error) {

params := &mysql.AddMySQLParams{
Body: mysql.AddMySQLBody{
NodeID: cmd.NodeID,
ServiceName: serviceName,
Address: host,
Socket: socket,
Port: int64(port),
PMMAgentID: cmd.PMMAgentID,
Environment: cmd.Environment,
Cluster: cmd.Cluster,
ReplicationSet: cmd.ReplicationSet,
Username: cmd.Username,
Password: cmd.Password,
AgentPassword: cmd.AgentPassword,
CustomLabels: customLabels,
NodeID: cmd.NodeID,
ServiceName: serviceName,
Address: host,
Socket: socket,
Port: int64(port),
PMMAgentID: cmd.PMMAgentID,
Environment: cmd.Environment,
Cluster: cmd.Cluster,
ReplicationSet: cmd.ReplicationSet,
Username: string(cmd.Username),
Password: cmd.Password,
AgentPassword: cmd.AgentPassword,
CustomLabels: customLabels,
ServiceParamsSource: cmd.ServiceParamsSource,

QANMysqlSlowlog: cmd.QuerySource == MysqlQuerySourceSlowLog,
QANMysqlPerfschema: cmd.QuerySource == MysqlQuerySourcePerfSchema,
Expand Down