[BUG] More than one service binding to cloud-service-broker app causes app to crash

[xyloman]

Description

We have been using spring cloud config to provide secrets to the cloud service broker (specifically the broker api username nad password that will also be registered with the cloud controller api). It appears that VCAP_SERVICES can come back with service instances in different orders but it appears that cloud-service-broker only attempts to resolve the first bound service instance instead of leveraging tags.

Expected Behavior

Allow more than just the database be bound to a application instance

Actual Behavior

Random crashes of the application with logs:

   2021-02-23T15:46:25.83-0600 [APP/PROC/WEB/0] OUT {"timestamp":"1614116785.835094929","source":"cloud-service-broker","message":"cloud-service-broker.Invalid VCAP_SERVICES environment variable - falling back to explicit environment variables","log_level":1,"data":{}}
   2021-02-23T15:46:25.83-0600 [APP/PROC/WEB/0] OUT {"timestamp":"1614116785.835577965","source":"cloud-service-broker","message":"cloud-service-broker.Database Setup","log_level":2,"data":{"error":"DB_HOST, DB_USERNAME and DB_PASSWORD are required environment variables."}}
   2021-02-23T15:46:25.83-0600 [APP/PROC/WEB/0] ERR {"timestamp":"1614116785.835577965","source":"cloud-service-broker","message":"cloud-service-broker.Database Setup","log_level":2,"data":{"error":"DB_HOST, DB_USERNAME and DB_PASSWORD are required environment variables."}}

Possible Fix

Inspect the VCAP_SERVICES and look for service instances with the tag "tags":["mysql"]

Steps to Reproduce

1. Bind another service instance to the broker and restage
2. Might have to cycle a number of times before the broker will crash

Context

Unable to store secrets that are used by the broker securely.

Your Environment

• Version used: 0.2.3
• Operating System and version (desktop): TAS 2.10.10
• Platform (Azure/AWS/GCP): AWS
• Applicable Services: N/A

[erniebilling]

@xyloman can you provide the result of cf env for the service broker so we can see what VCAP_SERVICES looks like?

The broker does use tags for mysql and the error indicates there was some problem parsing the VCAP_SERVICES json.

[xyloman]

Sorry for the delay:

VCAP_SERVICES: {
 "p.config-server": [
  {
   "binding_name": null,
   "credentials": {
    "credhub-ref": "/c/p.spring-cloud-services-scs-service-broker/e9ca123d-c312-4f57-88a4-aef09e887f83/9bf8581a-e760-4c2d-84bb-534b4dabbe6f/credentials-json"
   },
   "instance_name": "config-server",
   "label": "p.config-server",
   "name": "config-server",
   "plan": "standard",
   "provider": null,
   "syslog_drain_url": null,
   "tags": [
    "configuration",
    "spring-cloud"
   ],
   "volume_mounts": []
  }
 ],
 "p.mysql": [
  {
   "binding_name": null,
   "credentials": {
    "credhub-ref": "/c/548966e5-e333-4d65-8773-7b4e3bb6ca97/d33ef795-c769-4c03-ba3b-2387d5541890/8ca79dac-75b0-4ea7-bbfa-4a307de05dd4/credentials"
   },
   "instance_name": "csb-db",
   "label": "p.mysql",
   "name": "csb-db",
   "plan": "db-small",
   "provider": null,
   "syslog_drain_url": null,
   "tags": [
    "mysql"
   ],
   "volume_mounts": []
  }
 ]
}

[cf-gitbot]

We have created an issue in Pivotal Tracker to manage this:

https://www.pivotaltracker.com/story/show/182332915

The labels on this github issue will be updated when the story is started.

[pivotal-marcela-campo]

@xyloman we won't get to this anytime soon. If you to submit a PR we would be very happy to review.