Skip to content

Commit

Permalink
Firebase Data Connect (#7057)
Browse files Browse the repository at this point in the history
* Decode firemat.yaml

* Show codelenses only folders corresponding to firemat.yaml

* Update tests to reflect config changes

* Apply suggestions from code review

* Remove invocation to missing command

* Enable running queries/mutations in production

* Apply suggestions from code review

* Fuse User Mock view and Arguments view (#837)

* Add Firebase Data Connect view

* Add Instance Picker

* Update codelense logic to use instance picker

* Add instance in status bar

* Add project in statusbar

* Add tests for src/core/config.ts (#825)

* Update config parsing to handle custom operationSet keys

* Rename

* Extract functions

* Rename

* Include Firebase logo next to the selected project

* Merge firemat with master

* Fix codelenses incorrectly showing in the schema folder but not in the operation folder (#841)

* Fix merge error

* W

* Complete execution e2e and fix failing e2e

* Cleanup tests

* Add instance picker e2e

* Remove log and revert .vscode change

* Move pageobjects to a separate folder

* Apply suggestions from code review

* Can pick nothing

* initial lsp implementation

* Add lsp binary

* Add lsp binary

* Fix another vue-compiler dynamic dep

* Emulator should stop taking focus

* Don't pick an instance by default, and show statusbar in red

* fix execute codelens not showing in adhoc folder

* fix merge mistake

* Add start-emulator command

* Instance default to emulator

* Show execution codelens even if emulator isn't started

* Auto-start emulator when targeting emulator but it isn't started

* Add support for adhoc files. Update emulator to 0.1.4. Small fix to ls refresh

* update emulator file

* add dataconnect icon to statusbar. Change statusbar color

* Minor ui updates

* Commit

* Delete unused conf

* Fix e2e

* Add some e2e tests

* Fix tests

* fix execute codelens not showing in adhoc folder

* Prototype `deploy --only dataconnect` (#820)

* Scaffolding code and implementing list/createService

* Simple implementation of reading firemat.yaml and deploying schemas/operation sets

* More progress

* Adding source.ts

* Format and clean up bad merge

* cleaned up wrong tests

* Successfully deploying schemas, added cloudSql client

* E2E deploys working, cleaned up some cruft

* Add ensureApiEnabled, more cleanup

* Update src/commands/init.ts

Co-authored-by: Michael Bleigh <bleigh@google.com>

* remove listSchemas

* trying to get install from git url working

* nope

* Another try

* also no

* Another round of cleanup

* Cleaning up

* starting updates to match dataconect.yaml

* Fixed up deploy flow to read and use approved dataconnect.yaml

* Renaming it all to firemat

* one last spot

* Updating init as well

* Updating json schema

* one last spot

* self review fixes

* simplify init template

* Format and cleanup

* udpated json schema

* Update src/experiments.ts

Co-authored-by: Michael Bleigh <bleigh@google.com>

* Update src/emulator/constants.ts

Co-authored-by: Michael Bleigh <bleigh@google.com>

* PR fixes

* cleanup

* format

---------

Co-authored-by: Michael Bleigh <bleigh@google.com>

* Fix Emulators.FIREMAT rename

* Rename emulator to Dataconnect (#869)

The emulator was renamed in a previous commit, but it appears that there were some leftovers.

* Adding a parser for connector and service names (#866)

* Adding a parser for connector and service names

* no only

* update shrinkwrap

* Rename instance

* Update test

* More test update

* Update fishfood e2e

* Update e2e

* Fix emulator

* Fix console errors

* Fix path

* Fix test

* Fix inquirer error

* Rename folder

* Firemat > Data Connect

* webview id fix

* Revert firemat.yaml > dataconnect.yaml for now

* Format

* ignore wdio for vsix package

* Add initial deploy flow

* Add `dataconnect.list` command (#868)

* starting to implement dataconnect:list

* adding command file

* Adding a parser for connector and service names

* no only

* Adding dataconect:list command

* Add --json support

* silence warnings

* More silencing of warnings

* Pr fixes

* Update queries.gql (#889)

Turns out graphql-js is unable to parse empty argument lists

* Fix config file

* Adding a basic cloudsqladmin client (#879)

* starting to implement dataconnect:list

* adding command file

* Adding a parser for connector and service names

* no only

* Adding dataconect:list command

* Add --json support

* Adding a basic clodusqladmin client

* Minor cleanups

* formatting

* Delete is nonstandard

* format

* Connect to CloudSQL as a builtin user or service account (#878)

* starting to implement dataconnect:list

* adding command file

* Adding dataconect:list command

* Add --json support

* Adding a basic clodusqladmin client

* Minor cleanups

* formatting

* Delete is nonstandard

* Prototyping provisioning + connecting to an instance!

* Merge conflict

* Merge conflict

* Connecting via IAM users woprking with ADC auth

* format

* cleanup

* clean up!

* Provision CloudSQL resources on init and deploy (#882)

* starting to implement dataconnect:list

* adding command file

* Adding a parser for connector and service names

* no only

* Adding dataconect:list command

* Add --json support

* Adding a basic clodusqladmin client

* Minor cleanups

* formatting

* Delete is nonstandard

* Prototyping provisioning + connecting to an instance!

* Merge conflict

* Merge conflict

* Connecting via IAM users woprking with ADC auth

* format

* cleanup

* Added cloudSQL provisioning to deploy and init

* fixing instanceId

* Clean up

* npm-shrikwrapped

* Remove dependency on Option from readFirebaseJson (#893)

* Update configs

* Convert paths to absolute

* Fix path

* Fix move to connector

* location > source

* Cleanup

* Bump FDC emulator versions.

* In production, show connector ID

* Kinda update tes

* Fix FDC emulator absolute path.

* Remove log import.

* Lint.

* Fixing a bug from bugbash

* it builds now

* Pass the PG local connection string from an env variable to the emulator.

* Add utils for getting services and connectors. Service file uses correct serviceId. Small refactor

* Separate utils file to file-utils and config

* Pass `local_connection_string` flag to emulator.

* initial deploy commit

* pass config options to deploy

* Update deploy to pick from services, then connectors

* fix deploy

* resolve comments

* Default should be empty string.

* Ran npm run generate:json-schema

* Bump emulator version.

* Implement dataconnect:sql:diff and dataconnect:sgl:migrate (#897)

* Start scaffoling out code for schema migration

* progress on schema migration

* progress

* First time migrating a schema!!

* Adding a SQL formatter and breaking things

* Big improvements to permissions logic

* formats

* Self review

* fiddling with grants

* more cleanup and edge case fixing

* Run npm run generate:json-schema.

* Refactor "move to connector" to an assist

* initial stream compile errors implementation

* Handle multiple connectors for moveToConnector

* forgot to remove a test user

* forgot to remove a test user

* fixing undefined access error

* Validate and patch existing Cloud SQL instances.

* emulator bump

* Cleaning up and running a formatter.

* npm run format

* One more lint error.

* Linting one more time.

* Added support for connecting to cloudSQL using user accounts (#908)

* Added support for connecting to cloudSQL using user accounts

* finshing my sentences

* Add extra sequences grant, make URL optional

* More cleanup

* Address review comments.

* npm run lint:changed-files -- --fix

* emulator bump

* update api ver

* Re-add dataConnect specific api values as getters

* Fix path typo (#913)

* Fix sample project

* rename

* Cleanup

* Fix errors

* Revamp side panel to include postgres setter

* update dc staging api

* update staging url to use https

* ls package fixes

* dataconnect api flag

* update vscode env vars

* cli update

* Update src/utils.ts

Co-authored-by: joehan <joehanley@google.com>

* Tweak datasource in dataconnect:list (#909)

* format

* Adding schema files for dataconnect.yaml and connector.yaml (#918)

* Adding schema files for dataconnect.yaml and connector.yaml

* Fixed some errors

* add schema to contributes

* Perform schema migration during deploy as well (#917)

* Starting on adding schema migration to deploy

* Fixing force behavior

* format

* self review

* PR suggestions + 2 way choice when not destructive

* Make serviceID optional, prompt for deleting services (#925)

* Fix and refactor tests

* Updating firebase.json to accept array of dataconnects (#931)

* Updating firebase.json to accept array of dataconnects

* Fix type usage

* ===

* Add localConectionString into .firebaserc (#929)

* dataconnect production api

* add missing files

* change string concat

* format

* add language-service file

* fast fix for dataConnectSingle type

* typing fix for DataconnectMultiple

* remove ui for adding psql conn string

* address comments

* Bump FDC emulator version.

* Create CloudSQL instances with configuration needed for FDC free trial.

* Adding support for --only flags (#927)

* Adding support for --only falgs

* Minor cleanups

* Refactor for clarity

* deep equal

* deploy using a terminal

* respond to comments

* Remove backupConfiguration enablement, it already defaults to false.

* Add `dataconnect:sdk:generate` command (#934)

* Starting on generate

* Updating firebase.json to accept array of dataconnects

* More progress on generate

* Updating firebase.json to accept array of dataconnects

* Fix type usage

* ===

* Still getting enoent errors, unsure why

* Working!

* format

* Remove "connect to instance" logic and show running emulators in the status bar. (#947)

* Fixed package-lock and updated to use new emulator

* Refactor --force behavior match Freds proposal (#946)

* Refactor --force behavior match Freds proposal

* PR fixes

* adding comments

* Update emulator

* Add GraphQL.vscode-graphql-syntax as extension dependency (#948)

* Remove duplicate

* Updating emulator

* Correctly watch for dataconnect.yaml changes (#949)

* Validate local schema and connectors together before deploy (#952)

* Starting on build

* types

* Progress!

* Prettify errors

* Line numbers, and save deployment metadata

* Extremely defensive

* Update FDC free trial tag per latest decision. (#953)

* Only one free trial per project, please. (#955)

* Strating on pricing warning/free trial

* waiting until sync to continue

* Added free trial check

* clean up console.log

* Handle malformed dataconnect.yaml files (#950)

* Show FDC views only if FDC is enabled

* Enable Goole ML Integration if vector is used (#954)

* Code cleanup + add vector provisioning

* PR fixes

* Rebase

* Rebase

* Clean up todos

* PR fixes

* Use require vector everywhere

* format

* Update FDC Deploy view (#958)

* Starting point for piping logs to VSCode (#944)

* Starting point for piping logs to VSCode

* Add vsceLogEmitter

* format

* Bump emulator version. (#959)

* Bump emulator version.

* Enable size and checksum check for FDC emulator.

* Add sslMode=disable to default local conn string (#960)

* Added simple smoke test for deploy --only dataconnect (#840)

* Scaffolding code and implementing list/createService

* Simple implementation of reading firemat.yaml and deploying schemas/operation sets

* More progress

* Adding source.ts

* Format and clean up bad merge

* cleaned up wrong tests

* Successfully deploying schemas, added cloudSql client

* E2E deploys working, cleaned up some cruft

* Add ensureApiEnabled, more cleanup

* Update src/commands/init.ts

Co-authored-by: Michael Bleigh <bleigh@google.com>

* remove listSchemas

* trying to get install from git url working

* nope

* Another try

* also no

* Another round of cleanup

* Cleaning up

* Sketched up a dead simple integration/load test

* Updating to match new format

* Fixed this up to match new semantics

* formats

* Adding actual validation of what is deployed

* Made this way less hacky and easier to add more cases to in the future

* PR fixes

---------

Co-authored-by: Michael Bleigh <bleigh@google.com>

* add deploy all

* address comments

* Pass non demo- project id to emulator

* fix parse

* Adding shell:true to dataconnect emulator

---------

Co-authored-by: Remi Rousselet <darky12s@gmail.com>
Co-authored-by: Harold Shen <hlshen@google.com>
Co-authored-by: Michael Bleigh <bleigh@google.com>
Co-authored-by: TJ Lavelle <tlavelle@google.com>
Co-authored-by: Rosalyn Tan <rosalyntan@google.com>
Co-authored-by: Maneesh Tewani <mtewani@google.com>
  • Loading branch information
7 people committed Apr 26, 2024
1 parent 3a716c9 commit 24accb1
Show file tree
Hide file tree
Showing 201 changed files with 19,882 additions and 12,904 deletions.
2 changes: 2 additions & 0 deletions .eslintrc.js
Expand Up @@ -130,5 +130,7 @@ module.exports = {
"src/emulator/auth/schema.ts",
// TODO(hsubox76): Set up a job to run eslint separately on vscode dir
"firebase-vscode/",
// If this is leftover from "clean-install.sh", don't lint it
"clean/**",
],
};
40 changes: 18 additions & 22 deletions firebase-vscode/.eslintrc.json
@@ -1,24 +1,20 @@
{
"root": true,
"parser": "@typescript-eslint/parser",
"parserOptions": {
"ecmaVersion": 6,
"sourceType": "module"
},
"plugins": [
"@typescript-eslint"
// "react"
],
"rules": {
"@typescript-eslint/semi": "warn",
"curly": "warn",
"eqeqeq": "warn",
"no-throw-literal": "warn",
"semi": "off"
},
"ignorePatterns": [
"out",
"dist",
"**/*.d.ts"
]
"root": true,
"parser": "@typescript-eslint/parser",
"parserOptions": {
"ecmaVersion": 6,
"sourceType": "module"
},
"plugins": [
"@typescript-eslint"
// "react"
],
"rules": {
"@typescript-eslint/semi": "warn",
"curly": "warn",
"eqeqeq": "warn",
"no-throw-literal": "warn",
"semi": "off"
},
"ignorePatterns": ["out", "dist", "**/*.d.ts"]
}
3 changes: 2 additions & 1 deletion firebase-vscode/.gitignore
Expand Up @@ -4,4 +4,5 @@ dist/
resources/dist
.vscode-test
.wdio-vscode-service
logs
logs
!*.tgz
10 changes: 10 additions & 0 deletions firebase-vscode/.prettierignore
@@ -0,0 +1,10 @@
## The default
**/.git
**/.svn
**/.hg
**/node_modules

## The good stuff
dist
resources
package-lock.json
4 changes: 1 addition & 3 deletions firebase-vscode/.vscode/extensions.json
@@ -1,7 +1,5 @@
{
// See http://go.microsoft.com/fwlink/?LinkId=827846
// for the documentation about the extensions.json format
"recommendations": [
"dbaeumer.vscode-eslint"
]
"recommendations": ["dbaeumer.vscode-eslint"]
}
12 changes: 3 additions & 9 deletions firebase-vscode/.vscode/launch.json
Expand Up @@ -9,12 +9,8 @@
"name": "Run Extension",
"type": "extensionHost",
"request": "launch",
"args": [
"--extensionDevelopmentPath=${workspaceFolder}"
],
"outFiles": [
"${workspaceFolder}/dist/**/*.js"
],
"args": ["--extensionDevelopmentPath=${workspaceFolder}"],
"outFiles": ["${workspaceFolder}/dist/**/*.js"],
"preLaunchTask": "${defaultBuildTask}"
},
{
Expand All @@ -25,9 +21,7 @@
"--extensionDevelopmentPath=${workspaceFolder}",
"--extensionTestsPath=${workspaceFolder}/dist/test/suite/index"
],
"outFiles": [
"${workspaceFolder}/dist/test/**/*.js"
],
"outFiles": ["${workspaceFolder}/dist/test/**/*.js"],
"preLaunchTask": "${defaultBuildTask}"
}
]
Expand Down
5 changes: 4 additions & 1 deletion firebase-vscode/.vscodeignore
Expand Up @@ -17,4 +17,7 @@ webpack.*.js
../
*.zip
node_modules/
dist/test/
dist/test/
*.tgz
package-lock.json
.wdio-vscode-service/
28 changes: 28 additions & 0 deletions firebase-vscode/common/error.ts
@@ -0,0 +1,28 @@
/** An error thrown before the GraphQL operation could complete.
*
* This could include HTTP errors or JSON parsing errors.
*/
export class DataConnectError extends Error {
constructor(message: string, cause?: unknown) {
super(message, { cause });
}
}

/** Encode an error into a {@link SerializedError} */
export function toSerializedError(error: Error): SerializedError {
return {
name: error.name,
message: error.message,
stack: error.stack,
cause:
error.cause instanceof Error ? toSerializedError(error.cause) : undefined,
};
}

/** An error object that can be sent across webview boundaries */
export interface SerializedError {
name?: string;
message: string;
stack?: string;
cause?: SerializedError;
}
63 changes: 63 additions & 0 deletions firebase-vscode/common/graphql.ts
@@ -0,0 +1,63 @@
import { ExecutionResult, GraphQLError } from "graphql";

/** Asserts that an unknown object is a {@link ExecutionResult} */
export function assertExecutionResult(
response: any
): asserts response is ExecutionResult {
if (!response) {
throw new Error(`Expected ExecutionResult but got ${response}`);
}

const type = typeof response;
if (type !== "object") {
throw new Error(`Expected ExecutionResult but got ${type}`);
}

const { data, errors } = response;
if (!data && !errors) {
throw new Error(
`Expected ExecutionResult to have either "data" or "errors" set but none found`
);
}

if (errors) {
if (!Array.isArray(errors)) {
throw new Error(
`Expected errors to be an array but got ${typeof errors}`
);
}
for (const error of errors) {
assertGraphQLError(error);
}
}
}

export function isExecutionResult(response: any): response is ExecutionResult {
try {
assertExecutionResult(response);
return true;
} catch {
return false;
}
}

/** Asserts that an unknown object is a {@link GraphQLError} */
export function assertGraphQLError(
error: unknown
): asserts error is GraphQLError {
if (!error) {
throw new Error(`Expected GraphQLError but got ${error}`);
}

const type = typeof error;
if (type !== "object") {
throw new Error(`Expected GraphQLError but got ${type}`);
}

const { message } = error as GraphQLError;
if (typeof message !== "string") {
throw new Error(
`Expected GraphQLError to have "message" set but got ${typeof message}`
);
}
}
2 changes: 1 addition & 1 deletion firebase-vscode/common/messaging/broker.ts
Expand Up @@ -92,7 +92,7 @@ export function createBroker<
on<E extends keyof IncomingMessages>(
message: Extract<E, string>,
listener: (params: IncomingMessages[E]) => void
) {
): () => void {
return broker.addListener(message, listener);
},
delete(): void {
Expand Down
70 changes: 58 additions & 12 deletions firebase-vscode/common/messaging/protocol.ts
Expand Up @@ -8,6 +8,28 @@ import { User } from "../../../src/types/auth";
import { ServiceAccountUser } from "../types";
import { RCData } from "../../../src/rc";
import { EmulatorUiSelections, RunningEmulatorInfo } from "./types";
import { ExecutionResult } from "graphql";
import { SerializedError } from "../error";

export const DEFAULT_EMULATOR_UI_SELECTIONS: EmulatorUiSelections = {
projectId: "demo-something",
importStateFolderPath: "",
exportStateOnExit: false,
mode: "dataconnect",
debugLogging: false,
};

export enum UserMockKind {
ADMIN = "admin",
UNAUTHENTICATED = "unauthenticated",
AUTHENTICATED = "authenticated",
}
export type UserMock =
| { kind: UserMockKind.ADMIN | UserMockKind.UNAUTHENTICATED }
| {
kind: UserMockKind.AUTHENTICATED;
claims: string;
};

export interface WebviewToExtensionParamsMap {
/**
Expand All @@ -17,6 +39,13 @@ export interface WebviewToExtensionParamsMap {
addUser: {};
logout: { email: string };

/* Emulator panel requests */
getEmulatorUiSelections: void;
getEmulatorInfos: void;
updateEmulatorUiSelections: Partial<EmulatorUiSelections>;
/* Equivalent to the `firebase emulators:start` command.*/
launchEmulators: void;

/** Notify extension that current user has been changed in UI. */
requestChangeUser: { user: User | ServiceAccountUser };

Expand Down Expand Up @@ -64,27 +93,45 @@ export interface WebviewToExtensionParamsMap {
href: string;
};

/**
* Equivalent to the `firebase emulators:start` command.
*/
launchEmulators: {
emulatorUiSelections: EmulatorUiSelections;
};

/** Stops the emulators gracefully allowing for data export if required. */
stopEmulators: {};
stopEmulators: void;

selectEmulatorImportFolder: {};

definedDataConnectArgs: string;

/** Prompts the user to select a directory in which to place the quickstart */
chooseQuickstartDir: {};

notifyAuthUserMockChange: UserMock;

/** Deploy connectors/services to production */
"fdc.deploy": void;

/** Deploy all connectors/services to production */
"fdc.deploy-all": void;
}

export interface DataConnectResults {
query: string;
displayName: string;
results?: ExecutionResult | SerializedError;
args?: string;
}

export type ValueOrError<T> =
| { value: T; error: undefined }
| { error: string; value: undefined };

export interface ExtensionToWebviewParamsMap {
/** Triggered when the emulator UI/state changes */
notifyEmulatorUiSelectionsChanged: EmulatorUiSelections;
notifyEmulatorStateChanged: {
status: "running" | "stopped" | "starting" | "stopping";
infos: RunningEmulatorInfo | undefined;
};
notifyEmulatorImportFolder: { folder: string };

/** Triggered when new environment variables values are found. */
notifyEnv: { env: { isMonospace: boolean } };

Expand Down Expand Up @@ -136,10 +183,9 @@ export interface ExtensionToWebviewParamsMap {
*/
notifyPreviewChannelResponse: { id: string };

notifyEmulatorsStopped: {};
notifyEmulatorStartFailed: {};
notifyRunningEmulatorInfo: RunningEmulatorInfo;
notifyEmulatorImportFolder: { folder: string };
// data connect specific
notifyDataConnectResults: DataConnectResults;
notifyDataConnectRequiredArgs: { args: string[] };
}

export type MessageParamsMap =
Expand Down
2 changes: 1 addition & 1 deletion firebase-vscode/common/messaging/types.d.ts
Expand Up @@ -30,6 +30,6 @@ export interface EmulatorUiSelections {
firebaseJsonPath?: string;
importStateFolderPath?: string;
exportStateOnExit: boolean;
mode: "hosting" | "all";
mode: "hosting" | "all" | "dataconnect";
debugLogging: boolean;
}
4 changes: 2 additions & 2 deletions firebase-vscode/common/types.d.ts
@@ -1,8 +1,8 @@
export interface ServiceAccount {
user: ServiceAccountUser
user: ServiceAccountUser;
}

export interface ServiceAccountUser {
email: string;
type: 'service_account'
type: "service_account";
}
Binary file not shown.
Binary file not shown.

0 comments on commit 24accb1

Please sign in to comment.