Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Gemini help feature #6937

Draft
wants to merge 5 commits into
base: master
Choose a base branch
from
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
14 changes: 14 additions & 0 deletions npm-shrinkwrap.json

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

1 change: 1 addition & 0 deletions package.json
Expand Up @@ -95,6 +95,7 @@
},
"dependencies": {
"@google-cloud/pubsub": "^3.0.1",
"@google/generative-ai": "^0.3.1",
"abort-controller": "^3.0.0",
"ajv": "^6.12.6",
"archiver": "^5.0.0",
Expand Down
8 changes: 8 additions & 0 deletions src/commands/gemini-help.ts
@@ -0,0 +1,8 @@
import { Command } from "../command";
import * as gemini from "../gemini";

export const command = new Command("gemini:help [prompt]")
.description("Ask gemini a question about how to use firebase-tools")
.action(async (prompt: string) => {
await gemini.ask(prompt);
});
1 change: 1 addition & 0 deletions src/commands/index.ts
Expand Up @@ -3,11 +3,11 @@
/**
* Loads all commands for our parser.
*/
export function load(client: any): any {

Check warning on line 6 in src/commands/index.ts

View workflow job for this annotation

GitHub Actions / lint (20)

Unexpected any. Specify a different type

Check warning on line 6 in src/commands/index.ts

View workflow job for this annotation

GitHub Actions / lint (20)

Unexpected any. Specify a different type
function loadCommand(name: string) {

Check warning on line 7 in src/commands/index.ts

View workflow job for this annotation

GitHub Actions / lint (20)

Missing return type on function
const t0 = process.hrtime.bigint();
const { command: cmd } = require(`./${name}`);

Check warning on line 9 in src/commands/index.ts

View workflow job for this annotation

GitHub Actions / lint (20)

Unsafe assignment of an `any` value

Check warning on line 9 in src/commands/index.ts

View workflow job for this annotation

GitHub Actions / lint (20)

Require statement not part of import statement
cmd.register(client);

Check warning on line 10 in src/commands/index.ts

View workflow job for this annotation

GitHub Actions / lint (20)

Unsafe member access .register on an `any` value

Check warning on line 10 in src/commands/index.ts

View workflow job for this annotation

GitHub Actions / lint (20)

Unsafe call of an `any` typed value
const t1 = process.hrtime.bigint();
const diffMS = (t1 - t0) / BigInt(1e6);
if (diffMS > 75) {
Expand All @@ -15,7 +15,7 @@
// console.error(`Loading ${name} took ${diffMS}ms`);
}

return cmd.runner();

Check warning on line 18 in src/commands/index.ts

View workflow job for this annotation

GitHub Actions / lint (20)

Unsafe return of an `any` typed value

Check warning on line 18 in src/commands/index.ts

View workflow job for this annotation

GitHub Actions / lint (20)

Unsafe member access .runner on an `any` value

Check warning on line 18 in src/commands/index.ts

View workflow job for this annotation

GitHub Actions / lint (20)

Unsafe call of an `any` typed value
}

const t0 = process.hrtime.bigint();
Expand Down Expand Up @@ -138,6 +138,7 @@
client.functions.secrets.prune = loadCommand("functions-secrets-prune");
client.functions.secrets.set = loadCommand("functions-secrets-set");
client.help = loadCommand("help");
client.help = loadCommand("gemini-help");
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not needed for now since this is a draft/POC, but I definitely think we'd want to start this off behind an experiment

client.hosting = {};
client.hosting.channel = {};
client.hosting.channel.create = loadCommand("hosting-channel-create");
Expand Down
80 changes: 80 additions & 0 deletions src/gemini/index.ts
@@ -0,0 +1,80 @@
import { FirebaseError } from "../error";
import { logger } from "../logger";
import * as fs from "fs";
import * as path from "path";
import { promptOnce } from "../prompt";
import { execSync } from "child_process";
import * as clc from "colorette";

const { GoogleGenerativeAI } = require("@google/generative-ai");

// TODO(christhompson): Do we have an endpoint that's open and doesn't require a project w/ billing?

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pretty sure everything is going to require an API Key. You can get one from AIStudio tho pretty easily.

// Estimated QPS is around 1.
// TODO(christhompson): Add project ID for this.
// TODO(christhompson): Add preamble information about command flags.
// TODO(christhompson): Figure out how to test this.
const generationConfig = {

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Give this object a structural type.

maxOutputTokens: 200,
temperature: 0.9,

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Temp seems a little spicy here -- Did you experiment with a different values? My assumption is that if the temp needs to be this high, then we're likely not prompting enough.

topP: 0.1,
topK: 16,
};

const genAI = new GoogleGenerativeAI("AIzaSyDDqii7EVJbMgfC3vhp3ER2o-EYa9qOSQw");

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do you guys use a linter/formatter on the CLI? Generally recommend prettier, but the line length and formatting here seems a bit long.

const model = genAI.getGenerativeModel({ model: "gemini-pro", generationConfig });

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Consider adding "gemini-pro" to "generationConfig"


function getPreamble(): string {
try {
const TEMPLATE_ROOT = path.resolve(__dirname, "../../templates/");

let data = fs.readFileSync(path.join(TEMPLATE_ROOT, "gemini", "preamble.txt"), "utf8");
data = data + fs.readFileSync(path.join(TEMPLATE_ROOT, "gemini", "commandReadme.txt"), "utf8");
return data;
} catch (err) {
throw new FirebaseError("Error reading preamble file" + err);
}
}

async function run(prompt: string): Promise<string> {
let result;
try {
const newPrompt = getPreamble() + prompt;
logger.debug("Running prompt: " + newPrompt);

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: prefer a template string (for all string concatenations in file.)

logger.debug(`Running prompt: ${newPrompt}`);

result = await model.generateContent(newPrompt);
} catch (error) {
console.error("Promise rejected with error: " + error);
}
const response = await result.response;
return response.text();
}

export const ask = async function (prompt: string) {
const responseText = await run(prompt);
logger.info(clc.bold("Gemini Responded:"));
if (
responseText.length > 0 &&
responseText[0] === "`" &&
responseText[responseText.length - 1] === "`"
) {

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The response you get is actually markdown. It seems like you're relying a bit too much on some specifics in markdown parsing.

The preamble says to give one command and one command only, but we know that might not necessarily be the case.

The ideal would be to AST parse the markdown and then dig out the content of the first codeblock / code snip.

I know that yanking in a markdown parsing lib is hefty, but parsing this out manually yourself is going to be a bit problematic and brittle.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We already have marked as a dependency - consider using it here to parse

// Assume this is a single command with backticks on each side.
const trimmedResponse = responseText.slice(1, responseText.length - 1);

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is there a way to filter or validate this command?

Best: Against a known table

Okay: Validate the format.

logger.info(trimmedResponse);
const runNow = await promptOnce({
type: "confirm",
name: "runNow",
message: `Would you like to run this command?`,
default: true,
});

// Try to add the role to the service account
if (runNow) {
logger.info("Running: " + trimmedResponse);
const newCommandOutput = execSync(trimmedResponse).toString(); // Doesn't output to console correctly
// TODO(christhompson): This doesn't transition well, only good for one-off commands that
// don't spawn long running subprocesses (not like emulators:start).
logger.info(newCommandOutput);
}
} else {
logger.info(responseText);
}
};
135 changes: 135 additions & 0 deletions templates/gemini/commandReadme.txt
@@ -0,0 +1,135 @@
Below are some sample commands. If you're not able to recommend a command, please refer me to https://firebase.google.com/docs/cli.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is this file generated or did you hand craft this?

If you handcrafted it -- It's interesting for a demo, but it might make sense to have some sort of command metadata that lives alongside each and every command -- Then you can weave that metadata together into the prompt format.

Having a central file will be annoying to scale, as people will forget to update it.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks like this includes the output of firebase --help. If you move forward with productionizing this, we should at least add a script to package.json to update this (and a CI test that fails if running that script generates any changes)


## Commands

**The command `firebase --help` lists the available commands and `firebase <command> --help` shows more details for an individual command.**

If a command is project-specific, you must either be inside a project directory with an
active project alias or specify the Firebase project id with the `-P <project_id>` flag.

Below is a brief list of the available commands and their function:

### Configuration Commands

| Command | Description |
| -------------- | ----------------------------------------------------------------------------------------------------------------------------------------------- |
| **login** | Authenticate to your Firebase account. Requires access to a web browser. |
| **logout** | Sign out of the Firebase CLI. |
| **login:ci** | Generate an authentication token for use in non-interactive environments. |
| **login:add** | Authorize the CLI for an additional account. |
| **login:list** | List authorized CLI accounts. |
| **login:use** | Set the default account to use for this project |
| **use** | Set active Firebase project, manage project aliases. |
| **open** | Quickly open a browser to relevant project resources. |
| **init** | Setup a new Firebase project in the current directory. This command will create a `firebase.json` configuration file in your current directory. |
| **help** | Display help information about the CLI or specific commands. |

Append `--no-localhost` to login (i.e., `firebase login --no-localhost`) to copy and paste code instead of starting a local server for authentication. A use case might be if you SSH into an instance somewhere and you need to authenticate to Firebase on that machine.

### Project Management Commands

| Command | Description |
| ------------------------ | ---------------------------------------------------------- |
| **apps:create** | Create a new Firebase app in a project. |
| **apps:list** | List the registered apps of a Firebase project. |
| **apps:sdkconfig** | Print the configuration of a Firebase app. |
| **projects:addfirebase** | Add Firebase resources to a Google Cloud Platform project. |
| **projects:create** | Create a new Firebase project. |
| **projects:list** | Print a list of all of your Firebase projects. |

### Deployment and Local Emulation

These commands let you deploy and interact with your Firebase services.

| Command | Description |
| ----------------------------- | ----------------------------------------------------------------------------------------------------------------------------- |
| **emulators:exec** | Start the local Firebase emulators, run a test script, then shut down the emulators. |
| **emulators:start** | Start the local Firebase emulators. |
| **deploy** | Deploys your Firebase project. Relies on `firebase.json` configuration and your local project folder. |
| **serve** | Start a local server with your Firebase Hosting configuration and HTTPS-triggered Cloud Functions. Relies on `firebase.json`. |
| **setup:emulators:database** | Downloads the database emulator. |
| **setup:emulators:firestore** | Downloads the firestore emulator. |

### App Distribution Commands

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: It might make sense to say "Firebase App Distribution" (etc etc for all product names)


| Command | Description |
| ------------------------------ | ---------------------- |
| **appdistribution:distribute** | Upload a distribution. |

### Auth Commands

| Command | Description |
| --------------- | ------------------------------------------------------ |
| **auth:import** | Batch importing accounts into Firebase from data file. |
| **auth:export** | Batch exporting accounts from Firebase into data file. |

Detailed doc is [here](https://firebase.google.com/docs/cli/auth).

### Realtime Database Commands

| Command | Description |
| ----------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------- |
| **database:get** | Fetch data from the current project's database and display it as JSON. Supports querying on indexed data. |
| **database:set** | Replace all data at a specified location in the current project's database. Takes input from file, STDIN, or command-line argument. |
| **database:push** | Push new data to a list at a specified location in the current project's database. Takes input from file, STDIN, or command-line argument. |
| **database:remove** | Delete all data at a specified location in the current project's database. |
| **database:update** | Perform a partial update at a specified location in the current project's database. Takes input from file, STDIN, or command-line argument. |
| **database:profile** | Profile database usage and generate a report. |
| **database:instances:create** | Create a realtime database instance. |
| **database:instances:list** | List realtime database instances. |
| **database:settings:get** | Read the realtime database setting at path |
| **database:settings:set** | Set the realtime database setting at path. |

### Extensions Commands

| Command | Description |
| ----------------- | ------------------------------------------------------------------------------------------- |
| **ext** | Display information on how to use ext commands and extensions installed to your project. |
| **ext:configure** | Configure an existing extension instance. |
| **ext:info** | Display information about an extension by name (extensionName@x.y.z for a specific version) |
| **ext:install** | Install an extension. |
| **ext:list** | List all the extensions that are installed in your Firebase project. |
| **ext:uninstall** | Uninstall an extension that is installed in your Firebase project by Instance ID. |
| **ext:update** | Update an existing extension instance to the latest version. |

### Cloud Firestore Commands

| Command | Description |
| --------------------- | ------------------------------------------------------------------------------------------------------------------- |
| **firestore:delete** | Delete documents or collections from the current project's database. Supports recursive deletion of subcollections. |
| **firestore:indexes** | List all deployed indexes from the current project. |

### Cloud Functions Commands

| Command | Description |
| ----------------------------- | ------------------------------------------------------------------------------------------------------------ |
| **functions:log** | Read logs from deployed Cloud Functions. |
| **functions:list** | List all deployed functions in your Firebase project. |
| **functions:config:set** | Store runtime configuration values for the current project's Cloud Functions. |
| **functions:config:get** | Retrieve existing configuration values for the current project's Cloud Functions. |
| **functions:config:unset** | Remove values from the current project's runtime configuration. |
| **functions:config:clone** | Copy runtime configuration from one project environment to another. |
| **functions:secrets:set** | Create or update a secret for use in Cloud Functions for Firebase. |
| **functions:secrets:get** | Get metadata for secret and its versions. |
| **functions:secrets:access** | Access secret value given secret and its version. Defaults to accessing the latest version. |
| **functions:secrets:prune** | Destroys unused secrets. |
| **functions:secrets:destroy** | Destroy a secret. Defaults to destroying the latest version. |
| **functions:delete** | Delete one or more Cloud Functions by name or group name. |
| **functions:shell** | Locally emulate functions and start Node.js shell where these local functions can be invoked with test data. |

### Hosting Commands

| Command | Description |
| ------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| **hosting:disable** | Stop serving Firebase Hosting traffic for the active project. A "Site Not Found" message will be displayed at your project's Hosting URL after running this command. |

### Remote Config Commands

| Command | Description |
| ------------------------------ | ---------------------------------------------------------------------------------------------------------- |
| **remoteconfig:get** | Get a Firebase project's Remote Config template. |
| **remoteconfig:versions:list** | Get a list of the most recent Firebase Remote Config template versions that have been published. |
| **remoteconfig:rollback** | Roll back a project's published Remote Config template to the version provided by `--version_number` flag. |

Use `firebase:deploy --only remoteconfig` to update and publish a project's Firebase Remote Config template.

3 changes: 3 additions & 0 deletions templates/gemini/preamble.txt
@@ -0,0 +1,3 @@
I am a user using the firebase command line interface. I am logged in with a project ID and I need help writing valid commands. In this environment I can't render markdown, please put commands on a single line with inline backticks like this: `firebase emulators:start`.

Keep your responses to less than 300 characters.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

When I'm prompting, I generally find that leaving 3-5 examples of queries with the expected result is ideal.

Question: "How do I deploy remote config?"
Answer: *firebase whatever command xyz --blah blah*
Question: "How do I disable hosting?"
Answer: *firebase hosting:disable*
Question: _Actually put the users query here_
Answer: _actually leave this blank_