Enable json config options and non-interactive mode #1384

raymondfeng · 2018-05-30T22:15:19Z

The PR adds the following features:

Allow options to be passed in via --config from a json file, stdin, or stringified value
Allow --yes option so that optional questions will be skipped with default values

For example,

lb4 app my-app --yes
lb4 app --config config.json
lb4 app --config {"name":"my-app"}
cat config.json | lb4 app --config stdin
lb4 app --config stdin < config.json
lb4 app --config stdin << EOF
> {"name":"my-app"}
> EOF

Checklist

npm test passes on your machine
New tests added or existing tests modified to cover all changes
Code conforms with the style guide
API Documentation in code was updated
Documentation in /docs/site was updated
Affected artifact templates in packages/cli were updated
Affected example projects in examples/* were updated

bajtos · 2018-05-31T10:09:29Z

packages/cli/lib/base-generator.js

@@ -21,11 +22,113 @@ module.exports = class BaseGenerator extends Generator {
   * Subclasses can extend _setupGenerator() to set up the generator
   */
  _setupGenerator() {
+    this.option('json', {
+      type: String,
+      description: 'Stringified JSON object to configure options',


It makes me wonder how practical it is to pass JSON strings via exec arguments. IIRC, there are limitations on the maximum supported length - e.g. Linux limits the length to ~100KB according to this post, Windows has a limit of 32KB according to this post.

In my experience, CLI programs usually accept large content from stdin stream.

I am proposing to slightly different set of CLI options and behavior:

# Read JSON instructions from the given file $ lb4 app --json commands.json # No file was provided, read instruction from stdin $ cat commands.json | lb4 app --json

Thoughts?

bajtos · 2018-05-31T10:10:24Z

packages/cli/lib/base-generator.js

+    this.option('express', {
+      type: Boolean,
+      description:
+        'Run the generator in express mode to skip optional questions',


When I first saw the name of the options, I assumed express would be related to Express.js framework.

Can we find a better option name please, for example --non-interactive or --skip-prompts?

bajtos

Besides my comments below, please add documentation explaining how to use the new express/non-interactive mode. ~~Ideally, the pull request description should be updated to match the actual implementation too.~~

bajtos · 2018-06-14T08:44:33Z

packages/cli/lib/artifact-generator.js

+   * keyword 'loopback' under 'keywords' attribute in package.json.
+   * 'keywords' is an array
+   */
+  checkLoopBackProject() {


Don't we already have another implementation of checkLoopBackProject? I think @virkt25 was using one in his datasource work. Let's create a single implementation of this check and move it to a place where all generators can import it from.

Hmm, all the artifact generators call super.checkLoopBackProject() which is implemented in BaseGenerator which is what this generator (ArtifactGenerator) extends. This implementation seems to be the same as the one found in the base generator so not sure why it's duplicated.

@raymondfeng PTAL ☝️

bajtos · 2018-06-14T08:46:22Z

packages/cli/lib/base-generator.js

+    if (process.stdin.isTTY) {
+      this.log(
+        chalk.green(
+          'Please type in a json object line by line ' +


Uh oh, do you really expect people to type JSON by hand? What if they make a typo, e.g. forget to wrap a key in double quotes - how are they going to fix the type? Will they have to re-type everything again?

In my opinion, adding readline support is overkill. The JSON mode is intended for machines, humans should use interactive prompt mode.

@raymondfeng PTAL ☝️

I'm thinking about copy-paste of json objects into stdin.

Please also note the readline support is needed for unix pipe or redirection, for example:

cat config.json | lb4 app --config stdin

lb4 app --config stdin < config.json

lb4 app --config stdin <<EOF

The generator needs to read from stdin - which is the source of input from unix pipes. The readline module is used to enable that.

In Node.js, process.stdin is a regular stream that you can read the input from. No need to wrap it in readline, unless I am missing something specific to how Yeoman modifies stdin.

A typical solution for reading all input from stdin into a string is to pipe stdin to concat-stream.

A mock-up:

const concat = require('concat-stream'); // NOTE: returns Promise<Buffer>, not Promise<string>! function readStream(src) { return new Promise((resolve, reject) => { const concatStream = concat(resolve); src('error', reject); src.pipe(concatStream); }); } // usage const rawJsonBuffer = await readStream(stdin); const data = JSON.parse(rawJsonBuffer); // fortunately, JSON.parse works with buffers too

Well, that's just a different way of implementing readJSONFromStdin. Is leveraging readline to read from stdin a bad idea?

The other option is the use the following code:

function readBufferFromStdin() { const stdin = process.stdin; const data = []; let len = 0; return new Promise((resolve, reject) => { stdin.on('readable', () => { let chunk; while ((chunk = stdin.read())) { data.push(chunk); len += chunk.length; } }); stdin.on('end', () => { resolve(Buffer.concat(data, len)); }); stdin.on('error', err => { reject(err); }); }); };

Well, that's just a different way of implementing readJSONFromStdin. Is leveraging readline to read from stdin a bad idea?

I suppose using readline is not a bad idea per se. My main objection is against the complexity of the current implementation of _readJSONFromStdin. To me, the currently proposed code requires maintainers to know a lot about readline, TTY, SIGINT, etc. and the way how all things are wired together.

Compare it with my proposed solution that's just 5 lines of relatively straightforward code.

return new Promise((resolve, reject) => { const concatStream = concat(resolve); src.on('error', reject); src.pipe(concatStream); });

Your second example, using ReadableStream directly, looks better to me than the original readline-based one. But why to re-implement reading a stream into a buffer when concat-stream module already does it well? Dealing with streams is tricky and it's very easy to introduce subtle bugs that are difficult to spot by casual readers.

Not a big deal though. If you prefer a hand-written readBufferFromStdin then I am ok with that too.

bajtos · 2018-06-14T08:50:30Z

packages/cli/lib/base-generator.js

+      q.type === 'confirm'; // A confirmation
+
+    // Get the default answer for a question
+    const defaultAnswer = async q => {


async function defaultAnswer(q) { // ... }

The implementation of this function is way too long, please refactor it by extracting bits and pieces into standalone functions.

bajtos · 2018-06-14T08:50:48Z

packages/cli/lib/base-generator.js

+        return defaultVal != null ? defaultVal : true;
+      }
+      if (q.type === 'list' || q.type === 'rawList') {
+        // Default to 1st item


Handling of list/rawList is a prime candidate to be extracted into a new function.

bajtos · 2018-06-14T08:51:19Z

packages/cli/lib/base-generator.js

+          }
+        }
+      } else if (q.type === 'checkbox') {
+        if (def == null) {


Ditto here - please extract checkbox case into a function.

virkt25

Great PR 👏 ! I'm excited to see this PR land in the CLI :D -- Just a few quick comments but looks great otherwise.

virkt25 · 2018-06-15T19:34:41Z

packages/cli/lib/artifact-generator.js

+   * keyword 'loopback' under 'keywords' attribute in package.json.
+   * 'keywords' is an array
+   */
+  checkLoopBackProject() {


Hmm, all the artifact generators call super.checkLoopBackProject() which is implemented in BaseGenerator which is what this generator (ArtifactGenerator) extends. This implementation seems to be the same as the one found in the base generator so not sure why it's duplicated.

virkt25 · 2018-06-16T02:06:00Z

packages/cli/lib/base-generator.js

+    if (process.stdin.isTTY) {
+      this.log(
+        chalk.green(
+          'Please type in a json object line by line ' +


virkt25 · 2018-06-16T02:14:03Z

packages/cli/lib/base-generator.js

+
+    this.option('skip-optional-prompts', {
+      type: Boolean,
+      alias: 'b',


Why is the alias for this prompt b? I'd image maybe s for skip?

virkt25 · 2018-06-16T02:15:26Z

packages/cli/lib/base-generator.js

+      description: 'JSON file to configure options',
+    });
+
+    this.option('skip-optional-prompts', {


I'm not sure how I feel about a such a long flag name from a UX perspective. Can we consider a simpler / easier to type / use / remember flag such as --skip or --skip-optional or --express (but may be confusing with Express framework) or --quick or --defaults ... anything smaller than the current flag.

I was proposing --non-interactive, but that name has a slightly different meaning.

To be honest, I don't see much value in --skip-default-prompts unless it's used with the JSON mode. I am proposing to merge these two modes together and use the --config option to drive the behavior.

Answers are read from JSON - use non-interactive mode (skip default prompts)

Answers are read from UI prompts - use interactive mode (ask all prompts)

That way there is only one new CLI option to add (--config).

bajtos

FWIW, there are few older comments waiting to be addressed or discussed.

bajtos · 2018-06-26T14:13:46Z

packages/cli/lib/artifact-generator.js

+   * keyword 'loopback' under 'keywords' attribute in package.json.
+   * 'keywords' is an array
+   */
+  checkLoopBackProject() {


@raymondfeng PTAL ☝️

bajtos · 2018-06-26T14:13:51Z

packages/cli/lib/base-generator.js

+    if (process.stdin.isTTY) {
+      this.log(
+        chalk.green(
+          'Please type in a json object line by line ' +


@raymondfeng PTAL ☝️

raymondfeng · 2018-06-26T16:17:15Z

The checkLoopBackProject is a duplicate from merging. Will remove it.

raymondfeng · 2018-06-29T16:48:17Z

After switching to stream-based implementation of reading stdin, the following error is thrown:

tty.js:59
  this._handle.setRawMode(flag);
               ^

TypeError: Cannot read property 'setRawMode' of null
    at ReadStream.setRawMode (tty.js:59:16)
    at Interface._setRawMode (readline.js:243:16)
    at new Interface (readline.js:203:10)
    at Object.createInterface (readline.js:64:10)
    at new UI (/Users/rfeng/Projects/loopback4/loopback-next/packages/cli/node_modules/yeoman-environment/node_modules/inquirer/lib/ui/baseUI.js:15:26)
    at new PromptUI (/Users/rfeng/Projects/loopback4/loopback-next/packages/cli/node_modules/yeoman-environment/node_modules/inquirer/lib/ui/prompt.js:14:5)
    at TerminalAdapter.promptModule (/Users/rfeng/Projects/loopback4/loopback-next/packages/cli/node_modules/yeoman-environment/node_modules/inquirer/lib/inquirer.js:27:14)
    at TerminalAdapter.prompt (/Users/rfeng/Projects/loopback4/loopback-next/packages/cli/node_modules/yeoman-environment/lib/adapter.js:46:26)
    at StatusConflicter._ask (/Users/rfeng/Projects/loopback4/loopback-next/packages/cli/node_modules/yeoman-generator/lib/util/conflicter.js:172:18)
    at StatusConflicter.collision (/Users/rfeng/Projects/loopback4/loopback-next/packages/cli/node_modules/yeoman-generator/lib/util/conflicter.js:125:12)

I'm not sure how to fix it at the moment.

BTW, the stream based approach needs to deal with mock-stdin which seems to be a legacy stream implementation which only emits data events instead of readable.

raymondfeng · 2018-06-29T17:33:48Z

I was able to get around the issue using the following code after reading readline source.

/**
 * Read string from stdin
 */
exports.readStringFromStdin = function(log) {
  const stdin = process.stdin;

  if (stdin.isTTY && log) {
    log(
      chalk.green('Please type in a json object ' + '(Press <ctrl>-D to end):'),
    );
  }

  const data = [];
  let len = 0;

  function onData(chunk) {
    data.push(chunk);
    len += chunk.length;
  }

  // Restore stdin
  function close() {
    stdin.removeListener('data', onData);
    stdin.pause(); // Pause the stdin so that other prompts can happen
  }

  return new Promise((resolve, reject) => {
    // Set up `data` handler to make it compatible with mock-stdin which is
    // an old style stream which does not emit `readable` events
    stdin.on('data', onData);

    stdin.once('end', () => {
      const buf = Buffer.concat(data, len);
      close();
      resolve(buf.toString('utf-8'));
    });

    stdin.once('error', err => {
      close();
      reject(err);
    });
  });
};

@bajtos Do you really think this is better/simpler than the current code using readline? Please note inquirer internally uses readline too. IMO, it's probably more robust to keep the current code which leverages a core module from Node.js itself.

bajtos · 2018-07-02T13:53:48Z

BTW, the stream based approach needs to deal with mock-stdin which seems to be a legacy stream implementation which only emits data events instead of readable.

Bummer ☹️

Do you really think this is better/simpler than the current code using readline? Please note inquirer internally uses readline too. IMO, it's probably more robust to keep the current code which leverages a core module from Node.js itself.

Yeah, I agree with you that it's more robust to keep the readline-based implementation then. Thank you for looking under the hood and helping us both to better understand design constraints.

To me, this is another argument for not using yeoman (see #844).

bajtos

Looks mostly good.

Please address my comments below. Feel free to get somebody else (@virkt25?) to give a final approval for this pull request, so that you don't have to wait for me in case I am not around.

bajtos · 2018-07-02T13:57:11Z

packages/cli/lib/base-generator.js

+          if (err) return;
+          const jsonStr = lines.join('\n');
+          try {
+            const json = JSON.parse(jsonStr);


Please move handling of the input outside of the code dealing with stdin.

Ideally, I'd like to see something like the following in _readJSONFromStdin:

async _readJSONFromStdin() { const input = await readFromStdin(); try { return JSON.parse(input); } catch (err) { if (!process.stdin.isTTY) { debug(err, jsonStr); } throw err; } }

Fixed. Refactor the function into utils.

bajtos · 2018-07-02T13:58:28Z

packages/cli/lib/base-generator.js

+        this.log(
+          chalk.red('The stdin is not a terminal. No prompt is allowed.'),
+        );
+        process.exit(1);


Don't exit the process please, it disrupts mocha tests. Call this.exit or throw new Error instead.

bajtos · 2018-07-02T13:58:57Z

packages/cli/lib/base-generator.js

+          this.log(
+            chalk.red('The stdin is not a terminal. No prompt is allowed.'),
+          );
+          process.exit(1);


Ditto. Don't exit the process, end the generator execution only.

raymondfeng · 2018-07-02T16:20:03Z

@bajtos @virkt25 PTAL

bajtos · 2018-07-03T08:04:12Z

packages/cli/lib/utils.js

+  if (process.stdin.isTTY && log) {
+    log(
+      chalk.green(
+        'Please type in a json object line by line ' +


In my view, this message does not belong to readTextFromStdin as readTextFromStdin can be used to read arbitrary text, not only JSON.

I am proposing to move it back to _readJSONFromStdin and also remove the no-longer used log argument.

virkt25

Apart from the one comment left by @bajtos, the changes look good to me. Excited to see this PR land. :D

- Allow generator options to be passed in json format from a file, stdin, or stringified value - Allow prompts to be skipped if a prompt has default value or the corresponding anwser exists in options

raymondfeng requested a review from bajtos as a code owner May 30, 2018 22:15

raymondfeng changed the title ~~Enable json/json-file options and express mode~~ [WIP] Enable json/json-file options and express mode May 30, 2018

bajtos reviewed May 31, 2018

View reviewed changes

raymondfeng force-pushed the enable-cli-json branch 2 times, most recently from 9a48834 to d08ae6f Compare May 31, 2018 21:35

raymondfeng requested a review from shimks as a code owner May 31, 2018 21:35

raymondfeng force-pushed the enable-cli-json branch 4 times, most recently from 274cf05 to c96b219 Compare June 4, 2018 16:34

shimks mentioned this pull request Jun 5, 2018

Creating Express mode for generating applications(must have) and other artifacts(optional) through CLI #1340

Closed

6 tasks

raymondfeng force-pushed the enable-cli-json branch 4 times, most recently from db3600e to f88f1d9 Compare June 8, 2018 20:14

raymondfeng force-pushed the enable-cli-json branch from f88f1d9 to ce31b72 Compare June 13, 2018 17:22

bajtos suggested changes Jun 14, 2018

View reviewed changes

bajtos changed the title ~~[WIP] Enable json/json-file options and express mode~~ Enable json/json-file options and non-interactive mode Jun 14, 2018

virkt25 reviewed Jun 16, 2018

View reviewed changes

raymondfeng force-pushed the enable-cli-json branch 5 times, most recently from 9b49a49 to 041b5d2 Compare June 25, 2018 17:18

bajtos reviewed Jun 26, 2018

View reviewed changes

raymondfeng force-pushed the enable-cli-json branch from 041b5d2 to af291b3 Compare June 26, 2018 17:00

raymondfeng changed the title ~~Enable json/json-file options and non-interactive mode~~ Enable json config options and non-interactive mode Jun 26, 2018

raymondfeng force-pushed the enable-cli-json branch 2 times, most recently from 4718144 to aaab42d Compare June 26, 2018 17:21

bajtos mentioned this pull request Jul 2, 2018

[CLI] Get rid of Yeoman #844

Open

bajtos approved these changes Jul 2, 2018

View reviewed changes

raymondfeng force-pushed the enable-cli-json branch from aaab42d to e73a949 Compare July 2, 2018 16:19

bajtos approved these changes Jul 3, 2018

View reviewed changes

virkt25 approved these changes Jul 4, 2018

View reviewed changes

feat(cli): add config and yes options

662696c

- Allow generator options to be passed in json format from a file, stdin, or stringified value - Allow prompts to be skipped if a prompt has default value or the corresponding anwser exists in options

raymondfeng force-pushed the enable-cli-json branch from e73a949 to 662696c Compare July 7, 2018 16:24

raymondfeng merged commit 5778a2a into master Jul 8, 2018

bajtos deleted the enable-cli-json branch October 4, 2018 06:49

Enable json config options and non-interactive mode #1384

Enable json config options and non-interactive mode #1384

Conversation

raymondfeng commented May 30, 2018 • edited

Checklist

Choose a reason for hiding this comment

Choose a reason for hiding this comment

bajtos left a comment • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

raymondfeng Jun 26, 2018 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

raymondfeng Jun 26, 2018 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

virkt25 left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

bajtos left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

raymondfeng commented Jun 26, 2018

raymondfeng commented Jun 29, 2018

raymondfeng commented Jun 29, 2018 • edited

bajtos commented Jul 2, 2018

bajtos left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

raymondfeng commented Jul 2, 2018

Choose a reason for hiding this comment

virkt25 left a comment

Choose a reason for hiding this comment

raymondfeng commented May 30, 2018 •

edited

bajtos left a comment •

edited

raymondfeng Jun 26, 2018 •

edited

raymondfeng Jun 26, 2018 •

edited

raymondfeng commented Jun 29, 2018 •

edited