Support project references #817

andrewbranch · 2018-08-10T15:09:42Z

Closes #815

TODO:

support project references for transpileOnly
support project references for experimentalWatchApi
support project references for running with neither of the above
resolve loader output to built JS files for TS files that were part of a project reference
tests (~~for each of the three modes above,~~ one test for a successful build and one that fails because the dependent project isn’t built)
get Webpack to watch the built JS files instead of TS files in composite projects
manually test on a real-life big project
write execution tests

johnnyreilly · 2018-08-10T17:03:28Z

This is PR "1" of the potential 2 I guess? Great work so far! 👍

You've just reminded me that I need to update the comparison test pack for TS 3.0. I'll try and do that this weekend so you don't have to.

I'll message here when it's done.

johnnyreilly · 2018-08-11T13:59:58Z

I've upgraded the comparison test pack to 3.0 - feel free to merge that on to your pr

andrewbranch · 2018-08-13T18:23:00Z

Rebased against master with the updated tests 👍

Latest commit will fail; I'm going ahead and assuming we'll have typescript.getOutputJavaScriptFile via microsoft/TypeScript#26410.

andrewbranch · 2018-08-13T18:39:05Z

src/index.ts


+    if (!instance.compiler.sys.fileExists(mapFileName)) {
+      loader.emitWarning(


It might be obnoxious to do this for every file. Maybe I can memoize by referencedProject.sourceFile.fileName? Or would it make more sense to validate each project reference while first setting up the program/languageService/watch?

That seems reasonable; presumably that's not something that changes often at all so should be a completely reasonable choice

src/utils.ts

andrewbranch · 2018-08-14T14:58:14Z

@johnnyreilly can the comparison test’s "patch" feature be used to patch the test's webpack.config.js itself?

I'm thinking about the scenarios that need to be tested, and they include all three modes of creating a TS program—transpileOnly, experimentalWatchApi, and the normal language service route. Would you create a different comparison test for each even though it shouldn't theoretically change the output at all? Or is that something that's more appropriate for an execution test?

It's also definitely going to be worth testing the error cases, as an unbuilt project reference would compile perfectly fine if project references weren’t working, if that makes sense (TypeScript would just compile the source files per usual, instead of saying “hey this is part of a project reference that’s not built”). Are comparison tests useful for asserting failed builds at all?

Edit: I see the Webpack output is captured in a text file, so that looks pretty good for testing the error cases 👍

andrewbranch

Did a first pass at a comparison test. Not sure why there wasn't a .transpiled version generated like some of the others have though?

To generate outputs for older TS versions, do I need to manually install each older version and run again?

andrewbranch · 2018-08-14T17:04:41Z

test/comparison-tests/projectReferencesNoSourceMap/lib/tsconfig.json

+    "composite": true
+  },
+  "files": [
+    "./index.ts"


For some reason, without files or include, program.getProjectReferences()[0].commandLine.fileNames was empty. I thought it should have resolved to all *.ts files by default?

To generate outputs for older TS versions, do I need to manually install each older version and run again?

No - we only run comparison tests for the latest TypeScript version. So 3 alone is fine

For some reason, without files or include, program.getProjectReferences()[0].commandLine.fileNames was empty. I thought it should have resolved to all *.ts files by default?

Without project references in the mix then that should be the case. Is it different when rolling with project references perhaps?

Regards the lack of transpile; that's surprising but I wouldn't worry right now. This is the code that generates it BTW: https://github.com/TypeStrong/ts-loader/blob/master/test/comparison-tests/create-and-execute-test.js

When I was testing ad-hoc, the reference.commandLine.fileNames was working correctly, but I think I was using typescript@next... I'll have to investigate more.

Ah, I was mistaken:

All implementation files must be matched by an include pattern or listed in the files array. If this constraint is violated, tsc will inform you which files weren’t specified

https://www.typescriptlang.org/docs/handbook/project-references.html#composite

It seems it's working as intended, but via our programmatic route we don't get that warning. Maybe need to add our own.

can the comparison test’s "patch" feature be used to patch the test's webpack.config.js itself?

No I'm afraid not; the tests rely upon an unchanged webpack config between patches. Sorry

It seems it's working as intended, but via our programmatic route we don't get that warning. Maybe need to add our own.

I'm not sure if this can be done reliably and efficiently... without actually doing a build of the dependent projects, it's hard to tell if a given file was intended to be included by one of them, or was just intended to be processed by the parent project (i.e. in our case, run through the loader). Implementing a warning from ts-loader side while consuming project references might be out of scope—hopefully it's something users will notice while actually running tsc on those projects, since pre-building the referenced projects is currently required, and tsc will emit the error if files or include don't cover all the files.

… just skip them

andrewbranch · 2018-08-20T21:57:51Z

@johnnyreilly status update:

Implementation

Waiting on Export getOutputJavaScriptFileName from tsbuild microsoft/TypeScript#26421 (no movement there since opening 😕)
Anything else? Feel free to do a closer review now; I think it's pretty much done!

Tests

Is it a problem that the .transpiled one didn't show up for the projectReferences and projectReferencesNoSourceMap comparison tests?
Should I copy any of the three comparison tests I added and run with experimentalWatchApi?
Any other comments/feedback on tests?

johnnyreilly · 2018-08-28T06:52:58Z

Sorry for the delay in responding @andrewbranch - I'm away and just have my phone and poor connectivity. In fact I'm typing this surrounded by 🐑 whilst lost rambling!

Is it a problem that the .transpiled one didn't show up for the projectReferences and projectReferencesNoSourceMap comparison tests?

I'm not sure it is; my guess is that people will only use projectReferences with non-transpile mode. Either way this is fine for now in my view.

Should I copy any of the three comparison tests I added and run with experimentalWatchApi?

Thanks but no. What I actually mean to do at some point is tweak the comparison test pack so it runs in 3 modes: standard, transpileOnly and watchApi. So in the longer term separate tests for the watchApi shouldn't exist. It's just another part of the test matrix if you will.

Any other comments/feedback on tests?

They're not passing 😂

Jokes aside, I wouldn't worry about the tests for now. First get to an implementation that you're content with, let's review that together and then we can work together on the test that cover this new functionality until they are fully passing / covering the scenarios we care about.

Thanks again for all the work you're doing; I really appreciate it. Much ♥️🌻

I should be back online in a week or so. Assuming I find the way out of this field 😉

andrewbranch · 2018-08-28T16:38:17Z

Thanks! Hope you are enjoying your vacation.

The build is failing because I wrote a line that assumes microsoft/TypeScript#26421 will be merged. I think maybe we just need to press on without that for now. I'll give it a few more days but have something mergeable by the time you're back.

ajafff · 2018-08-29T08:34:09Z

src/utils.ts

+      program &&
+      program
+        .getProjectReferences()!
+        .find(ref => ref!.commandLine.fileNames.indexOf(filePath) > -1)


do you handle missing referenced projects somewhere else or how can you be sure ref is always defined?
ref will be undefined if a referenced project cannot be resolved, i.e. missing folder or missing tsconfig.json.

It will be a holey array? I didn't realize that; it feels a bit unintuitive, but I guess the types agree. Let me do something here. Thanks for the feedback!

ajafff · 2018-08-29T08:38:50Z

src/index.ts

-  const { outputText, sourceMapText } = options.transpileOnly
-    ? getTranspilationEmit(filePath, contents, instance, loader)
-    : getEmit(rawFilePath, filePath, instance, loader);
+    const jsFileName = instance.compiler.getOutputJavaScriptFileName(


given that the whole tsbuild API is now internal, this uses an internal function? I couldn't find a custom implementation if it in this PR

ajafff · 2018-09-20T05:23:36Z

@andrewbranch that change was published in typescript@next a few days ago

johnnyreilly · 2018-09-20T05:25:44Z

Thanks @ajafff! All yours @andrewbranch!

andrewbranch · 2018-09-20T05:47:48Z

@andrewbranch that change was published in typescript@next a few days ago

This doesn't seem to be the case insofar as what npm i typescript@next does. There’s no instance of getResolvedProjectReference in node_modules/typescript/lib/typescript.js. I also tried installing typescript@master and building from source, but master fails to build for me.

At any rate, ddc5cc7 runs getResolvedProjectReferences instead of getProjectReferences if it exists, so I think that should be ok.

Am I free to hack on your branch?

By all means!

I'll work on ignoring the comparison test on Windows. I'll also write some execution tests so we have some in place.

🙏

ajafff · 2018-09-20T06:19:06Z

I'm already using program.getResolvedProjectReferences() and it compiles just fine with typescript@3.1.0-dev.20180915

https://github.com/fimbullinter/wotan/blob/0518a9a78ed0f38c490cde1533b00d7681821c99/packages/wotan/src/runner.ts#L327-L330

andrewbranch · 2018-09-21T03:32:02Z

Ok, cool, thanks @ajafff. Confirmed that typescript@3.1.0-dev.20180915 works with the new API; typescript@3.1.0-dev.20180920 I think was just itself broken ¯\(ツ)/¯

Working on the README now.

andrewbranch · 2018-09-21T04:40:56Z

@johnnyreilly as I'm writing this README, I'm thinking that it honestly sounds kind of silly at this point without yet having support for building upstream projects. I think probably very few people will want to try out the feature before there's a solid solution for having everything built for you.

A somewhat interesting effect of ts-loader not yet supporting project references in master is that people who have project references in their tsconfig and are using ts-loader on those projects today are not actually using project references, but everything should be working perfectly for them, because each file request is just getting resolved to the source file, just as if they didn't have references in their tsconfig at all.

If we ship this now, enabled by default, without support for rebuilding upstream projects, we'll technically break those people, because all of a sudden, project references will start half-working instead of being ignored. Folks may not realize that ts-loader stopped building their files.

I'm thinking we need to put all this work behind a loader option. What do you think?

johnnyreilly · 2018-09-21T04:44:34Z

Oooh - good catch! Yes that makes sense. If you could include that information in the README that would be good. What do you think would be a good name for the ~~feature flag~~ loader option? projectReferences?

I'll probably write a blog post when this ships to publicise that this feature is available and that it's opt-in. I'll plan to use some of your README tweaks for part of that (giving credit where it's due of course ❤️ 😄 )

Working on ignoring the test on Windows now.

andrewbranch · 2018-09-21T05:10:31Z

Awesome. After so much work, it's slightly disappointing that it’s not super usable yet, but I really think the hardest part is done. (🤞🙈) I may not be able to pick back up right away, but if nobody else jumps on it, I’ll want to see it through and figure out building upstream projects. That said, I’d be thrilled if someone else does want to champion that part, but my own sense of sunk cost won’t let me leave it half-done. 😄

johnnyreilly · 2018-09-21T05:25:35Z

Awesome. After so much work, it's slightly disappointing that it’s not super usable yet, but I really think the hardest part is done. (🤞🙈)

Completely agree.

I may not be able to pick back up right away, but if nobody else jumps on it, I’ll want to see it through and figure out building upstream projects. That said, I’d be thrilled if someone else does want to champion that part, but my own sense of sunk cost won’t let me leave it half-done. 😄

😄

johnnyreilly · 2018-09-21T06:55:53Z

Working on ignoring the test on Windows now.

I'd forgotten just how fiddly / flaky the comparison test pack can be.

Decided to use the opportunity to introduce a whole suite of fancy logic around ignoring tests. Managed to break the existing test pack in the process 🤦

Totally unnecessary.

Will replace https://github.com/TypeStrong/ts-loader/blob/master/test/comparison-tests/create-and-execute-test.js#L55 with something as simple as:

if (testName !== 'projectReferencesOutDir' || require('os').platform() !== 'win32') {
        // @ts-ignore
        it('should have the correct output', createTest(testToRun, testPath, {}));
}

Not tested this yet; wrote it on a Google Pixel on a train whilst reading the node docs on a spotty connection. 😄

Heading into work now; will try and take another look tonight / this weekend

johnnyreilly · 2018-09-21T10:48:24Z

Cool - that worked. I'll see if I can make a start on execution tests this weekend.

johnnyreilly · 2018-09-23T14:26:50Z

Hey @andrewbranch ,

It's time to 🚢! I'm planning to push out this as version 5.2.0. I'll put a note in the README about the outDir Windows issue and probably put together an example repo too.

I'm going to write a blogpost to let people know about this new feature. I'm planning to plagiarise your excellent update to the README pretty heavily. I hope that's okay!

Thanks again for all your hard work. It's greatly appreciated! Much ❤️

andrewbranch · 2018-09-23T18:03:28Z

Woop! 🙌🏼 Thanks for all the support throughout the process. This has been fun! Plagiarize away! 😄

johnnyreilly · 2018-09-23T19:20:42Z

Consider it done 😄 https://blog.johnnyreilly.com/2018/09/ts-loader-project-references-first-blood.html

andrewbranch commented Aug 13, 2018

View reviewed changes

andrewbranch commented Aug 14, 2018

View reviewed changes

This was referenced Aug 15, 2018

Added support for TypeScript declaration map #821

Merged

Fixed issue with incorrect output path for declaration files #822

Merged

andrewbranch added 14 commits August 20, 2018 12:53

Pass project references to language service

92f23b3

Use parsed project references, not raw content

55d8efa

Don’t fail while trying to emit/diagnose files in project references;…

75a5bdb

… just skip them

Be tolerant of older TypeScript versions

05ee14d

Add project references to transpileOnly mode

157211e

Add project references to experimental watch API path

ee61c63

Use JS output for files in project references

0b81aa3

Fix loader emit typings

0a42045

Get program out of language service too

c01af17

Add comparison test for project references without source maps

0d2854c

Add comparison test for project references with source map (no warnings)

615bcbb

Add test for unbuilt project references

2f4ff59

Only warn once per project per compilation about source maps

1e765b2

Make comments more accurate

d262970

johnnyreilly mentioned this pull request Aug 28, 2018

Original TS internal error is hidden #824

Closed

andrewbranch mentioned this pull request Aug 28, 2018

Support for TS 3 project references #815

Closed

ajafff reviewed Aug 29, 2018

View reviewed changes

andrewbranch added 2 commits September 1, 2018 09:43

Implement getOutputJavaScriptFileName

700e967

Add test for project references with outDir (and fix implementation)

741bf9b

Update README with project references

6c87188

andrewbranch added 2 commits September 20, 2018 21:58

Put projectReferences behind a loaderOption

611e8d9

Update validateLoaderOptions and test loader options

32e2882

Update validateLoaderOptionNames test

e47664b

ignore projectReferencesOutDir on windows in non-transpile mode

2f196b3

johnnyreilly added 8 commits September 22, 2018 05:52

add execution test for project references

b235502

General tweaks; I should really get tslint properly set up

5d190fa

some yarn lint stuff

16a9e7c

further tslint fixes

b6510a0

fix broken appenSuffixes

fbed24b

add linting to build

308f0dd

update changelog

a6b203c

Merge branch 'master' into project-references

ca2c576

johnnyreilly merged commit 635f745 into TypeStrong:master Sep 23, 2018

andrewbranch mentioned this pull request Nov 26, 2018

Fix projectReferences with rootDir #871

Merged

ahnpnl mentioned this pull request Apr 15, 2020

fix(compiler): make projectReferences work with isolatedModules: true kulshekhar/ts-jest#1527

Merged

2 tasks


		if (!instance.compiler.sys.fileExists(mapFileName)) {
		loader.emitWarning(

Support project references #817

Support project references #817

Conversation

andrewbranch commented Aug 10, 2018 • edited by johnnyreilly

johnnyreilly commented Aug 10, 2018 • edited

johnnyreilly commented Aug 11, 2018

andrewbranch commented Aug 13, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

andrewbranch commented Aug 14, 2018 • edited

andrewbranch left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

andrewbranch commented Aug 20, 2018

Implementation

Tests

johnnyreilly commented Aug 28, 2018 • edited

andrewbranch commented Aug 28, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ajafff commented Sep 20, 2018

johnnyreilly commented Sep 20, 2018

andrewbranch commented Sep 20, 2018

ajafff commented Sep 20, 2018

andrewbranch commented Sep 21, 2018 • edited

andrewbranch commented Sep 21, 2018

johnnyreilly commented Sep 21, 2018 • edited

andrewbranch commented Sep 21, 2018

johnnyreilly commented Sep 21, 2018

johnnyreilly commented Sep 21, 2018 • edited

johnnyreilly commented Sep 21, 2018

johnnyreilly commented Sep 23, 2018

andrewbranch commented Sep 23, 2018

johnnyreilly commented Sep 23, 2018

andrewbranch commented Aug 10, 2018 •

edited by johnnyreilly

johnnyreilly commented Aug 10, 2018 •

edited

andrewbranch commented Aug 14, 2018 •

edited

johnnyreilly commented Aug 28, 2018 •

edited

andrewbranch commented Sep 21, 2018 •

edited

johnnyreilly commented Sep 21, 2018 •

edited

johnnyreilly commented Sep 21, 2018 •

edited