tsc --build / Project References Feedback & Discussion

[RyanCavanaugh]

Continuation of #3469 "Medium-sized Projects" which has grown too large for GitHub to display comfortably

Please use this thread for feedback, general discussion, questions, or requests for help / "is this a bug" discussions.

Active bugs / PRs in this area:

• Project References TypeScript-Handbook#788 Handbook page

Possible next steps:

• (none yet; @andy-ms) Find unloaded downstream projects during rename
• Infer project references from common monorepo patterns / tools #25376 Infer project references from monorepo tooling package.json files
• Terse mode output from tsc --build #25562 Terse mode output for tsc -b
• (none yet; @weswigham ?) Intermediate build outputs to temp folders
• (none; stretch) Parallelization / worker processes under tsc -b

Other interesting links:

• https://github.com/RyanCavanaugh/learn-a - sample project references repo using lerna
• https://github.com/RyanCavanaugh/project-references-outfile - sample project references repo using --outFile
• Support "medium-sized" projects #3469 (comment) Ryan's outline of what we're doing/did and why
• Support "medium-sized" projects #3469 (comment) pre-draft of handbook page
• https://github.com/RyanCavanaugh/learn-a/pull/3/files learn-a repo using yarn workspaces!

[timfish]

A couple of things:

1. watch doesn't currently output an initial build. It waits until the first file change. This differs to non build mode.
2. How is build order determined? I found I had to order the project references in dependency order to get the build to complete first time without error

[RyanCavanaugh]

@timfish thanks! Re 1 - PR up at #25610. For the other issue, can you sketch out the files you have? The order is supposed to be a simple topological sort that should always yield a correct ordering. I'll try to repro locally in the meantime based on what you've described

Edit: Not sure how to repro - I changed the TypeScript repo's src/tsconfig.json to be in effectively random order and it still figured out a valid build order. Will need details on this.

[EisenbergEffect]

@RyanCavanaugh Were you all able to land the cross-project rename for the RC?

[RyanCavanaugh]

@EisenbergEffect we have "upstream" renames (renaming a usage in client affects declarations in shared) but not "downstream" (renames in shared affecting uses in server and client) yet

[EisenbergEffect]

Gotcha. Any ideas is you'll be able to get the downstream rename in for the final release? or have you already determined that it needs to push out to 3.1 or beyond?

[RyanCavanaugh]

It looks like we can probably get downstream renames for loaded projects in by 3.0 final. Detecting which other projects on your computer need to be loaded to do an "exhaustive downstream rename" is looking dicey - we haven't found any mechanisms that would let us short-circuit that work when the symbol being renamed is exported from the current file.

[newtack]

I am using the API to build Typescript projects, including createWatchProgram and createWatchCompilerHost.

1. Are these updated to use the new project references and 2) do I need to do anything else in my code other than update the tsconfig.json?

[RyanCavanaugh]

If you're hosting the compiler API, you don't need to do anything new for project references; everything should work as-is. If you want to take advantage of the new --build mode features, the entry point is createSolutionBuilder and you'll need to provide a BuildHost and CompilerHost along with the file timestamp APIs on the latter.

To migrate to using project references in your code itself, updating your tsconfig.json to a) add references and b) add composite: true may be sufficient - if not, you'll see errors informing you what else needs to happen. Writing a comprehensive migration guide has been a difficult task because project setups are so varied - I'm hoping we can find migrations from early adopters as something to point people to, but it's hard to give guidance without seeing specific build layouts.

[newtack]

Thanks @RyanCavanaugh

Are there any examples for "createSolutionBuilder" and "timestamp API"?

[timfish]

@RyanCavanaugh ref: Build order.

I'll try to get a repo reproducing this although it won't be until next week.

If I'm just adding all references to a root tsconfig, it should work out the dependency tree from that?

[RyanCavanaugh]

@newtack the compiler source code itself (src/compiler/sys.ts for timestamp APIs and src/compiler/tsc.ts & tsbuild.ts) are good references.

@timfish

If I'm just adding all references to a root tsconfig, it should work out the dependency tree from that?

Correct

[OliverJAsh]

Trying this out with the 3.0 RC.

declarationMaps
We've also added support for declaration source maps.
If you enable --declarationMap, you'll be able to use editor features like "Go to Definition" and Rename to transparently navigate and edit code across project boundaries in supported editors.

I tried using this but for some reason "Go to Definition" didn't seem to work. I'm probably doing something wrong, but filed an issue with a minimal reproduction case here: #25662

Will this also handle "Find All References"? If not, is there any way that could be supported?

[kohlmannj]

Based on @RyanCavanaugh and @rosskevin's work to demonstrate Project References within a Lerna-based monorepo¹²³, I've made another not-for-merging Pull Request which uses both Yarn workspaces and Babel 7: RyanCavanaugh/learn-a#4

Hopefully this example helps others learn! I've been working on a Lerna-based monorepo similar to this, and now (I think) I have a better idea of how to configure Project References with it.

—

1. https://github.com/RyanCavanaugh/learn-a
2. (NOT FOR MERGING)convert to use yarn workspaces RyanCavanaugh/learn-a#3
3. Support "medium-sized" projects #3469 (comment)

[drew-y]

I was hoping to get some advice on how to handle path resolutions. My directory currently looks something like this:

- client/
  - assets/
  - views/
  - src/
  - tsconfig.json
  - webpack.config.js
- shared/
  - src/
  - tsconfig.json
- server/
  - src/
  - tsconfig.json

Im struggling to reference the shared directory. I was surprised to learn that the "paths" section along with baseURL doesn't allow for path aliasing like webpack does. So now I'm not sure how to reference the shared directory at all.

For example, the following doesn't work:

// server/src/index.ts
import { mySharedFunc } from "../../shared/src"

// This doesn't work either. "shared" is still shared after compilation
import { mySharedFunc } from "shared"

Because the project compiles down to:

- dist/
  - client/
    - bundle.js
  - shared/
    - index.js
  - server/
    - index.js

As you can see, it is invalid to reference the src dir as it doesn't exist after compilation.

Should I restructure my repo somehow? Is it possible to get my path aliasing to work?

My struggles appear to be related to #25682

[drew-y]

I've updated my folder structure to have client, server, and shared under one src directory. This resolved my issue.

[zpdDG4gta8XKpMCd]

@RyanCavanaugh any luck with my code? #3469 (comment)

[ryanstaniforth]

I assumed that with "composite": true, "declaration": true would always be forced. However this only seems to apply with tsc --build.

I found this out while doing a normal build (tsc) and not seeing expected definition files.

Is this expected behaviour? It seems counter-intuitive to me.

[sledorze]

I found myself using the same technique as the references system in our temporary custom incremental build system (comparing d.ts files) to decide what to rebuild in a non incremental scenario.

However, after finding strange results, I've discovered that the typescript compiler (3.0-RC) do NOT generate deterministically d.ts files.
Indeed, when Unions are involded, the displayed order of those seems to depend on the nodeJS run, which makes to system detect false positives.

ex: { status: "400"; body: KoStatus; } | { status: "200"; body: OkStatus; }
vs:
{ status: "200"; body: OkStatus; } | { status: "400"; body: KoStatus; }

I think you may have found yourselves those issues and wonder if that's a know/accepted limitation between runs or if I should open an issue (have not found an existing one yet).

[rosslovas]

Trying it out on the codebase I'm working on, I ran into a lot of Output file '.../_.d.ts' has not been built from source file '.../_.ts' errors. I managed to strip down a repro to almost nothing, and found that when rootDir and outDir are involved as well as a project reference, source files within an individual project that has a dependency project would be compiled in alphabetical order.

The consequence is that an A.ts that imports from B.ts will fail with Output file '.../B.d.ts' has not been built from source file '.../B.ts' but if I rename A.ts to C.ts everything works, as does removing the project reference or not using root/outDir. The problem occurs in 3.0.0-rc as well as next right now.

Edit: In retrospect this is clearly enough a bug to warrant its own issue, so I made #25864.

[josundt]

I just tested TS 3 RC project references in a typical npm package setup with included tests. The tests were set up as a TS project referencing the implementation TS project.

I surprisingly found that project references do not adhere to the outDir compilerOption of referenced or referencing projects, and that the relative paths for module imports in the compiled referencing project's JS files therefore are wrong; paths point to the TS folder, not the compiled output folder.

A somewhat simplified view of my PoC:

• src/
  (contains implementation code, TS project REFERENCED by tests)
• dist/
  (compiled output from src/; using the outDir compilerOption)
• test/
  (tests for implementation code in src/; project REFERENCING src/)

Expected behavior:
Relative paths of module imports in the compiled code for the referencing project should be calculated from:

• relative path from outDir compilerOption of referenced project
• relative path from outDir compilerOption of referencing project

I put up a small repository to demonstrate the problem:
https://github.com/josundt/tsprojects-outdirproblem

UPDATE: Separate issue created: #26036

[gentoo90]

Is <TypeScriptBuildMode> msbuild parameter pllaned for 3.0.1? It's not working in 3.0.0, tsc.exe still runs with --project, even if it's set to true.

[massimocode]

@RyanCavanaugh This project thing looks really cool so I decided to give it a try. It hasn't been going so well for me so I thought I'd share my experiences and the code-base in case it's helpful for you guys.

Basically, I have the following folders in my project.

• back-end - A Node.JS back-end. References code from shared.
• front-end - An Angular 2 front-end. References code from shared.
• shared - Vanilla JS code. Doesn't reference any of the other folders.
• specs - Some tests written in typescript. References back-end, front-end and shared.

My previous set-up (TypeScript v2.x) was to have a tsconfig.json in the root with the most "forgiving" settings and that would give my IDE all the intellisense etc and cross project renames. I then had a compile.json in back-end (which was a tsconfig file in disguise), and another compile.json in the front-end (same concept). The reason for these files is that I wanted to only compile front-end code (for webpack) or back-end code (for ts-node) separately from the rest. Needless to say, they wouldn't have any issues picking up the code that I referenced in the shared folder as the paths were all relative etc. This all worked perfectly fine for my work flow and I've been super happy with it (except about a year and a half ago the Angular CLI didn't like it at all, but I'm doing webpack manually and haven't retried since...)

So, I upgraded TypeScript to the @next version and switched over to this whole tsconfig per folder thing and removed the tsconfig.json that was at the root. I'm using the latest version of the VSCode Insiders build, but it doesn't really seem to work at all. In the sense that when I'm looking at some back-end code that references something in the shared folder, it says that thing simply doesn't exist.

The ts-node for my back-end only works if I compile the shared folder and emit to disk, something that I'd rather not have to do because my back-end compiles down to ES6 and my front-end down to ES5. Although I guess it wouldn't be so bad if it emitted declarations only?

Anyway, I've checked it all in, would you mind taking a look and seeing what you think? I wouldn't mind if you wanted to use the repo (or part of it) as some kind of example project. Thanks!

Here's the link: https://github.com/SMH110/Pizza-website/tree/TS_Projects

The previous working setup not using the new projects functionality is at this commit: SMH110/Pizza-website@9548bdd

[Bessonov]

@aminpaks
Do you really mean VSCode? I found VSCode to be very unreliable for error checking because it only checks opened files. However, I believe this is due to optimizations, and I haven't looked into it for a long time.

I haven't used project references and incremental build for years, and my experience may no longer be relevant. But we encountered all types of bugs. Ultimately, we switched to webpack, ts-loader, and fork-ts-checker-webpack-plugin, and it was the only setup that worked more or less reliably. The chain is still extreme slow. Now, I am waiting for the stabilization of Bun workspaces to test it with plain tsc --build --incremental --watch + project references for error checking.

By the way, you may be interested in #33685. Found it by accident 😄

[aminpaks]

@aminpaks Do you really mean VSCode? I found VSCode to be very unreliable for error checking because it only checks opened files. However, I believe this is due to optimizations, and I haven't looked into it for a long time.

I haven't used project references and incremental build for years, and my experience may no longer be relevant. But we encountered all types of bugs. Ultimately, we switched to webpack, ts-loader, and fork-ts-checker-webpack-plugin, and it was the only setup that worked more or less reliably. The chain is still extreme slow. Now, I am waiting for the stabilization of Bun workspaces to test it with plain tsc --build --incremental --watch + project references for error checking.

We're tying to improve the DX, and are focused on making the types in our projects faster. Overall I find the incremental build not working well in our project. We have issue in our tsconfigs that I'm addressing, and then removing the circular dependencies but I'm not sure that'll solve the issue we're seeing with incremental build in our projects.

We still see the "Configure excludes" in vscode, and ts server crashes. I was expecting when I open a file in vscode but I'm not touching it it wouldn't rebuild it, but it seems that's not the case. ts server loads all the files, and eventually crashes due to too many files being loaded.

By the way, you may be interested in #33685. Found it by accident 😄

Yeah that's great if happens, found that the other day.

[aminpaks]

Also something that we discovered is that we can't exclude test files from referenced projects but they contribute a large amount of time to compilation time.
For example Project A that depends on Project B, it's true that Project A can work without ProjectB's test files, but we couldn't find anyway to exclude them from the projects.

[unional]

Also something that we discovered is that we can't exclude test files from referenced projects but they contribute a large amount of time to compilation time

That you can do, by referencing the tsconfig directly:

// project-a/tsconfig.build.json
{
  "references": [{
    "path": "../project-b/tsconfig.build.json"
  }]
}

Also I think ts-loader does not support it: TypeStrong/ts-loader#1632

[aminpaks]

Does anyone know how to extend jest in referenced projects. We have some custom extensions but I can't add them in tsconfig.include as it's gonna be added in all referenced projects, and it breaks the incremental build. I don't wanna add them to each test either, is there a way to make it available in globals, and just add the extensions in the jest setup?

Update: We just needed to provide the custom matchers in d.ts.

[aminpaks]

@RyanCavanaugh

Here are our thoughts when we used the project references at Shopify:
Configuring composite projects is not hard but it adds an overhead, and it becomes difficult when we try to split projects without moving files. On the other hand the referenced projects still break for the entry points that imports all files. For some reasons TS compiler imports files as soon as it finds an import statement even though they are excluded in the tsconfig.exclude, I think this should be a compilation error instead of magically making it work. For example we have a project entry A that acts as the foundation, and imports all the other sections of the app, let's say 28 composite projects, as soon as we open any file from within the project A, TS eagerly imports all files from 28 projects. This means TS compiler has to load all files in these projects, parse them, generate type declaration, and run type check on them. This process is extremely slow, in our case it takes about 2 minutes.

We think a different approach might work better for the type checking in vscode. You folks know the TS compiler inside out, and you have enough context to tell if this idea is doable. What if we keep the d.ts files with the source code, and commit them to the repository, then the type checker reads them from the file system, and on another process we run the type declaration emitter, if there is any mismatch between the source, and the type declaration the compiler can issue an error. This allows to speed up the initial load, allow a better editor DX, and also we might be able to make this process multi-threaded.

[MartinJohns]

For some reasons TS compiler imports files as soon as it finds an import statement even though they are excluded in the tsconfig.exclude

This is working as intended and is documented quite prominently: https://www.typescriptlang.org/tsconfig#exclude

Important: exclude only changes which files are included as a result of the include setting. A file specified by exclude can still become part of your codebase due to an import statement in your code, a types inclusion, a /// <reference directive, or being specified in the files list.

It is not a mechanism that prevents a file from being included in the codebase - it simply changes what the include setting finds.

Differently said: exclude only filters include, nothing more.

[RyanCavanaugh]

Thanks for the feedback!

For some reasons TS compiler imports files as soon as it finds an import statement even though they are excluded in the tsconfig.exclude

exclude is only a filter on include, the docs are very explicit about this

TS eagerly imports all files from 28 projects. This means TS compiler has to load all files in these projects, parse them, generate type declaration, and run type check on them

It sounds like you're talking about in the editor? You can turn on the disableSourceOfProjectReferenceRedirect option to disable this loading.

What if we keep the d.ts files with the source code, and commit them to the repository, then the type checker reads them from the file system...

This is more or less what composite projects / project references do already, especially under incremental. It sounds like you might have some configuration issues based on what you've provided.

[aminpaks]

Thanks for your quick feedback.

exclude is only a filter on include, the docs are very explicit about this

True, but this behaviour easily leads to misconfiguration, incorrect, and unintended results. It would make our jobs much easier to detect the root cause by the compiler.

This is more or less what composite projects / project references do already, especially under incremental. It sounds like you might have some configuration issues based on what you've provided.

That's possible, we recently started looking into our configs. Is there a way for us to warm up the TS Server (maybe generate the d.ts files before hand) so we reduce the initial load times in vscode?

[aminpaks]

@RyanCavanaugh

It sounds like you're talking about in the editor? You can turn on the disableSourceOfProjectReferenceRedirect option to disable this loading.

Yes, we're working on improving our developer experience, and reduce the vscode memory consumption. With our project, vscode very much is unusable. Getting IntelliSense, or code-completion in a file takes up to a minute in some places, and eventually the TS server crashes.
I tried turning on disableSourceOfProjectReferenceRedirect setting, and the result is odd but the memory consumption is much better. When a file is opened I see lots of Output file 'built/path/to/file.d.ts' has not been built from 'path/to/file.ts' ts(6305) error on imports from referenced projects.
Not sure how it's supposed to work but I was expecting that the compiler builds the referenced projects for the imported files (in the background), and then free the memory, specially cause these imports are not open in the editor. Would you confirm is that error the intended behaviour?

Update:
This is our project entry's trace. It takes about 50s to load, and then TS server crashes (with disableSourceOfProjectReferenceRedirect: false), it takes about 13s to load, and parse all references.
It seems With disableSourceOfProjectReferenceRedirect: true, it's much faster as it doesn't load the referenced projects' source. Parsing references takes as long as load, parse, and type check of the project entry which is about 25s in total. We think this is reasonable, if we could make sure the references don't get stale when changed, using disableSourceOfProjectReferenceRedirect: true would be the way to go for us but that doesn't seem to be true.

[sheetalkamat]

disableSourceOfProjectReferenceRedirect options means that instead of picking up source from the referenced project we pick built d.ts files just like we do during compilation. This means that you would need to keep the output files of referenced project upto date. We do not do background in memory build for projects

[aminpaks]

@sheetalkamat

Thanks for the reply,

Is the expectation to run the TSC in a watch mode to compile the referenced projects separately? I don't know how efficient that would be. It seems none of the available options works well for our project.

[chbdetta]

I'm using a npm workspace setup, where my local workspaces are symlinked to the root node_modules. An interesting behavior I notice when using project reference in this setup is tsc doesn't seem to find the .d.ts declaration of a referenced project properly:

{
"references": [
  {'path': '../path/to/packageA/tsconfig.json'}
]
}

// importing a JS file
import {A} from '@my-scope/packageA/src/SomeJSModule'
// complains: can't not find the declaration file for ..., you can install '@types/my_scope_packageA'

However, the error goes away if I import this js file using a relative path.

The workaround I come up with is just using path to map the absolute imports to relative:

paths: {
  "@my-scope/packageA/*": ["../path/to/packageA/*"]
}

[RyanCavanaugh]

@chbdetta that would seem to indicate something isn't set up right with the symlinks. In general for a monorepo workspace, tsc -p even without project references should be resolving to packageA's .d.ts

[sheetalkamat]

Also you can use --traceResolution to see where typescript is looking for files to resolve

[chbdetta]

@RyanCavanaugh @sheetalkamat Here's a reproducing repo: https://github.com/chbdetta/ts-project-reference-playground

[RyanCavanaugh]

@chbdetta I don't get any errors running that

[chbdetta]

@RyanCavanaugh I left a @ts-expect-error on the error line

[sheetalkamat]

@chbdetta issue there is that js files discovered with node_modules differ from how ts are treated. The js file take into account maxNodeModuleJsDepth which defaults to 0 in the tsconfig, You would want to set that to different value to include js file. The current setup will give you error even if project B did not have reference to project A

[chbdetta]

@sheetalkamat does increasing maxNodeModuleJsDepth have some performance impact? And will it change the type resolution for non-workspace packages (e.g. it'll try to infer from a .js file of a non-workspace package instead of asking for installing a @type/* package?)

[sheetalkamat]

https://www.typescriptlang.org/tsconfig#maxNodeModuleJsDepth

[eabrouwer3]

Hey all! I tried searching through GitHub issues to see if there's a reason this wouldn't work or hasn't been done, but I've got a project with 8 libs and 3 apps using project references. It follows this basic structure:

tsconfig.base.json
workspaces/
  apps/
    app-1/
      tsconfig.json
    app-2/
      tsconfig.json
    .../
  libs/
    lib-1/
      tsconfig.json
    lib-2/
      tsconfig.json
    .../

There's a little more complexity, but that's the basic idea. However, I have a ton of repetition between all my tsconfig.json files that basically all look like this:

{
  "extends": ["../../../tsconfig.base.json"],
  "references": [
    {
      "path": "../../libs/lib-1"
    },
    {
      "path": "../../libs/lib-2"
    },
    // ...
  ]
}

It would be super nice if we could use a wildcard/glob in situations like this, where I could simply have all of my tsconfig.json files look like this:

{
  "extends": ["../../../tsconfig.base.json"],
  "references": ["../../libs/*"]
}

It also makes it way easier to add new libs/apps in the future to the monorepo. Also, I did try this and it does indeed not work:

error TS5083: Cannot read file '/Users/ebrouwer.dev/source/taxbit/belt/workspaces/libs/*/tsconfig.json'.

I'm happy to try and write a PR if people think this is possible and useful! Also happy to make a separate issue for this specifically, but this seemed like a fairly active ticket and the right place to start the discussion. Thanks!

[RyanCavanaugh]

@eabrouwer3 yes, a separate issue on that would be great. Thanks!

[eabrouwer3]

#56279 in case anyone's interested.

[raihle]

--clean confuses me. It only deletes files that would be emitted, but that seems almost exactly the opposite of what would be useful. How can I clean out "leftover" output files after renaming or removing an input file, without trashing the incremental build behavior?

[RyanCavanaugh]

How can I clean out "leftover" output files after renaming or removing an input file

Run --clean before deleting the file. Once you've removed the input file, tsc has no idea whether it's a file you made by hand, or a file from a previous invocation.

[Bessonov]

Run --clean before deleting the file. Once you've removed the input file, tsc has no idea whether it's a file you made by hand, or a file from a previous invocation.

I agree with @raihle - I'm not sure how it is useful. On the other hand, cleaning up stale files inside a CI pipeline with cached build artifacts could speed up the pipeline without worrying about leftovers. Perhaps something like .tscleanignore to let tsc know about files created manually? (I'm not sure if I have any, except .tsbuildinfo inside the build folder.)

EDIT: rsync has an --delete and --exclude flags:

--delete This tells rsync to delete extraneous files from the receiving side (ones that aren’t on the sending side), but only for the directories that are being synchronized. You must have asked rsync to send the whole directory (e.g. "dir" or "dir/") without using a wildcard for the directory’s contents (e.g. "dir/*") since the wildcard is expanded by the shell and rsync thus gets a request to transfer individual files, not the files’ parent directory. Files that are excluded from the transfer are also excluded from being deleted unless you use the --delete-excluded option or mark the rules as only matching on the sending side (see the include/exclude modifiers in the FILTER RULES section).

[RyanCavanaugh]

We're really not interested in breaking into jail with a feature that can delete files that aren't ones we would have normally overwritten anyway. The last thing I want to deal with is "tsc deleted all my source code because I misconfigured it, why would you even ship this??" reports. git clean exists; if your output folder truly has no other files, then rm exists, etc.. There are many, many, many dev tools capable of creating outputs that don't have built-in means to delete those outputs, and tsc isn't particularly unique here. Maybe shipping --clean in the first place was a mistake but I don't want to spend good money after bad if that's the case.

[Bessonov]

Thanks for taking the time to share your thoughts, @RyanCavanaugh! It's truly appreciated!

Personally, I can follow the 'rm'-way. The challenge lies in matching the configuration of tsc, such as rootDir and outDir, and different artifacts like *.(js|mjs|d.ts|d.mts|js.map|mjs.map|...).

(I don't expect a response - I am OK with it.)