Wallaby is really slow on large angular project

[tamascsaba]

Tipical angular based project with 2005 files and lot of dependencies.
Webpack compile takes long time (8-10s).

I tried:
https://github.com/stephencookdev/speed-measure-webpack-plugin
Onfurtunetly i don't see any console message or outputTarget does not emmit :(

Video:
https://youtu.be/Ao2jnnOyK_E

Config:

const wallabyWebpack = require('wallaby-webpack');
const path = require('path');
const compilerOptions = require('./tsconfig.json').compilerOptions;

module.exports = function (wallaby) {
  const webpackPostprocessor = wallabyWebpack({
    entryPatterns: [
      'src/client/testing/setup-wallaby.js',
      'src/client/**/*.test.js'
    ],
    module: {
      rules: [
        {test: /\.css$/, loader: 'raw-loader'},
        {test: /\.html$/, loader: 'raw-loader'},
        {test: /\.ts$/, loader: '@ngtools/webpack', include: /node_modules/, query: {tsConfigPath: 'tsconfig.json'}},
        {test: /\.js$/, loader: 'angular2-template-loader', exclude: /node_modules/},
        {test: /\.styl$/, loaders: ['raw-loader', 'stylus-loader']},
        {test: /\.less$/, loaders: ['raw-loader', 'less-loader']},
        {test: /\.scss$|\.sass$/, loaders: ['raw-loader', {loader: 'sass-loader', options: {
          includePaths: ['src/styles']
        }}]},
        {test: /\.(jpg|png)$/, loader: 'url-loader?limit=128000'}
      ]
    },

    resolve: {
      extensions: ['.js', '.ts'],
      modules: [
        path.join(wallaby.projectCacheDir, 'src/')
      ]
    },

    node: {
      fs: 'empty',
      net: 'empty',
      tls: 'empty',
      dns: 'empty'
    }
  });

  return {
    debug: true,
    files: [
      {pattern: 'src/**/*.ts', load: false},
      {pattern: 'src/**/*.d.ts', ignore: true},
      {pattern: 'src/**/*.css', load: false},
      {pattern: 'src/**/*.scss', load: false},
      {pattern: 'src/**/*.html', load: false},
      {pattern: 'src/**/*.json', load: false},
      {pattern: 'src/**/*test.ts', ignore: true}
    ],

    tests: [
      {pattern: 'src/**/*test.ts', load: false}
    ],

    testFramework: 'jasmine',

    compilers: {
      '**/*.ts': wallaby.compilers.typeScript(compilerOptions)
    },

    middleware: function (app, express) {
      app.use('/assets', express.static(path.join(__dirname, 'src/assets')));
    },

    env: {
      kind: 'chrome'
    },

    postprocessor: webpackPostprocessor,

    setup: function () {
      window.__moduleBundler.loadTests();
    }
  };
};

[ArtemGovorov]

Thanks for providing the details.

Webpack compile takes long time (8-10s).

It looks like the most time is spent in Webpack compilation of your project and what Wallaby is doing is just calling Webpack, so the time is consumed by Webpack (not by Wallaby directly).

• How long does it take karma to run your tests (including project rebuild/ng build)?
• Can you create a simple spec file (with no dependencies) in your project, add an empty it('works', () => {}) test to it and see how much time it takes Webpack to build when you change the spec?
• What version of wallaby-webpack module are you using?

[tamascsaba]

I use the latest "wallaby-webpack": "^3.9.11",. I created a video from karma and wallaby speed difference, unfortunetly wallaby is slower with 5 secunds.

https://youtu.be/wed3ewzrVFc

My karma.conf.js use similar configuration like wallaby.js file.
It use webpack under the hood with @angular-devkit/build-angular

module.exports = (config) => {
  config.set({
    basePath: '',
    frameworks: ['jasmine', '@angular-devkit/build-angular'],
    plugins: [
      require('karma-jasmine'),
      require('karma-chrome-launcher'),
      require('karma-coverage-istanbul-reporter'),
      require('@angular-devkit/build-angular/plugins/karma')
    ],
    client:{
      clearContext: false
    },
    coverageIstanbulReporter: {
      dir: require('path').join(__dirname, 'coverage'),
      reports: ['html', 'lcovonly', 'cobertura', 'json-summary'],
      fixWebpackSourcePaths: true
    },
    reporters: ['progress', 'coverage-istanbul'],
    hostname: '127.0.0.1',
    port: 9876,
    colors: true,
    logLevel: config.LOG_INFO,
    autoWatch: true,
    browserNoActivityTimeout: 9000000,
    browsers: ['Chrome'],
    singleRun: true,
    captureTimeout: 60000
  });
};

[ArtemGovorov]

Thanks for providing the details. From your video it looks like you're adding a simple test to an existing spec with some other tests/dependencies.

Can you please create a empty spec file (with no dependencies) in your project, add an empty it('works', () => {}) test to it and see how much time it takes Webpack to build when you change the spec?

[ArtemGovorov]

Could you please also do the following:

• open ./node_modules/wallaby-webpack/ folder

cd ./node_modules/wallaby-webpack/

• install v8-profiler module in there (or in your projects’ root node modules):

npm install v8-profiler

• replace the ./node_modules/wallaby-webpack/index.js file content with this content (it is the same code but with added CPU profile recording).
• start wallaby, let the initial run finish, then make any change to the test file you were changing (that takes Webpack 10 seconds), and once the test results are ready, there’ll be a new file created in ./node_modules/wallaby-webpack/ - profile1.json, please send us the file to hello@wallabyjs.com or attach it to this issue.

[tamascsaba]

Unfortunately github does not support json attacment, so i renamed to log.
profile1.log
I tried with full empty spec, but is is also very slow:
https://youtu.be/O4T9UbShQIs

[ArtemGovorov]

Thanks for providing the profile log and checking an empty spec.

From the profile log it looks like Webpack is doing a lot of file reads that it shouldn't really be doing (assuming that you have attached the profile file from making a change in an existing file, not running Wallaby from the scratch).

I have prepared some more logging to find out what file Webpack is trying to read in your case.

Could you please also do the following:

• open ./node_modules/wallaby-webpack/ folder
• replace the ./node_modules/wallaby-webpack/index.js file content with this content (it is the same code but with added CPU profile recording + logging helper).
• replace the ./node_modules/wallaby-webpack/lib/WallabyInputFileSystem.js file content with this content (it is the same code but with added logging of what files Webpack is reading).
• edit your Wallaby config to set debug: false (to log less unrelated stuff),
• start wallaby, let the initial run finish, then make any change to the empty spec test file, and once the test results are ready, please open Wallaby Console and paste the results from it here.

With this setup, every file path that Webpack is trying to read (when you are doing incremental changes to the empty spec file) should be logged.

A couple of other questions:

• do you have any nested node_modules in your src folder?
• do you have any symlinks in your src folder?

[tamascsaba]

profile1.txt
I don't have any symlinks and nested node_modules. I use yarn with an angular-cli based project.

[ArtemGovorov]

@tamascsaba Thanks, but you have attached a profile, and I was asking for Wallaby Console output. Could you please follow these steps above and attach Wallaby Console output?

[tamascsaba]

Huups, i forgot.
console_output.log
It is one full build and test and a change in an angular based test.

[ArtemGovorov]

Thanks!

Could you please share

• your package.json file's dependencies and devDependencies sections (where all your angular dependencies are listed);
• your tsconfig.json file.

Also, I understand that it may not be possible, but still is worth asking - is there a chance that you could share the project's access with us (privately) so we could profile it and reproduce the issue locally?

[ArtemGovorov]

Also, could you please share your src/client/testing/setup-wallaby.ts file?

[tamascsaba]

I'am so sorry, but is is a private enterprise project. I can't give access for you, but I am open for a conference call (skype, hangouts, messanger, slack, bluejeans or any tool is perfect for me).

My working time is in UTC/GMT+2 timezone, so UTC 8:00 AM - 2:00 PM timerange fits for me the best, but we could also arrange a meeting in other time, if necessary.

package.json

  "dependencies": {
    "@angular/common": "^6.1.7",
    "@angular/compiler": "^6.1.7",
    "@angular/core": "^6.1.7",
    "@angular/forms": "^6.1.7",
    "@angular/http": "^6.1.7",
    "@angular/platform-browser": "^6.1.7",
    "@angular/platform-browser-dynamic": "^6.1.7",
    "@angular/router": "^6.1.7",
    "billboard.js": "1.5.x",
    "classlist-polyfill": "^1.2.0",
    "core-js": "^2.5.7",
    "d3": "^5.7.0",
    "d3-sankey": "^0.7.1",
    "flatpickr": "4.3.0",
    "fuzzysearch-js": "^0.2.0",
    "lodash": "^4.17.10",
    "material-design-icons-iconfont": "^3.0.3",
    "moment": "^2.22.2",
    "moment-duration-format": "^2.2.2",
    "moment-timezone": "^0.5.21",
    "navigator": "^1.0.1",
    "rxjs": "^6.3.2",
    "smooth-scrollbar": "^7.4.1",
    "tslib": "^1.9.3",
    "zone.js": "^0.8.26"
  },
  "devDependencies": {
    "@angular-devkit/build-angular": "^0.8.1",
    "@angular/cli": "^6.2.1",
    "@angular/compiler-cli": "^6.1.7",
    "@angular/language-service": "^6.1.7",
    "@balabit/lint": "^3.5.0",
    "@slack/client": "^4.6.0",
    "@types/chai": "^4.1.4",
    "@types/d3": "^5.0.0",
    "@types/jasmine": "^2.8.8",
    "@types/lodash": "^4.14.116",
    "@types/moment-timezone": "^0.5.8",
    "@types/node": "^8.10.25",
    "angular2-template-loader": "^0.6.2",
    "browserstack-local": "^1.3.4",
    "chai": "^4.1.2",
    "codeceptjs": "^1.3.3",
    "codelyzer": "^4.4.4",
    "express": "^4.16.3",
    "gherkin": "^5.1.0",
    "husky": "^0.14.3",
    "jasmine-core": "^3.2.1",
    "karma": "^3.0.0",
    "karma-chrome-launcher": "^2.2.0",
    "karma-coverage-istanbul-reporter": "^2.0.3",
    "karma-jasmine": "^1.1.2",
    "mockdate": "^2.0.2",
    "nightmare": "^3.0.1",
    "nightmare-upload": "^0.1.1",
    "sass-lint": "^1.12.1",
    "snyk": "^1.95.3",
    "ts-node": "^7.0.1",
    "tsconfig-paths": "^3.5.0",
    "tslint": "^5.11.0",
    "typescript": "2.9.x",
    "unirest": "^0.5.1",
    "v8-profiler": "^5.7.0",
    "wallaby-webpack": "^3.9.11",
    "webdriverio": "^4.13.2"
  },

tscofing.json

{
  "compilerOptions": {
    "allowSyntheticDefaultImports": true,
    "allowUnreachableCode": true,
    "baseUrl": "src/client",
    "declaration": false,
    "emitDecoratorMetadata": true,
    "esModuleInterop": true,
    "experimentalDecorators": true,
    "importHelpers": true,
    "lib": ["es5", "es6", "es7", "dom"],
    "module": "commonjs",
    "noImplicitAny": false,
    "noUnusedLocals": true,
    "outDir": "build",
    "pretty": true,
    "removeComments": false,
    "sourceMap": false,
    "strictNullChecks": false,
    "target": "es5",
    "types": [
      "node",
      "jasmine",
      "chai",
      "../../src/client/testing/e2e/types"
    ]
  },
  "exclude": [
    "node_modules"
  ],
  "awesomeTypescriptLoaderOptions": {
    "useWebpackText": true,
    "forkChecker": true,
    "useCache": true
  },
  "compileOnSave": false,
  "buildOnSave": false,
  "atom": { "rewriteTsconfig": false },
  "angularCompilerOptions": {
    "preserveWhitespaces": false
  }
}

setup-wallaby.ts

import 'zone.js/dist/zone';
import 'zone.js/dist/long-stack-trace-zone';
import 'zone.js/dist/proxy';
import 'zone.js/dist/sync-test';
import 'zone.js/dist/fake-async-test';
import 'zone.js/dist/jasmine-patch';

import {TestBed} from '@angular/core/testing';

import {
  BrowserDynamicTestingModule,
  platformBrowserDynamicTesting
} from '@angular/platform-browser-dynamic/testing';

TestBed.initTestEnvironment(
  BrowserDynamicTestingModule,
  platformBrowserDynamicTesting()
);

[ArtemGovorov]

Could you please record Webpack profile with the following change in your Wallaby config:

    module: {
      rules: [
        {test: /\.css$/, loader: 'raw-loader'},
        {test: /\.html$/, loader: 'raw-loader'},
        {test: /\.ts$/, loader: '@ngtools/webpack', include: /node_modules/, query: {tsConfigPath: 'tsconfig.json'}},
        {test: /\.js$/, loader: 'angular2-template-loader', exclude: /node_modules/},
        {test: /\.styl$/, loaders: ['raw-loader', 'stylus-loader']},
        {test: /\.less$/, loaders: ['raw-loader', 'less-loader']},
-       {test: /\.scss$|\.sass$/, loaders: ['raw-loader', {loader: 'sass-loader', options: {
-         includePaths: ['src/styles']
-       }}]},
+       {test: /\.scss$|\.sass$/, loader: 'raw-loader'},
        {test: /\.(jpg|png)$/, loader: 'url-loader?limit=128000'}
      ]
    },

[tamascsaba]

profile1.log
Huuu, It works much faster than bebore. The compile time is reduced to 3-4s.

[ArtemGovorov]

Awesome, thanks for checking!
For some reason sass-loader in your project was causing full project re-scan by Webpack, I'm not sure why it was happening (can't see it from the log/profile), but using raw-loader is a good solution.

Please also update your wallaby-webpack package to v3.9.12, it should further improve the performance for you - let me know how much time the incremental changes take now (with raw-loader and wallaby-webpack v3.9.12).

[tamascsaba]

@ArtemGovorov thank you very much. It was a very good support process. We are very happy with the result 👍

Sidenote: I think it is the related sass-loader issue is webpack-contrib/sass-loader#296

[ArtemGovorov]

Awesome, thanks for the update!

Had updating wallaby-webpack package to v3.9.12 improved the performance a bit more for you?

[tamascsaba]

Yes, it is improved the recompile time, which is 1-2s now. It is amazing 👍

I found an other solution for sass-loader issue, becouse with raw-loader failed lot of tests (all components tests), Angular could not compile the component styles and TestBed.createComponent failed.

For this reason sass compiling is necessary, so i found https://github.com/yibn2008/fast-sass-loader

{test: /\.scss$|\.sass$/, loaders: ['raw-loader', {loader: 'fast-sass-loader', options: {
    includePaths: ['src/styles']
}}]},

It works perfectly and blazingly fast.

Other basic option or workaround use https://github.com/zuanoc/empty-string-loader, but it is not mantained any more, i think. Last commit was 2 years ago.