Task Hierarchy & Prerequisites

[JasonGore]

Describe the bug

I'm not sure if this is intended behavior, a bug or a potential perf optimization, but I will write this up as a bug for initial discussion. I long thought lage was behaving a certain way but it turns out I was wrong the whole time. The behavior described in this issue seems to be consistent for all versions from 0.29.0 to 2.5.19.

Given a lage config:

module.exports = {
  "pipeline": {
    "build": [
      "^build"
    ],
    "test": [
      "build"
    ],
    "lint": []
  },
  "npmClient": "yarn"
};

that defines build of dependent packages a prerequisite for build of a given package. Additionally, it defines build of its own package as a prerequisite for test task defined in that package.

However, it appears that this actually requires build of the entire repo when running yarn lage test, even for standalone packages and leaf nodes that aren't dependents of any package with test tasks.

The package hierarchy is:

flowchart TD
    base["package-base"]
    leaf["package-leaf"]
    test["package-test"]
    standalone["package-standalone"]

    test --> base
    leaf --> base

Loading

ONLY package-test defines a test task. Combined with lage.config.js this should result in the task hierarchy for yarn lage test:

flowchart TD
    basebuild["package-base#build"]
    testbuild["package-test#build"]
    testtest["package-test#test"]

    basebuild --> testbuild
    testbuild --> testtest

Loading

To Reproduce

Steps to reproduce the behavior:

1. git clone https://github.com/JasonGore/lage-leaf-nodes
2. yarn
3. yarn lage test

Actual Behavior

Running yarn lage test actually runs build against all packages in the repo, even standalone packages and leaf nodes with no test scripts.

D:\git\repros\lage-leaf-nodes>yarn lage test
yarn run v1.22.17
$ lage test
info Lage task runner - let's make it
info package-base build ▶️ start
info package-leaf build ▶️ start
info package-standalone build ▶️ start
info package-base build ✔️ done  - 0.29s
info package-test build ▶️ start
info package-leaf build ✔️ done  - 0.29s
info package-standalone build ✔️ done  - 0.29s
info package-test build ✔️ done  - 0.28s
info package-test test ▶️ start
info package-test test ✔️ done  - 0.28s
info � Summary
info
info [Tasks Count] success: 5, skipped: 0, incomplete: 0
info ----------------------------------------------
info Took a total of 1.00s to complete
Done in 1.83s.

Expected behavior

A directed command like yarn lage test only runs the prerequisites in the task hierarchy. In other words, build for packages that have test tasks or are dependencies of packages with test tasks. This is also the behavior that occurs when yarn lage test --to package-test is run.

$ lage test
info Lage task runner - let's make it
info package-base build ▶️ start
info package-base build ✔️ done  - 0.29s
info package-test build ▶️ start
info package-test build ✔️ done  - 0.28s
info package-test test ▶️ start
info package-test test ✔️ done  - 0.28s
info � Summary
info
info [Tasks Count] success: 3, skipped: 0, incomplete: 0
info ----------------------------------------------
info Took a total of 1.00s to complete
Done in 1.83s.

Environment/Troubleshooting

If this is an issue with run order of tasks, please paste in the result of lage info test:

info {
info   "command": [
info     "test"
info   ],
info   "scope": [
info     "package-base",
info     "package-leaf",
info     "package-standalone",
info     "package-test"
info   ],
info   "packageTasks": [
info     {
info       "id": "package-test#test",
info       "command": [
info         "yarn",
info         "run",
info         "test"
info       ],
info       "dependencies": [
info         "package-test#build"
info       ],
info       "workingDirectory": "packages/package-test",
info       "package": "package-test",
info       "task": "test"
info     },
info     {
info       "id": "package-base#build",
info       "command": [
info         "yarn",
info         "run",
info         "build"
info       ],
info       "dependencies": [],
info       "workingDirectory": "packages/package-base",
info       "package": "package-base",
info       "task": "build"
info     },
info     {
info       "id": "package-leaf#build",
info       "command": [
info         "yarn",
info         "run",
info         "build"
info       ],
info       "dependencies": [],
info       "workingDirectory": "packages/package-leaf",
info       "package": "package-leaf",
info       "task": "build"
info     },
info     {
info       "id": "package-standalone#build",
info       "command": [
info         "yarn",
info         "run",
info         "build"
info       ],
info       "dependencies": [],
info       "workingDirectory": "packages/package-standalone",
info       "package": "package-standalone",
info       "task": "build"
info     },
info     {
info       "id": "package-test#build",
info       "command": [
info         "yarn",
info         "run",
info         "build"
info       ],
info       "dependencies": [
info         "package-base#build"
info       ],
info       "workingDirectory": "packages/package-test",
info       "package": "package-test",
info       "task": "build"
info     }
info   ]
info }

[kenotron]

Okay, chatting offline with @JasonGore - the intended behavior should be the same as if you did the --to when the "test" script didn't apply to the "entry packages".

The root cause is that there are "phantom" scripts that we pretend all packages to have just so we can still run dependencies-tasks for packages that didn't clearly define something. This is so we don't "under" run stuff. However, in this case, the "entry package" of "standalone" does not have a "test" script. The question would be - should we keep running the phantom script's dependencies?

[dobesv]

I wonder if we could have different type values in the pipeline, for example type: 'virtual' would be a task that always exists even if there's no script. type: 'script' would run a script. And type: 'optional' would only create a dependency graph if there's a script there.

[dobesv]

I worked on it a bit in here:

#865

One thing I did find is that I can workaround this in lage.config by writing some code that calculates which packages should have the script and only defining the script in those packages. This avoids the phantom dependency issue. It would be nice not to have to do that.