Package Activation metrics

[walles]

Hi!

What about collecting metrics about package activation times?

When plugins crash there is automation in place for reporting that as bugs, but when plugins are slow to activate, that's really un-obvious (for package authors).

If package activation time stats could be published it would help package authors know if their packages are slow to activate.

In my case, I just discovered timecop, uninstalled git-plus and a number of other packages, and got my startup time improved by a lot. However, if these stats were easily available, maybe the authors of those packages could have fixed their packages instead, thereby improving everybody's startup time (and enabled me to keep using their packages).

[walles]

Pull request: #71

[thomasjo]

I'm on 👎 on this since the intended audience for the collected metrics is GitHub, and data is collected in order to gather useful information that might help improve the core experience. The intent is not to collect wide swath of metadata for the purpose of publication.

Package authors who actually care about their users can easily benchmark their packages using e.g. timecop. By publishing such statistics we risk, in some sense, of ending up with a "wall of shame".

If a package author doesn't care their package is slow, or does not have the resources to fix the problem, submit a PR (or fork the package).

/cc @atom/feedback

[walles]

Depends on whether you want to optimize the Atom core experience or the Atom experience.

And you are automatically informing plugin authors about crashes in their plugins, which indicates you actually do care about the full Atom experience rather than just the core one.

I'd argue that neither plugin authors nor plugin users actually know about the startup time footprints, so they aren't even able to care, and publicizing package startup time information would be really beneficial to everybody.

But all of this is of course up to you to decide, I'm fine with a decision either way.

[thomasjo]

First of all, we are not automatically informing package authors about runtime bugs. Rather end-users are presented with an option to open a (partially filled in) issue on the package repo.

There's a fundamental difference between a package having bugs (or crashing), and having poor (start-up) performance. Crashes/bugs can to a large extent be affected by other packages, and the number of package permutations is extremely high. That makes it very difficult for a package author to make their product bug free, etc.

On the other hand, checking the activation/start-up performance is trivial in comparison. Package activation is (when done appropriately) unaffected by other packages. Unless a package absolutely must activate when Atom loads, it should use deferred activation via supported mechanisms. All of this can be easily tested and benchmarked by package authors prior to releasing code. There might be edge cases here, but nothing that (in my opinion) requires collecting metrics at this scale.

[walles]

I still think:

1. TimeCop is great!
2. It's not necessarily obvious to plugin authors that they could / should run it to check for startup time.

Also, if one plugin is very slow that's easy enough for a plugin author to find and fix, but if lots of plugins are kind of slow that can still affects end users a lot without it being obvious at all to plugin writers, even the ones who actually do profile their plugins.

[Ben3eeE]

It's not necessarily obvious to plugin authors that they could / should run it to check for startup time.

What we can do and would accept a PR for is to better document timecop, deferred activation and other start up performance related issues and fixes in the Atom flight manual to make it more obvious to package authors. It is currently mostly covered in the debugging section which might not be the first thing an aspiring package author checks before publishing. How can we make it more obvious?

[thomasjo]

If you think it's not obvious to package authors that they can/should benchmark their packages, using for example timecop, then that is solved by educating them. It should not be solved by collecting relevant metrics, and then publicly "shaming" slow packages (and a package might have a perfectly valid reason for being slow, which is well understood by its users).

The proposed solution (collecting metrics and publishing it) is masking the real the problem, which is that we probably need better/more documentation around "best" practice package development.

[winstliu]

I agree with all the points @thomasjo has presented here, and am similarly 👎 on including this in metrics. If it is non-obvious that Timecop displays package load times, we should make it more obvious as @Ben3eeE said.

[damieng]

We could take a leaf out of the Windows playbook and put up a warning if Atom is slow to start and offer to pop up Timecop.

[thomasjo]

We could take a leaf out of the Windows playbook and put up a warning if Atom is slow to start and offer to pop up Timecop.

Unless that would only be enabled in dev mode, we would need to introduce a permanent way to disable that globally (or possibly whitelisting certain packages), since some packages might "have to" be slow to activate. And that might be fine for users of said hypothetical package. We shouldn't annoy end-users because a package they are using is slow. That just adds injury to insult, right?

At the end of the day, this boils down to educating and assisting package developers.

[damieng]

I was thinking we could just put something in the status bar that says "Atom took 2.3 seconds to start. Show package start-up performance." with the second part being a link to timecop.

That way it's non-invasive.

"We shouldn't annoy end-users because a package they are using is slow. That just adds injury to insult, right?"

I'd say they are already annoyed but put the blame squarely at Atom's feet.