Unit Testing with jbang

[maxandersen]

I've been pondering how to go about writing unit tests for jbang scripts/apps.

There are two parts afaics, one on how the tests look like the other how to run them.

The first part on how they look like I think is fairly straight forward now we have //SOURCES feature.

i.e. if you have script looking like:

///usr/bin/env jbang "$0" "$@" ; exit $?

class helloworld {

    String greeting(String who) {
          return "Hello " + who;
    }

    public static void main(String[] args) {
        helloworld hw = new helloworld();

        if(args.length==0) {
            System.out.println(hw.greeting("World"));
        } else {
            System.out.print(hw.greeting(args[0]));
              
        }
    }
}

Then the most basic test of that would be:

//DEPS org.junit.jupiter:junit-jupiter-engine:5.2.0
//DEPS org.hamcrest:hamcrest-library:2.1

//SOURCES helloworld.java

import java.io.PrintWriter;

import org.junit.jupiter.api.Test;
import static org.hamcrest.MatcherAssert.assertThat;
import static org.hamcrest.Matchers.equalTo;

public class TestLauncher {

    @Test 
    void testGreeting() {
        helloworld hw = new helloworld();
        assertThat(hw.greeting("me"), equalTo("Hello me"));
        assertThat(hw.greeting(null), equalTo("Hello"));
    }
}

This setup actually works (kinda) today and you can use idea with jbang edit TestLauncher and you can run the tests just fine directly from the ide.

I'm wondering how/if we should add some native test support to jbang to make it easier.

i.e. jbang unittest TestLauncher.java could generate what corresponds to:

jbang org.junit.platform:junit-platform-console-standalone:1.7.0 --classpath 'jbang info classpath testscript.java' --scan-classpath

where jbang info classpath generates the classpath.

then junit 5 test runner would find all identifiable tests and run them.

Could also just tell users to run the above command but a) its quite long and b) it won't work on windows.

Ideas/thoughts on testing with jbang?

[OLibutzki]

My thoughts:

I don't use a CI server to build my scripts continuously. So when to run the tests?

What about linking the test classes in the script file itself?

When you run the script the tests are executed first and the script is only executed if the tests pass.

You need to add some convenience stuff like caching the results if the class under test did not change or skipping tests explicitly, but this is my first take on this.

[OLibutzki]

could be; but again no real way to check if a test is 100% dependent on just the code could also be dependent on external changes.

Taking about a unit test, I assume that the test results only depend on the test class and the class under test. If both don't change, the test should always return the same result.

Otherwise bayou would get into trouble with your run tests before pushing approach as you don't know if the script breaks right after running tests because of some external changes.

I always run my tests locally before I push to a CI server...I assume same applies here for the scripts.

Like a CI server it's just an additional safety net. Of course the developer should execute the tests before pushing the changes.

Instead of linking the test classes from the script one could think about a naming convention, but in my opinion it's ok that the script knows its tests as they act as a kind of specification.

Regarding convenience:

• cache test results if test classes and classes under test didn't change

Add options to

• skip test execution
• run script regardless of test results
• force test execution although classes did not change

[maxandersen]

If both don't change, the test should always return the same result.

you are assuming those who write jbang scripts with tests will have done so perfectly all knowing and with all mutating state mocked away... I think that is pretty unrealistic to assume is the case for all users :)

what is the point of "run script regardless of test results" ?
what is the value of running the tests to then just run the script ?

[OLibutzki]

you are assuming those who write jbang scripts with tests will have done so perfectly all knowing and with all mutating state mocked away... I think that is pretty unrealistic to assume is the case for all users :)

Yes, that's my assumption and my understanding of what a unit test is all about.

what is the point of "run script regardless of test results" ?

The default flow could be:
(if tests are present && at least one class is changed since last test run) run tests -> (if tests ran successfully) run script

You need some jbang options to overrule the default flow:

• skip tests
• run script although at least one test failed
• force test execution although no classes changed since last test execution

Of course one could also think about another default flow which does not execute any tests by default. Then you need to specify explicitly explcitly from the command line that you want to execute the tests.

[helpermethod]

Something like executable examples would be desirable (similar to doctest in Python).

https://byexamples.github.io/byexample/languages/java

[quintesse]

@helpermethod I wanted to point out that we have support for running MarkDown files, which would be very similar, I think, to what you linked to: https://www.jbang.dev/documentation/guide/latest/usage.html#running-markdowns-md-experimental

But I see that part of the documentation is completely wrong (basically just a copy&paste from the previous section). So we need to fix that first. (See #1239)

[kenyee]

Would also be nice to have a template with unit tests for a JBang script w/ multiple source files as an example.

[albertocavalcante]

Just wondering if there's a current recommendation / best practice regarding this.

[maxandersen]

Another one I realize is possible given junit-team have a jbang catalog.

jbang cli@junit-team execute  --scan-classpath -cp `jbang info classpath <yourtests>`

that yourtests can be anything runnable by jbang, but for this usecase a myTest.java which depends on jars and/or sources with what you want to test.

[albertocavalcante]

I'm having issue trying these approaches. I assume the catalog has changed in the meantime?

 jbang init --fresh --verbose -t junit myscript.java
[jbang] [0:140] jbang version 0.124.0
[jbang] [0:146] Reading catalog from classpath:/jbang-catalog.json
[jbang] [0:161] Reading catalog from classpath:/jbang-catalog.json
[jbang] [0:225] Requesting HTTP GET https://raw.githubusercontent.com/jbanghub/jbang-catalog/main/jbang-catalog.json
[jbang] [0:225] Headers {User-Agent=[JBang/0.124.0 (Mac OS X/15.1/aarch64) Java/21.0.6/Homebrew]}
[jbang] [0:590] Downloaded file https://raw.githubusercontent.com/jbanghub/jbang-catalog/main/jbang-catalog.json
[jbang] [0:591] Deleting folder /Users/albertocavalcante/.jbang/cache/urls/9e5f036f349563e036ce916208ce6a69cea71bd388666b1793f22c8ed96ad73a.old
[jbang] [0:593] Deleting folder /Users/albertocavalcante/.jbang/cache/urls/9e5f036f349563e036ce916208ce6a69cea71bd388666b1793f22c8ed96ad73a-meta.old
[jbang] [0:593] Reading catalog from https://raw.githubusercontent.com/jbanghub/jbang-catalog/main/jbang-catalog.json
[jbang] [0:595] Obtained catalog from https://raw.githubusercontent.com/jbanghub/jbang-catalog/main/jbang-catalog.json
[jbang] [0:595] [ERROR] Could not find init template named: junit. Try run with --fresh to get latest catalog updates.
dev.jbang.cli.ExitException: Could not find init template named: junit. Try run with --fresh to get latest catalog updates.
        at dev.jbang.cli.Init.doCall(Init.java:68)
        at dev.jbang.cli.BaseCommand.call(BaseCommand.java:145)
        at dev.jbang.cli.BaseCommand.call(BaseCommand.java:21)
        at picocli.CommandLine.executeUserObject(CommandLine.java:2041)
        at picocli.CommandLine.access$1500(CommandLine.java:148)
        at picocli.CommandLine$RunLast.executeUserObjectOfLastSubcommandWithSameParent(CommandLine.java:2461)
        at picocli.CommandLine$RunLast.handle(CommandLine.java:2453)
        at dev.jbang.cli.JBang$3.handle(JBang.java:150)
        at dev.jbang.cli.JBang$3.handle(JBang.java:145)
        at picocli.CommandLine$AbstractParseResultHandler.execute(CommandLine.java:2273)
        at picocli.CommandLine$RunLast.execute(CommandLine.java:2417)
        at picocli.CommandLine.execute(CommandLine.java:2170)
        at dev.jbang.Main.main(Main.java:15)
[jbang] [0:595] If you believe this a bug in jbang, open an issue at https://github.com/jbangdev/jbang/issues

[albertocavalcante]

I can't find any reference to JUnit here: https://github.com/search?q=repo%3Ajbanghub%2Fjbang-catalog%20junit&type=code

Am I looking in the wrong place?

Also:

❯ jbang --verbose cli@junit-team 
[jbang] [0:146] jbang version 0.124.0
[jbang] [0:153] Resolving resource ref: cli@junit-team
[jbang] [0:155] Reading catalog from /Users/albertocavalcante/dev/workspace/rewrite-jbang/jbang-catalog.json
[jbang] [0:240] Requesting HTTP GET https://junit.jbang.dev/jbang-catalog.json
[jbang] [0:240] Headers {User-Agent=[JBang/0.124.0 (Mac OS X/15.1/aarch64) Java/21.0.6/Homebrew]}
[jbang] [0:257] Retry using cache-evict: cli@junit-team
[jbang] [0:257] Requesting HTTP GET https://junit.jbang.dev/jbang-catalog.json
[jbang] [0:257] Headers {User-Agent=[JBang/0.124.0 (Mac OS X/15.1/aarch64) Java/21.0.6/Homebrew]}
[jbang] [0:258] [ERROR] Could not download https://junit.jbang.dev/jbang-catalog.json
dev.jbang.cli.ExitException: Could not download https://junit.jbang.dev/jbang-catalog.json
        at dev.jbang.source.resolvers.RemoteResourceResolver.fetchFromURL(RemoteResourceResolver.java:119)
        at dev.jbang.source.resolvers.RemoteResourceResolver.resolve(RemoteResourceResolver.java:41)
        at dev.jbang.source.resolvers.CombinedResourceResolver.lambda$resolve$0(CombinedResourceResolver.java:27)
        at java.base/java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:197)
        at java.base/java.util.Spliterators$ArraySpliterator.tryAdvance(Spliterators.java:1034)
        at java.base/java.util.stream.ReferencePipeline.forEachWithCancel(ReferencePipeline.java:129)
        at java.base/java.util.stream.AbstractPipeline.copyIntoWithCancel(AbstractPipeline.java:527)
        at java.base/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:513)
        at java.base/java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:499)
        at java.base/java.util.stream.FindOps$FindOp.evaluateSequential(FindOps.java:150)
        at java.base/java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
        at java.base/java.util.stream.ReferencePipeline.findFirst(ReferencePipeline.java:647)
        at dev.jbang.source.resolvers.CombinedResourceResolver.resolve(CombinedResourceResolver.java:29)
        at dev.jbang.source.ResourceResolver.resolve(ResourceResolver.java:52)
        at dev.jbang.source.ResourceRef.forResource(ResourceRef.java:131)
        at dev.jbang.catalog.Catalog.getByRef(Catalog.java:190)
        at dev.jbang.catalog.Catalog.getByName(Catalog.java:145)
        at dev.jbang.catalog.Alias.fromCatalog(Alias.java:278)
        at dev.jbang.catalog.Alias.merge(Alias.java:202)
        at dev.jbang.catalog.Alias.get(Alias.java:167)
        at dev.jbang.source.resolvers.AliasResourceResolver.resolve(AliasResourceResolver.java:39)
        at dev.jbang.source.ResourceResolver.resolve(ResourceResolver.java:52)
        at dev.jbang.source.ProjectBuilder.lambda$resolveChecked$1(ProjectBuilder.java:268)
        at dev.jbang.util.Util.withConfig(Util.java:223)
        at dev.jbang.util.Util.withConfig(Util.java:216)
        at dev.jbang.util.Util.withCacheEvict(Util.java:210)
        at dev.jbang.util.Util.withCacheEvict(Util.java:206)
        at dev.jbang.source.ProjectBuilder.resolveChecked(ProjectBuilder.java:268)
        at dev.jbang.source.ProjectBuilder.build(ProjectBuilder.java:245)
        at dev.jbang.cli.Run.doCall(Run.java:73)
        at dev.jbang.cli.BaseCommand.call(BaseCommand.java:145)
        at dev.jbang.cli.BaseCommand.call(BaseCommand.java:21)
        at picocli.CommandLine.executeUserObject(CommandLine.java:2041)
        at picocli.CommandLine.access$1500(CommandLine.java:148)
        at picocli.CommandLine$RunLast.executeUserObjectOfLastSubcommandWithSameParent(CommandLine.java:2461)
        at picocli.CommandLine$RunLast.handle(CommandLine.java:2453)
        at dev.jbang.cli.JBang$3.handle(JBang.java:150)
        at dev.jbang.cli.JBang$3.handle(JBang.java:145)
        at picocli.CommandLine$AbstractParseResultHandler.execute(CommandLine.java:2273)
        at picocli.CommandLine$RunLast.execute(CommandLine.java:2417)
        at picocli.CommandLine.execute(CommandLine.java:2170)
        at dev.jbang.Main.main(Main.java:15)
Caused by: java.net.UnknownHostException: junit.jbang.dev
        at java.base/sun.nio.ch.NioSocketImpl.connect(NioSocketImpl.java:567)
        at java.base/java.net.SocksSocketImpl.connect(SocksSocketImpl.java:327)
        at java.base/java.net.Socket.connect(Socket.java:751)
        at java.base/sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:304)
        at java.base/sun.security.ssl.BaseSSLSocketImpl.connect(BaseSSLSocketImpl.java:181)
        at java.base/sun.net.NetworkClient.doConnect(NetworkClient.java:183)
        at java.base/sun.net.www.http.HttpClient.openServer(HttpClient.java:531)
        at java.base/sun.net.www.http.HttpClient.openServer(HttpClient.java:636)
        at java.base/sun.net.www.protocol.https.HttpsClient.<init>(HttpsClient.java:264)
        at java.base/sun.net.www.protocol.https.HttpsClient.New(HttpsClient.java:377)
        at java.base/sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.getNewHttpClient(AbstractDelegateHttpsURLConnection.java:193)
        at java.base/sun.net.www.protocol.http.HttpURLConnection.plainConnect0(HttpURLConnection.java:1252)
        at java.base/sun.net.www.protocol.http.HttpURLConnection.plainConnect(HttpURLConnection.java:1138)
        at java.base/sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.connect(AbstractDelegateHttpsURLConnection.java:179)
        at java.base/sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1690)
        at java.base/sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1614)
        at java.base/java.net.HttpURLConnection.getResponseCode(HttpURLConnection.java:531)
        at java.base/sun.net.www.protocol.https.HttpsURLConnectionImpl.getResponseCode(HttpsURLConnectionImpl.java:307)
        at dev.jbang.util.Util.handleRedirects(Util.java:1094)
        at dev.jbang.util.Util.access$400(Util.java:66)
        at dev.jbang.util.Util$ResultHandler.lambda$redirects$0(Util.java:897)
        at dev.jbang.util.Util.connect(Util.java:819)
        at dev.jbang.util.Util.downloadFileAndCache(Util.java:748)
        at dev.jbang.util.Util.downloadAndCacheFile(Util.java:699)
        at dev.jbang.source.resolvers.RemoteResourceResolver.fetchFromURL(RemoteResourceResolver.java:116)
        ... 41 more
[jbang] [0:259] If you believe this a bug in jbang, open an issue at https://github.com/jbangdev/jbang/issues

[quintesse]

It's a typo, it should be -t junit@jbangdev

[quintesse]

Btw @maxandersen it would be nice if the AppStore would also show templates, right?

[vorburger]

FTR https://github.com/junit-team/jbang-catalog has a README where this is (now) nicely documented.