Hidden modules frustrating from end-user perspective

[AaronFriel]

There are a few minor issues with the module layout which have made it somewhat frustrating for me to develop using test-framework, one of which is that the MutuallyExcluded constructor is not exported from Test.Framework.Core. I am simply, it seems, not able to create my own sets of mutually exclusive tests, instead, I must mark everything in an entire tree mutually exclusive?

It seems this should be exported:

data MutuallyExcluded t = ME (MVar ()) t
    deriving Typeable

As well, let's suppose I would like to run tests in conjunction with other code. Perhaps I need to ensure a database, a script, or something else is initialized. That is, there is some resource my tests depend on and without it, the tests are pointless. I would greatly prefer to be able to use something very much like the defaultMainWithOpts, except it does this:

defaultMainWithOpts :: [Test] -> RunnerOptions -> IO ()
defaultMainWithOpts tests ropts = do
    let ropts' = completeRunnerOptions ropts

    when (unK$ ropt_list_only ropts') $ do 
      putStr $ listTests tests
      exitSuccess

    ...

    exitWith $ if ts_no_failures test_statistics'
               then ExitSuccess
               else ExitFailure 1

The exitWith guarantees I can run nothing after defaultMainWithOpts. I cannot implement defaultMainWithOpts myself because it and its dependencies in Test.Framework.Runners.Console references the following things that are hidden:

• completeRunnerOptions - not exported and depends on:
  • processorCount in Test.Framework.Processors - though I could perform the short-circuit myself of using numCapabilities
• listTests - not exported and depends on:
  • runTests in Test.Framework.Runners.Core not exported and depends on:
    • type RunningTest, data RunTest (..), and data SomeImproving (..) in Test.Framework.Runners.Core - cannot even reimplement because these are data types
    • testPatternMatches in Test.Framework.Runners.TestPattern - not exported and has many dependencies
    • executeOnPool in Test.Framework.Runners.ThreadPool - not exported and has many dependencies
• showRunTestsTop - not exported and depends on:
  • Types RunningTest, FinishedTest again
  • hideCursorDuring in Test.Framework.Runners.Console.Utilities - not exported
  • initialTestStatistics and totalRunTestsList in Test.Framework.Runners.Statistics
• gatherStatistics and ts_no_failures - not exported from Test.Framework.Runners.Statistics
• XML.produceReport - entire module not exported
• And so on and so forth.

In short, there is no simple or feasible way for the test-framework to be used by an end-user to create their own runner as a part of a program. I am, essentially, stuck with the kludge of using buildTestBracketed.

Now, I would like to propose a couple changes and I would like your feedback on them:

1. defaultMainWithOpts should be composed of several components, each of which ought to be exported.
2. The exported components should probably be something like:
   • A function for parsing command line arguments, possibly with a prefix added.
   • A function for running the tests, which would take a result consumer and return a data type describing the tests and their results.
   • A function for consuming the result stream of the tests and displaying live output to the console. This would be just one possible consumer - alternative consumers could render a web page and display the results by posting them as they are completed via Web Sockets or AJAX or what-not. As well, one possible consumer could be a database component to store results over time.
   • A function for post-rendering of the tests, in conjunction with the runner. The XML output renderer should be just one possible renderer. (A JSON renderer for example, should be possible to implement by end users.) The rendering output should be an appropriate data type, not just a ByteString, to allow interoperability with other APIs. The rendering API should expose a method of displaying it to the console - for example, to allow pretty-printing the JSON or XML as opposed to normalized / compact JSON or XML.
   • A function for determining if the tests have overall succeeded or failed according to the parameters specified.
3. MutuallyExcluded should be examined as a data type to determine if it's fit to be exported. Is there a reason why it is not?

I am willing to do some of the work above, but the test-framework internals are complex to the uninitiated and it would take me some time to understand all of the working parts.