[Discussion] Port End to End tests away from bash

[ridv]

Working on trying to fix the end to end test has reminded me how painful the process has become for debugging end to end tests when something goes wrong.

The current end to end test stand at 1000+ lines of bash. Running individual tests involves modifying the script and adding new tests is a very involved process.

Ideally we should rewrite these tests in a language that's more apt for the job and easier to contribute to.

I'll throw out some languages to get the discussion started in order of preference:

• Python
• Golang
• Node
• Ruby

Theoretically, we could use thrift bindings directly instead of the aurora client, which would decrease coverage for the client itself, but the client already has its own suite of tests.

I realize porting this code might be a big undertaking but it needs to be done sooner than later in my opinion.

Looking forward to hearing everyone's opinions.

[StephanErb]

Adopting python and pytest here would probably be the easiest option. This would also integrate easily with pants if needed.

[mauri]

A quick glance for the 1000+ lines of bash looks like a lot of code is used to setup/tear down environments by moving files around or starting services. I don't feel it's going to be of great help moving out to a different language, dealing with processes are always easier in bash for sure (with help of BATS you can make a very neat test suite). Maybe restructuring those files can make the task lest daunting. Moving to just rely on python bindings instead of the client is a different story of course

[ridv]

Great points on both counts. Had not heard of Bats so that might be something worth looking into.

@mauri I think for me the main pain points of the current way it is done is when we hit functions like this https://github.com/apache/aurora/blob/master/src/test/sh/org/apache/aurora/e2e/test_end_to_end.sh#L689 which has 8 parameters 😮 This won't change if written in another language but bash syntax is hard to parse in my opinion.

I do think you bring up a good point that a good portion of the script deals with operational stuff.

Maybe we can abstract away the operational stuff to smaller, more pointed bash scripts (i.e. restart and verify that the scheduler came back up), and leave the more intricate stuff (i.e. verifying that a payload is correct) to a python script running the show.

So maybe a combination of the two would be a good idea?

[mauri]

yeah I think that sounds like the most sensible approach :)

[ridv]

Thank you both for your comments 👍
I've started working on a POC for this here https://gist.github.com/ridv/0fd6ca4db1ec411a2879da7fe1ea7884

It's in early stages but I already feel like it's going to be a big win in the end.

I'm writing it in Python3.6+ since f-strings make the most sense for generating some of the commands we need right now.

Few notes:

• In order to move ahead and start working on it, I'm calling the aurora client using the subprocess library.
• When we move the Aurora client to python3 support, we should be able to tap into the CLI object directly using something like what is done here here
• I'm not a pythonista so I'm gonna need help making this more idiomatic once it's ready for review.
• Return statements will be changed to raise errors with the correct errors instead once I'm done.

It's not an elegant solution by any means, but once all the pieces fall into place, we should be able to refactor it into something much more user friendly.

As soon as I'm ready to raise a PR I will do so.