-
Notifications
You must be signed in to change notification settings - Fork 0
Description
The codesniffer is unique in that it's not really a binary "yes/no" detector. In fact, it works in a strange kind of 3-way state in most cases:
"This must be X" vs "This must not be X" vs "This must be x^" vs "This must not be x^" vs "I don't care really."
Or using an example, you can write a sniff that dictates the opening brace must not appear "on the same line," but that doesn't necessarily preclude it from being two lines down:
if (...) { // fail
//...
}
//vs
if (...) // pass
{
//...
}
//vs
if (...) // pass(?)
{
//...
}Writing a rule in a "blacklist" fashion like this allows you to prevent a single bad behavior while not specifying a single preferred behavior. The alternative is a whitelist style rule that says, "this one way is accepted, all others are errors". Depending on what you want to allow and how flexible you want your standard to be, one or the other (or both) will be most appropriate in each case, but this makes testing hard.
In fact this typically means that good test cases have two sides to them: a file (or files) that fail a specific sniff as example(s) of breaking a rule, and other file(s) that demonstrates "proper" or "all acceptable" behavior that passes all sniffs.
To aid in understanding and reinforcing this dynamic, I'm suggesting grouping the sample files we test the ruleset against into folders that make their purpose a little more explicit: snifftests/files/must/ and snifftests/files/must_not/. There's more information in the README about the intended purpose of this, but the basic idea is that whitelist tests for rules that enforce a single specific style go in must/ and blacklist tests that ban a single specific behavior go in must_not/.
In fact, the README could probably stand to be updated to include a little more of this "theory" explanation too.