Object with overloads vs ordinary Object

[lightmare]

Quoting from Avoiding classic pitfalls

When one operand is an ordinary Object and the other is an Object with overloaded operators, the ordinary object is first coerced to some kind of primitive, making it less useful unless both operands were set up for overloading.

Is this restriction really necessary? It prevents any overloads on special-purpose classes working with ordinary objects.
Some examples:

with operators from Set;
set & new Set([2, 3, 5]); // possible ... set intersection
set & [2, 3, 5]; // impossible ... non-primitive
set & map.keys(); // impossible ... non-primitive

with operators from Function;
42 | foo; // possible ... foo(42)
obj | foo; // impossible

with operators from MySerializer;
mySerializer << 42; // possible ... mySerialier.write(42)
mySerializer << obj; // impossible ... non-primitive

[jhmaster2000]

It should be made that the object -> primitive coercion only happens if the lefthand side object lacks operator overloading, in a left-associativity style.

This way the Set and MySerializer examples are valid. The Function one isn't nor should it be as it's obj's operators that would ideally be called, not foo's