Was dynamic scoping of `$0` a mistake?

[memreflect]

What is the output of the following code?
(Hint: it's not the shell invocation or the name of the script as one might expect.)

`{echo echo $0}

This behavior is documented in the man page:

If, as the most common case, a function variable is bound to a lambda, when the function is invoked, the variable $0 is bound (dynamically, see below) to the name of the function.

It sounds reasonably intuitive, but it's awful in practice, and i'm not the first to be confused by it:

I tried to get the name a script was called by hence I did:

NAME=`{basename $0}

giving me a value of '%backquote' for '$NAME'.

Similarly, the following excerpt from examples/esrc.haahr once worked because fn-%backquote = $&backquote was used, meaning $0 was not set, but %backquote became a normal lambda, leading to the same trouble:

fn -- {
  eval `` '' { $0^p $* }
#want: `` '' { --p $* }
#have: `` '' { %backquotep $* }
}

If one defines fn-if = @ { $&if $* }, then many other examples are broken as well:

# examples/number.es
if { ~ $* *[~0-9,.]* } \
     { throw error $0 $0: invalid character in argument. }
#want: throw error number number: invalid character in argument.
#have: throw error if if: invalid character in argument. }

# examples/friedman/lib/subr.es
if { ~ $* () '' } \
     { * = `{ local (fn-$0 =) $0 } }
#want: * = `{ local (fn-whoami =) whoami }
       # undefine fn-whoami, so whoami(1) is invoked
#have: * = `{ local (fn-%backquote =) whoami }
       # invokes fn-whoami repeatedly, not whoami(1)

The general expectation appears to be that $0 is not dynamically scoped.
Can anybody think of a good reason to retain this confusing behavior?

[jpco]

Here's another example:

; echo $0 && true
es
; fn-%and = $fn-%and(2)  # removing the $&noreturn
; echo $0 && true
%and

In general, I would quibble that I'm not convinced the problem is dynamic scoping exactly, but I'm definitely on board that the current setup is really funky and hard to use.

Is there a clear better behavior? I'd love some kind of proof-of-concept. My only real opinion is that I'd like to maintain the behavior introduced in #194 ideally.

[memreflect]

Is there a clear better behavior? I'd love some kind of proof-of-concept. My only real opinion is that I'd like to maintain the behavior introduced in #194 ideally.

It should be a straightforward change that doesn't disturb the changes made in that PR.
The varpush()/varpop() logic in the case nLambda: of eval() would be replaced with

context = mkbinding("0",
                    mklist(mkstr(funcname), NULL),
                    context);

if the function is named, which would bind $0 lexically.

The current shell behavior only defines $0 in the case of named lambdas like fn-f = @ *{echo $0}, not named thunks such as fn-g = {echo $0}.
Since x = {echo $0} requires execution using $x, which means $0 is not assigned, the $0 behavior of lambdas bound to fn-name could be extended to thunks bound to fn-name.
I don't know if this is a good idea, but i have yet to find any problems with doing so.
In short, fn-true = {result 0} would bind true to $0.

Proof-of-concept changes

[jpco]

Oh, wow, this is great. I was somewhat expecting using lexical binding to cause a ton of messy %closure (0 = ...) {}s but because this binds $0 so late, that doesn't happen. You can end up with those, with something like fn-x = {echo {$0}}; x, but writing code like that you're just asking for that kind of noise.

This all makes $0 predictable. I am a convert to your lexical way of thinking on this -- why was I doubting the power of lexical binding? It Just Works :)

; fn bingbong {echo $0; $*}
; bingbong {echo $0}
bingbong
es

I think that's exactly right.

The only thing I can find that's still "wrong" is behavior around, for example, $&noreturn. The following is not very intuitive at all:

; fn-x = @ {echo $0}
; fn-y = $&noreturn @ {echo $0}
; x
x
; y
es

As another example, I sometimes like to define a function as a call to map (the classic higher-order function) like fn-do-all = map @ {commands for each...}. Much like the above $&noreturn example, I would prefer that $0 inside the lambda is do-all rather than es.

A mildly cynical thought is that lexically binding $0 in functions might defeat the point -- why would I care to look up what $0 is when I know exactly what it's going to be, since I'm in the middle of defining the function? But I guess it's not actually entirely predictable, as in cases like this, so there's still value to it.

; printem = {echo $0}
; fn-x = $printem
; fn-y = $printem
; x
x
; y
y

[memreflect]

Oh, wow, this is great. I was somewhat expecting using lexical binding to cause a ton of messy %closure (0 = ...) {}s but because this binds $0 so late, that doesn't happen. You can end up with those, with something like fn-x = {echo {$0}}; x, but writing code like that you're just asking for that kind of noise.

That's why the binding needs to happen when the lambda is being evaluated rather than when it is being parsed.
Otherwise, you would indeed wind up with a bunch of %closure(0=...), which could become problematic because execve() can limit the collective size of the argument list and environment variable list when executing external processes.

The only thing I can find that's still "wrong" is behavior around, for example, $&noreturn.

I agree this is less than ideal, and i have been annoyed by this additional side effect of $&noreturn in the past.
It's easy to make a special exception for $&noreturn by adding a -n name option to $&noreturn whose option-argument serves as the definition for $0 and prepending the -n name option to the argument list of $&noreturn if necessary:

; fn-x = $&noreturn @ {echo $0}
; x
x
; fn-y = $&noreturn -n z @ {echo $0}
; y
z

See memreflect/es-shell@4b9393e for the changes.

(Given the discussion in #219, i'm hesitant to use the binding list of $&noreturn to pass $0, but that is also a possible remedy.)

As another example, I sometimes like to define a function as a call to map (the classic higher-order function) like fn-do-all = map @ {commands for each...}. Much like the above $&noreturn example, I would prefer that $0 inside the lambda is do-all rather than es.

I'm not sure about applying $0 binding to lists in general as in the case of fn-do-all = map @ {commands for each...}.
If %closure(0=do-all;...) should be a part of any anonymous lambda passed to map, you'd want something like fn-do-all = $&noreturn @ * { map @ {commands for each...} $* }, which is reminiscent of function currying in a lambda calculus.
Or maybe es needs a new primitive to set a custom invocation name rather than hacking $&noreturn as i did?

A mildly cynical thought is that lexically binding $0 in functions might defeat the point -- why would I care to look up what $0 is when I know exactly what it's going to be, since I'm in the middle of defining the function? But I guess it's not actually entirely predictable, as in cases like this, so there's still value to it.

; printem = {echo $0}
; fn-x = $printem
; fn-y = $printem
; x
x
; y
y

One could extend mklambda() with a name parameter to bind a hard-coded %closure(0=name) at parse-time, but i find the lexical binding at eval-time to be more useful for exactly the reason you specify.

[jpco]

I'm not sure about applying $0 binding to lists in general as in the case of fn-do-all = map @ {commands for each...}.

I'm realizing that, intuitively, what I want is something like the following: Any time you look up a function that looks like

fn-X = Y

then what you get is as if that was really

let (0=X)
fn-X = Y

I think that's the most straightforward and easiest to understand version of $0. As an example, if you have

; fn-X = $&noreturn @ {echo hi}

then when

; X

is entered and the shell looks up fn-X for evaluation, then what it ends up actually evaluating looks like:

; $&noreturn %closure(0=X)@ *{echo hi}

(Though this isn't technically valid syntax, it is valid in memory; see #229.) Because the 0=X binding would be attached to the lambda term, the $&noreturn wouldn't need to refer to its own lexical context at all, so we wouldn't need to worry about #219.

Does this make sense? I'm thinking of it from the point of view of a user, so I'm not really considering the implementation, though I guess what I'm proposing does involve changing the moment of $0 assignment to function-lookup time, rather than lambda-evaluation time.

[jpco]

I took a crack at implementing the above: https://github.com/jpco/es-shell/tree/lexzero

It's a little odd, but the core idea is at var.c:179-183. When we look up the definition of a function with fnlookup, we scan through the definition and add the lexical binding "0" = name to ~any term that can be closed over.

This does what I expect with every example given so far.

; `{echo echo $0}
./es
; fn --p {echo echo --p called with $*}; fn -- { eval `` '' { $0^p $* } }
; -- hello world
--p called with hello world
; fn-if = @ {$&if $*}
; if {~ $0 ./es} {echo got it!}
got it!
; fn-if = $&noreturn @ {$&if $*}
; if {~ $0 ./es} {echo got it with $&noreturn!}
got it with $&noreturn!
; echo $0 && true
./es
; fn-%and = $fn-%and(2)
; echo $0 && true
./es
; fn-do-all = map @ {echo $0 $*}
; do-all one two three
do-all one
do-all two
do-all three

[memreflect]

I took a crack at implementing the above: https://github.com/jpco/es-shell/tree/lexzero

It's a little odd, but the core idea is at var.c:179-183. When we look up the definition of a function with fnlookup, we scan through the definition and add the lexical binding "0" = name to ~any term such a closure can exist.

This does what I expect with every example given so far.

That's much cleaner than my attempts.
If anything, it's my implementation that is odd, not yours.

If there's an example where your code doesn't work, i have yet to find it.
Here is what i used to test your implementation:

; fn-x = (
    $&noreturn @ {
      let (cmd = {echo $0 $*}) {
        echo $0
        echo $cmd
        result $cmd
      }
    }

    # "curried" args
    hello
  )
; fn-y = <={x world}
x
%closure(0=x;*=hello world){echo $0 $*}
; x
x
%closure(0=x;*=hello){echo $0 $*}
; y
y hello world

It works with $&noreturn lambdas, thunks, lists acting like curried functions, etc.
I'd say it's pretty much done aside from editing the man page and adding some tests.