Make PatternToken public

[skeggse]

glob provides nice parsing and matching facilities, and sometimes it's important to reach into the Pattern and inspect the individual PatternToken elements themselves.

Use case

Given some fn extend_with_literals(input: &str, pattern: Pattern), identify whether the input is a valid prefix of pattern and determine any subsequent literal characters from the pattern that would better qualify the input.

Example

assert_eq!(
    extend_with_literals("pr", Pattern::new("pr[e]?ix").unwrap()),
    "pre"
);

assert_eq!(
    extend_with_literals("pre", Pattern::new("pr[e]?ix").unwrap()),
    "pre"
);

assert_eq!(
    extend_with_literals("pref", Pattern::new("pr[e]?ix").unwrap()),
    "prefix"
);

This makes it possible to optimize glob in over a nonstandard async filesystem where enumeration is very expensive and irrelevant matches are common.

It's easier to implement this by having access to the PatternToken and not needing to reimplement AnyWithin/AnyExcept parsing.