How to get parametrized readers/writers in dol

[thorwhalen]

dol helps you make facades for the storage concern. It limits the methods to the base builtin collections methods (those of Collection, Mapping and MutableMapping types).

What do you do, though when you need to have more control over your storage operations?

Let's take the example of needing to do chunked reads. That is, you want to be able to say store.get_chunks(key, chk_size) and get an iterator of chunks of chk_size. The Mapping interface doesn't give you that naturally. It just gives you store[this]. So you write your own new method for this (see for example this (rejected) pull request.
The biggest con to that is that it's not a method that dol knows of, so if you use any of dol's tools, stuff will break.

What can you do?

Is it really a need?

Sometimes the need isn't really a need: The "need" comes from constraints of approach you're taking, or have to take.
Many times your best move is to rethink your approach to fit it to the patterns/interfaces that the python gods decided to include as the fundamental methods, or write a adapter to an existing system so it will comply to those fundamental interfaces.

In the case of the store.get_chunks(key, chk_size), you might, for example, realize that really, your chk_size will be fixed (perhaps different, but fixed for every given context). In that case you could just have the values of your store be iteratables, or iterators of chunks:

from typing import Iterator
store = ChunksStore(chk_size=2048)
value = store[key]
assert isinstance(value, Iterator)
chunk = next(value)
assert len(chunk) == 2048

tuple keys

If you really "need" to not fix your chk_size for a give context, to enable a user to ask for any chk_size when they read, but while avoiding writing another method, you define your __getitem__ to be something like this:

def __getitem__(self, k):
    src_key, chk_size = k  # the key is actually a tuple
    # then do what you need to do to get that iterator...

Though it is arguably "better" than writing a new method, it's not ideal. If you were only getting items, it would be fine, but a Mapping has other methods, and you might be misaligned how a user might expect the methods to interact.
For example, that __iter__ should give you a list of all valid keys, and that these keys can all be used to get items.
What are you going to do? List all possible (src_key, chk_size) combinations? Certainly not!

Instead, you could:

def __getitem__(self, k):
    src_key, *chk_size = k  # the key is actually a tuple
    if len(chk_size) == 0:  # then only a src_key:
        # return the full resource
    else:
         # then do what you need to do to get that iterator...

So now your __iter__ gives you all valid src_keys, which you can use to get items.
But still, a purist would sneer at it (and if they do, just mention pandas, which does do such "expanded item getting".

Opinion: There's a missing collections.abc type: A Getter. One that just has a __getitem__, but not necessarily __iter__, __len__ etc.

a cleaner (but not necessarily always better) way

Make the object that __getitem__ returns be something that has a __getitem__ itself, that will give you the chunk iterator.
The usage would be something like:

value = store[src_key]
chunks_iterator = value[2048]
assert len(next(chunks_iterator]) == 2048