Threading on ticker mode

[zsmoore]

Hello all,

We have been using ticker mode for awhile now and are moving away from it due to a troubling amount of threads spawned / cpu / gc overload.

We currently have 47 DLs in production using a dispatch all basic predicate and looking at the code

java-dataloader/src/main/java/org/dataloader/registries/ScheduledDataLoaderRegistry.java

Line 237 in 918203c

if (!closed) {

See that dispatches are scheduled on a per DL level rather than a dispatch all, if i am doing calculations correctly then I believe a tick time of 5 ms , with 47 DLs, and a call time of 400ms == 3760 threads per call.

There is room for tuning here but I was wondering why we need a per dataloader dispatch vs, a single callback which runs a dispatchAll on a single thread for the reschedule. This seems like it would significantly decrease thread overhead for the ticker mode.

Beyond this we are moving towards a more deterministic, level by level deterministic dispatch which i have been thinking about how to contribute back to OS but haven't found a great way to generalize it yet.

Wondering about thoughts specifically on if a per DL reschedule is necessary vs a dispatch all at once when the default dispatch all predicate is used.

Thanks!

[andimarek]

Hi @zsmoore,

Can you help clarify a few things?

What is ticker mode exactly?

In GraphQL Java we always had a per level mode by default, do you mean that when you talk about more predictable mode?

We just recently added a new mode in GraphQL Java which is called exhausted strategy, which works similar to the JS dispatching mode.

Best

[zsmoore]

Sure thing, we are using dataloaders without graphql in our app to benefit from the caching per request + batching decentralization.

Ticker mode I am referring to is with the scheduledDLRegistry here -
https://github.com/graphql-java/java-dataloader?tab=readme-ov-file#scheduleddataloaderregistry-ticker-mode

So unfortunately we are not able to benefit from the GQL general functionality as we are only leveraging the underlying library.

[andimarek]

Ah ... thanks for clarifying.

What kind of level you are referring to then if you don't mean the "GraphQL Java level"?

Do you mean chained data loader "levels"?

In general we are open to contributions in this direction as we don't see necessarily data loader bound to GraphQL Java.

As I said we also just solved the chained data loaders problem in GraphQL Java and I was thinking about contributing back to a data loader a more general "strategy" abstraction.

See #243

[zsmoore]

looks like your dispatch strategy is pretty much exactly what we need.

When i say level by level i mean in the context of thinking like the dataloaders as a tree and the root call being the first level with subsequent loads becoming subsequent levels. In theory we are moving towards a deterministic BFS dispatch of that tree in order to optimize batching across the entire request.

The current alternative which im working on releasing is a hybrid between a deterministic dispatch which tracks loads queued and loads completed as well as a work tracker which says if no work found but work not completed, use a fallback thread to reschedule a dispatch

i.e.

Async Path

• A
  • B (async)
    • E
    • F
  • C
    • G (async)
    • H
  • D
    • I
    • J

A dispatches immediately, queues B, C, D
B, C, D dispatch queue G, H, I, J
Thread is spawned to wait for B to finish and retry every 30 ms
B finishes, queues E, F
E, F, G, H, I, J dispatched

With each of these the callback is a .dispatchAll on the registry rather than a per-dl reschedule

[andimarek]

Without know the details of you system I wondering if you actually need any kind of time windows (I mean maybe you do and then we are still happy to accept strategies/solution for that of course)

The main "idea" of the PR #243 is to inform the strategy every time .load is called. This was the key aspect to solve it in GraphQL Java together with the notion of "being busy". In GraphQL Java the new exhausted strategy always dispatches once the engine is not busy anymore. This works really well in practice.

So if you have any kind of notion or concept during a request that will indicate "request is actively executed" or "nothing actively todo right now" you could leverage the same kind of strategy as we do now in GraphQL Java (once #243 or similar PR would be merged): you just inform the strategy about "being busy" or "finished being busy".

For example in GraphQL Java we use https://github.com/graphql-java/graphql-java/blob/master/src/main/java/graphql/execution/ExecutionStrategy.java#L374 to tell the strategy the engine is busy and then two lines later not busy anymore.

The huge advantage of this is that you don't have any Threads scheduled and you never have any scaling issues caused by this: The strategy simply tracks the "being busy" + how many DataLoader.load are outstanding to be dispatched and dispatches them immediately once "not being busy" anymore.

[zsmoore]

I will definitely look into this tomorrow, thank you for the thoughtful response and i'll let you know how it goes! Once i was going over this again i realize i might not even need the fallback scheduler with the futures and might be able to just take the idea of being busy as you mentioned as a signal that the request is not finished

Outside of this im wondering what the long term plan is for ticker mode in scheduled data registry. I think it makes sense for small projects but most likely not for large scaling and wondering how you all think about the experimental api guidelines for the project

[andimarek]

We will probably have to tighten a bit the guidelines around ticker mode, I agree. The thing is that so far we don't have a better data loader native solution for chained data loaders, so if you are in a pure data loader world, next tick is probably your best bet.

cc @bbakerman for his thoughts on this matter, as he is the main author of dataloader.