Loading `Verdict::Experiment` derived classes in `/app/classes` won't work

[philschoefer]

This is to document an issue we ran in at Burst.

If you derive your experiment from the Verdict::Experiment class and put it into the /app/experiments folder you will run into an issue where you'll end up with a unitialized constant error.

The reason for this is that Verdict removes the /app/experiments folder from config.eager_load_paths in order to unify the way classes are required within the gem which needs to be compatible with non-rails applications.

Code:
https://github.com/Shopify/verdict/blob/master/lib/verdict/railtie.rb#L5-L6

This means while local autoloading on your dev environment might work perfectly, on staging or production where eager loading is active you'll run into an error where your custom experiment class won't be found.

There are several ways to work around that. The easiest one that https://github.com/Shopify/partners and https://github.com/Shopify/stock-photos chose to go with is to move your Verdict::Experiment derived class into the /app/models folder where it will be autoloaded and eager loaded correctly.

Another way would be to rename /app/experiments to something else. All folders within /app are usually automatically picked up on by rails.

Also note that while putting class definitions in /app/experiments will fail, you can still put your MyExperimentClass.define calls in there. The code will still be executed. It just won't be eager loaded.

Hope this helps someone in the future.

Is this solvable?

From a rails user perspective it seems like the removal of /app/experiments from eager_load_paths could just be removed. At least we didn't see a downside to that in our rails app. However that could cause complications in non-rails apps.