Running kitchen env with oracle, low success rate

[shaoyifei96]

Thanks for putting this out here!

I tried to run the kitchen environment. After dealing with some version issues. For those also trying to run: Here are the required changes and dependencies: mujoco 2.3.1.post1, gymnasium-robotics using the fork but actually the dependency gymnasium 0.28.1, in settings.py num_train_tasks = 1, bilevel_plan_without_sim = True the kitchen environment does work!

I have a few questions:

1. what is the typical success rate for other methods
2. Are you guys planning to support more mujoco environments like robocasa?
3. It seems to achieve low success rate, even with oracle Options and Predicates. Is that expected? (I tried running 10 test with "kettle_only" kitchen_randomize_init_state = True, and got 1 success out of 10)

[NishanthJKumar]

Hi @shaoyifei96 - thanks for the questions. We're currently quite busy pushing for a research deadline, so will be able to provide a more thorough response after a week or so, but in brief:

1. In kitchen oracle should be getting 100%: probably there is something finnicky with the goal predicates or something given the mujoco version(s) - we've had some numerical issues with that environment in the past.
2. Currently there's no concrete plan to do that, but we'd be happy to review a PR or something if you'd like to try to implement it! We can potentially discuss the best ways to do something like that
3. No this is unexpected - like i said above, the success rate for oracle should be at or near 100%