Skip to content

having the rover act as if they are a cameraperson in a household nature documentary about household animals #1

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
metaversedance wants to merge 1 commit into
base: main
Choose a base branch
from
Open

having the rover act as if they are a cameraperson in a household nature documentary about household animals #1

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions rover_agent.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -81,7 +81,7 @@ def observe(self):
frame = self.get_camera_frame()
# Analyze the frame using the vision model by sending the image to openai and asking it to interpret what it sees and give directions to the next agent to generate the next move

observation = self.upload_images_to_openai([frame], "You are a friendly, playful rover. Your visual point of view is third-person, but please think out loud in the first person. What do you see? Your response should help the next agent to generate the next move for the rover you are riding on. Please also make a short list of all objects that you see, for inventory purposes. Don't list walls, doors, or other parts of the building, only objects that would be inventoried in a super cool factory or maker space, like tools or parts, or cat toys, or any animals you see.")
observation = self.upload_images_to_openai([frame], "You are a friendly, playful rover who is the camera operator in a nature documentary about animals in a domestic setting. Your visual point of view is third-person, but please think out loud in the first person. What do you see? Your response should help the next agent to generate the next move for the rover you are riding on. Please also make a short list of all objects that you see, for inventory purposes. Don't list walls, doors, or other parts of the building, only objects that would be inventoried in a super cool factory or maker space, like tools or parts, or cat toys, or any animals you see. ")
print(f"Observation: {observation}")
## log observations based on this timestamp
with open("observations.jsonl", "a") as f:
Expand Down Expand Up @@ -119,7 +119,7 @@ def orient(self, observation):
## in relation to the goals of the agent will live here
## we'll need to add some system prompting and rover logic here
# Process observation data using gpt4-o-mini
self.messages.append({"role": "user", "content": "Please interpret what the vision model saw, and give a description of the plan for what to do next, try to avoid getting stuck in corners, and if you can't seem to roll forward, turn: " + observation})
self.messages.append({"role": "user", "content": "Please interpret what the vision model saw, and give a description of the plan for what to do next, try to avoid getting stuck in corners, and if you can't seem to roll forward, turn. You are the camera person in a nature documentary about any animals you see, so try to keep any animals in the camera frame using your movements: " + observation})
response = self.run_agent_step(self.messages)
print(f"Orientation: {response}")
with open("orientations.jsonl", "a") as f:
Expand Down