Equirectangular image and IMU extrinsics calibration

[fushi219]

Hi, Steffen, i have a question and wish to get your advice.

Now i have got a gopro-max camera whose output 360 video is equirectangular(from mp4 file) or EAC(extracted from ffmpeg) and i have no ways to extract original dual fisheye images and calibrate lenses with common camera models.

So the only practicable way to run VIO on gopro-max is to develop a equirectangular VIO system based on other open source code which i have almost done.

However, there exists a problem calibrating extrinsics between the equirectangular 'lens' and imu as both your code, kalibr or so on do not support such projection models. I got two ideas solving the problem. One is to write code to support such model. And the other is to project equirectangular into cubemaps and use front cube image to estimate the extrinsics. I prefer the second as it could save much time. However, i wonder if this method make sense as will equirectangular 'lens' and front cubemap 'lens' share the same orientation and location?

Look forward to your advice and thanks!!

[urbste]

Hey, cool. It should not be too complicated to implement that camera model here. Actually I would also like to have it in pyTheiaSfM as a camera model. Do you have a calibration video sequence? If you want I could have a look at it as well. I just do not have a 360 camera myself atm. Or do you know of some dataset?

[urbste]

Usually to implement a new camera model the way to go is:

1. Implement the model in pyTheiaSfM here
2. Also write some tests to check if it works
3. Then add the camera model here and here to the if-clause.
4. Adapt io functions: here

[urbste]

I just figured, that you would also need to think about the intrinsic initializer. At the moment I am using DLT for pinhole and a p4pfr solver for other radial distortion cameras. For the equirectangular you would need to implement a solver that works on bearing vectors. MLPnP would be a solution, but it is currently not implemented in pyTheiaSfM.

[fushi219]

Usually the way to go is:

1. Implement the model in pyTheiaSfM here
2. Also write some tests to check if it works
3. Then add the camera model here and here to the if-clause.

You should be good to go.

Cool!! I would like to do the relative work above，then make a PR for your beautiful job. However, as i need to spend much time doing my formal job, it would takes sometime for the PR.

[fushi219]

I just figured, that you would also need to think about the intrinsic initializer. At the moment I am using DLT for pinhole and a p4pfr solver for other radial distortion cameras. For the equirectangular you would need to implement a solver that works on bearing vectors. MLPnP would be a solution, but it is currently not implemented in pyTheiaSfM.

Yes, the relative bearing vector triangulation and pose solver is also needed.

[urbste]

Yeah no worries. Would be happy to help, as soon as you have a branch and a dataset just ping me ;)

[fushi219]

Yeah no worries. Would be happy to help, as soon as you have a branch and a dataset just ping me ;)

It's my pleasure.👏