Pyffi overhaul

[Candoran2]

@niftools/blender-niftools-addon-reviewer

Overview

1. Changes kf and nif import/export to make use of the new nif statically generated nif library.
2. Added support for SSE meshes (import only).

Detailed Description

1. Since most functions were ported straight from pyffi, functionally little code needed to be changed. The classes are still easily accessible, but under generated.formats.nif in the classes variable (it's actually a dict that also allows value access through dict.key in addition to dict['key']).
2. SSE nifs use BSTriShapes, so the import process had to be partially adjusted. Getting the geometry information from the nif (vertex positions, triangles etc) was decoupled from setting it. BSTriShapes were given the skin_partition property as alternative access to the skin field because it pretty much refers to the same thing as the skin_partition field on NiGeometry. Ideally this would be adjusted in the xml so it isn't necessary, but that would probably be a breaking change for other software that use the xml.

Fixes Known Issues

• Doesn't completely fix, but is a first step towards Please add support for Fallout 4/76 Nif Files #473, Is there any plan to support SE mesh, textures? #474, kf root "NiSequenceData" #480, ValueError: Nif version 30.2.0.3 not supported. #534 and I'm sure a number of others.
• Importing of triangles stored only in skin partitions and not in NiTriShapeData #521

Documentation

[Overview of updates to documentation]

Testing

[Overview of testing required to ensure functionality is correctly implemented]

Manual

[Set of steps to manually verify updates are working correctly]

Automated

[List of tests run, updated or added to avoid future regressions]

Additional Information

[Anything else you deem relevant]

[HENDRIX-ZT2]

Could we change the import to put the .classes part into the import statements instead of each NifFormat call? Not sure if that works though

[Candoran2]

Could we change the import to put the .classes part into the import statements instead of each NifFormat call? Not sure if that works though

Yes. I just did it.

[HENDRIX-ZT2]

A few remarks, one important todo. Stellar job so far!

[Allofich]

Hello. I've been keeping an eye on this project as I'm interested in it. May I ask if there is an update on this PR?

I was motivated to comment because, when looking at open PRs at the PyFFI repository, I saw niftools/pyffi#74, which looks, at least, like a similar situation to this PR. It has one reviewer giving it an OK, with another requested to review, and seems to have remained in that state for 2 years. I feel it would be a shame for this PR to end up in a similar situation. The contributor might leave when they could have instead been motivated to continue sending in additional improvements if the PR was accepted.

So, this is just to let you know that those like myself who are interested in the project, and either a merge of PRs like this or the one I linked to, or a comment about why merging is not yet possible, etc. would be appreciated and keep interest going. Thanks.

Edit: Looks like the contributor is a member of Niftools, so what I said about them possibly leaving may not be applicable.

[Candoran2]

Hello. I've been keeping an eye on this project as I'm interested in it. May I ask if there is an update on this PR?

I was motivated to comment because, when looking at open PRs at the PyFFI repository, I saw niftools/pyffi#74, which looks, at least, like a similar situation to this PR. It has one reviewer giving it an OK, with another requested to review, and seems to have remained in that state for 2 years. I feel it would be a shame for this PR to end up in a similar situation. The contributor might leave when they could have instead been motivated to continue sending in additional improvements if the PR was accepted.

So, this is just to let you know that those like myself who are interested in the project, and either a merge of PRs like this or the one I linked to, or a comment about why merging is not yet possible, etc. would be appreciated and keep interest going. Thanks.

Edit: Looks like the contributor is a member of Niftools, so what I said about them possibly leaving may not be applicable.

Seems like Neomonkeus (who normally does the merging/releases) isn't available at the moment. My plan so far is to wait until the 19th and merge it myself if there hasn't been any update by then.

In the meantime I'm still working on the addon, I'm just not yet pushing the commits to GitHub.

[Allofich]

Thanks for the update. I didn't know you were able to merge it yourself (I didn't realize you were part of the Niftools team either until I looked at your profile after making my comment), and it's good to hear that work is continuing. It'll be great if we can start importing and exporting .nif files with current versions of Blender, which doesn't seem to work yet (I've only tested with Oblivion .nif files, so I don't know if other games work).

[Candoran2]

Thanks for the update. I didn't know you were able to merge it yourself (I didn't realize you were part of the Niftools team either until I looked at your profile after making my comment), and it's good to hear that work is continuing. It'll be great if we can start importing and exporting .nif files with current versions of Blender, which doesn't seem to work yet (I've only tested with Oblivion .nif files, so I don't know if other games work).

Importing and exporting nif files with current versions of Blender should already be possible with the released versions of the plugin (especially Oblivion nifs, which have long been supported) - if you're encountering errors while doing that, please make an issue and post the concerning nifs so that we can help.

[Allofich]

The problems I encountered are covered by #453. Trying to import meshes/architecture/anvil/anvilaltar01.nif failed because of "'int' object has no attribute 'material'" errors. A fix is mentioned in that issue report, and it did allow the import to proceed, but I don't know if it's a correct fix.

The exported .nif then supposedly lacks collision in-game. I can't remember if I tested that part myself, but I was left with the impression that the plugin wasn't functional yet, at least for Oblivion .nif files.

[Allofich]

Well I just downloaded the latest Blender release (3.3), installed the latest blender_niftools_addon, and meshes/architecture/anvil/anvilaltar01.nif imports without an error now. I thought I had already tested the latest blender_niftools_addon and found that it still didn't work, and I thought that no changes related to the "material" attribute had been made, so it's a bit of a surprise. #453 may need updating, then. I'm going to keep playing around with it for a while.

Sorry, I remembered the problem incorrectly. The "material" error happens on export, not import, and it still occurs. I tested this with the latest development code (newer than the release code, which fails to export with an element-wise multiplication error that has since been fixed).

[Allofich]

I remembered I had this conversation niftools/pyffi#69 over at the PyFFI project, where I was told the "material" workaround wasn't a good solution (it would "break other things"). I was told you and HENDRIX-ZT2 were working on a proper solution. So that's where I got the impression things were WIP and it would be some time before Oblivion .nifs could be exported successfully.

I don't know how common the "material" issue is with Oblivion .nifs, but testing a few others I quickly came upon another with the same error: meshes/architecture/anvil/anvilfgfirstfloor.nif.

Anyway I didn't mean to get off-topic. I know this PR is not specifically targeting the issue I mentioned, but it looks like it may be the proper solution I was told about, so I'm hopeful it will fix it.

[HENDRIX-ZT2]

I think we can just go ahead and merge this. IMO we don't need to do micro-releases after every PR. It just clogs down development.

[enpinion]

What is the 'generated' module actually? Is it missing directory to be pushed?
Because if I pull the current develop branch and build blender throws generated modules are missing. It looks similar as pyffi but that module has no array::Array.

[Candoran2]

What is the 'generated' module actually? Is it missing directory to be pushed? Because if I pull the current develop branch and build blender throws generated modules are missing. It looks similar as pyffi but that module has no array::Array.

It's a module that is generated using codegen.py from cobra-tools: https://github.com/OpenNaja/cobra-tools using this code: https://github.com/Candoran2/new-pyffi as source and the updated version of nifxml: https://github.com/Candoran2/nifxml/tree/develop .

If you have any questions about it, you're free to ask them here, but the NifMania discord would also work for (likely) somewhat more instant communication.

[enpinion]

Thank you for the reply. Now it works.

[Allofich]

@Candoran2 Could you explain a bit more how to create the needed module?

I've downloaded the latest cobra-tools and followed the instructions to install Python and run "pip install pyqt5 imageio". I've also downloaded the latest sources from https://github.com/Candoran2/new-pyffi and https://github.com/niftools/nifxml/tree/develop (which seems to be newer than https://github.com/Candoran2/nifxml/tree/develop), but I don't know how to proceed and can't find any information on using codegen with pyffi.

[Candoran2]

@Candoran2 Could you explain a bit more how to create the needed module?

I've downloaded the latest cobra-tools and followed the instructions to install Python and run "pip install pyqt5 imageio". I've also downloaded the latest sources from https://github.com/Candoran2/new-pyffi and https://github.com/niftools/nifxml/tree/develop (which seems to be newer than https://github.com/Candoran2/nifxml/tree/develop), but I don't know how to proceed and can't find any information on using codegen with pyffi.

For simple operation: Copy or softlink the formats/nif folder from pyffi into the cobra-tools source/formats folder. Make sure that the nif xml is also in that source/formats/nif folder. Make sure the spells and the utils folder from new-pyffi are similarly in the source folder of cobra-tools.

Then, simply run codegen.py. There's a lot of other code in the cobra-tools repository, but that and the codegen folder are the only part that we need (and some other base format libraries like base and dds that get generated similarly). This will fill the generated folder in cobra-tools with a lot of python code, including that necessary for reading nifs. You can copy or link that in the dependencies folder of the niftools addon, and everything should work.

[Allofich]

Thank you for the instructions. I was able to get the plugin working (with problems, see below) by running codegen.py and then copying the generated folder into the dependencies folder of the niftools addon (and then zipping the addon into a .zip file for installation in Blender).

With only the generated folder in dependencies, the options for importing or exporting .NIF, etc. were not added to the import menu, so I tried adding the dependencies as taken from the latest release (v0.014): the bin, pyffi and PyFFI-2.2.4.dev3.dist-info folders. With these added to the dependencies folder, the import and export options appeared, but are these dependencies from the v0.014 release file suitable for the newer code? I'm trying the latest develop branch code (f5211a9). I haven't tried the newer code from #541.

While the addon is at least working to some degree, I'm seeing errors and actually get somewhat less far in my test case (meshes/architecture/anvil/anvilaltar01.nif from Oblivion) than with the v0.014 release. I don't know if this is just because it is WIP, or if I maybe have an incompatibility in the dependencies.

I'm testing two versions of anvilaltar01.nif. One is the version you get from installing from GOG or Steam. The other is from the DVD release (the DVD release has slightly different mesh files that report a later NIF version).

With the v0.014 addon both versions of anvilaltar01.nif import successfully. They both fail to export due to the problems mentioned in #453.

With the newer addon code from the develop branch I'm trying now, only the GOG/Steam version imports successfully. The DVD version fails with

niftools:INFO:Executing - Niftools : Blender Niftools Addon v0.0.15(running on Blender 3.3.1, Nif xml version 0.9.3.0)
niftools:INFO:Importing ****\anvilaltar01 CD.nif
niftools:INFO:NIF file version: 14000005
niftools:INFO:Reading .nif file
Traceback (most recent call last):
  File "****\AppData\Roaming\Blender Foundation\Blender\3.3\scripts\addons\io_scene_niftools\operators\nif_import_op.py", line 134, in execute
    return NifImport(self, context).execute()
  File "****\AppData\Roaming\Blender Foundation\Blender\3.3\scripts\addons\io_scene_niftools\nif_import.py", line 74, in execute
    self.load_files()  # needs to be first to provide version info.
  File "****\AppData\Roaming\Blender Foundation\Blender\3.3\scripts\addons\io_scene_niftools\nif_import.py", line 142, in load_files
    scene.import_version_info(NifData.data)
  File "****\AppData\Roaming\Blender Foundation\Blender\3.3\scripts\addons\io_scene_niftools\modules\nif_import\scene\__init__.py", line 55, in import_version_info
    scene.game = possible_games[0].name
TypeError: bpy_struct: item.attr = val: enum "UNKNOWN" not found in ('NONE', 'ATLANTICA_ONLINE', 'AURA_KINGDOM', 'AXIS_AND_ALLIES', 'BLOOD_BOWL', 'BULLY_SE', 'CIVILIZATION_IV', 'CULPA_INNATA', 'DARK_AGE_OF_CAMELOT', 'DIGIMON_MASTERS_ONLINE', 'DIVINITY_2', 'DIVINITY_2_0_X_10000', 'EMERGE', 'EMPIRE_EARTH_II', 'EMPIRE_EARTH_III', 'FALLOUT_3', 'FALLOUT_4', 'FALLOUT_76', 'FALLOUT_NV', 'FANTASY_FRONTIER', 'FFT_ONLINE', 'FLORENSIA', 'FREEDOM_FORCE', 'FREEDOM_FORCE_VS_THE_3_RD_REICH', 'GHOST_IN_THE_SHELL_FIRST_ASSAULT', 'HOWLING_SWORD', 'IRIS_ONLINE', 'KOHAN_2', 'KRAZY_RAIN', 'LAZESKA', 'LEGO_UNIVERSE', 'LOKI', 'MAPLE_STORY_2', 'MICRO_VOLTS', 'MORROWIND', 'MUNCH_S_ODDYSEE', 'NEO_STEAM', 'OBLIVION', 'OBLIVION_KF', 'PRISON_TYCOON', 'PRO_CYCLING_MANAGER', 'RAGNAROK_ONLINE_2', 'RED_OCEAN', 'ROCKSMITH', 'ROCKSMITH_2014', 'SHIN_MEGAMI_TENSEI_IMAGINE', 'SID_MEIER_S_PIRATES', 'SID_MEIER_S_RAILROADS', 'SKYRIM', 'SKYRIM_SE', 'THE_GUILD_2', 'WARHAMMER', 'WILDLIFE_PARK_2', 'WIZARD_101', 'WORLD_SHIFT', 'ZOO_TYCOON_2')

The GOG/Steam version successfully imports but has an error

  File "****\AppData\Roaming\Blender Foundation\Blender\3.3\scripts\addons\io_scene_niftools\operators\nif_import_op.py", line 134, in execute
    return NifImport(self, context).execute()
  File "****\AppData\Roaming\Blender Foundation\Blender\3.3\scripts\addons\io_scene_niftools\nif_import.py", line 131, in execute
    self.import_root(root)
  File "****\AppData\Roaming\Blender Foundation\Blender\3.3\scripts\addons\io_scene_niftools\nif_import.py", line 159, in import_root
    b_obj = self.import_branch(root_block)
  File "****\AppData\Roaming\Blender Foundation\Blender\3.3\scripts\addons\io_scene_niftools\nif_import.py", line 240, in import_branch
    b_children.extend(self.import_collision(n_block))
  File "****\AppData\Roaming\Blender Foundation\Blender\3.3\scripts\addons\io_scene_niftools\nif_import.py", line 184, in import_collision
    return self.bhkhelper.import_bhk_shape(n_node.collision_object.body)
  File "****\AppData\Roaming\Blender Foundation\Blender\3.3\scripts\addons\io_scene_niftools\modules\nif_import\collision\havok.py", line 93, in import_bhk_shape
    return self.process_bhk(bhk_shape)
  File "****\Blender 3.3\3.3\python\lib\functools.py", line 889, in wrapper
    return dispatch(args[0].__class__)(*args, **kw)
  File "****\AppData\Roaming\Blender Foundation\Blender\3.3\scripts\addons\io_scene_niftools\modules\nif_import\collision\havok.py", line 151, in import_bhk_ridgidbody_t
    b_rot = bhk_shape.rotation
AttributeError: 'bhkRigidBodyT' object has no attribute 'rotation'

When trying to export, it fails with the following

niftools:INFO:Exporting
niftools:INFO:Exporting geometry and animation
niftools:INFO:Writing NIF version 0x14000004
niftools:INFO:Created meta root because blender scene had 11 root objects
niftools:INFO:Exporting NiNode block
niftools:INFO:Exporting <bpy_struct, Object("AnvilAltar01") at 0x000001DC552F5D08> as NiNode block
niftools:INFO:Exporting <bpy_struct, Object("AltarCloth01") at 0x000001DC552F6408> as NiNode block
niftools:INFO:Exporting <bpy_struct, Object("AltarCloth01:0") at 0x000001DC552F4F08>
niftools:INFO:Exporting <bpy_struct, Object("AltarCloth01:0") at 0x000001DC552F4F08> as NiTriShape block
niftools:INFO:Exporting <bpy_struct, Object("AltarCloth01:0") at 0x000001DC552F4F08> as NiTriShapeData block
niftools:INFO:Exporting NiStencilProperty block
niftools:INFO:Exporting NiSpecularProperty block
niftools:INFO:Exporting NiMaterialProperty block
niftools:ERROR:Export version expected shader. No shader applied to mesh '<bpy_struct, Material("RFINTxIron") at 0x000001DC52D61008, evaluated>', these cannot be exported to NIF.Set shader before exporting.
niftools:ERROR:****\AppData\Roaming\Blender Foundation\Blender\3.3\scripts\addons\io_scene_niftools\modules\nif_export\property\shader\__init__.py:56

Any ideas?

[Candoran2]

So the generated module is indeed in addition to the existing dependencies, you can't remove the old ones quite yet. The first error is indeed fixed in #541, not sure about the second one. It very well might be fixed there (I did make some updates for collision) but it's possible I missed some places.

As for export, I haven't made any huge changes there. Just note that, since Oblivion has a lot of nif versions, it's probably safest to set the game yourself right before exporting.

[Allofich]

I updated to Blender 3.4 (I doubt that makes a difference) and am testing the newer code from #541 now.

I can confirm the DVD version of anvilaltar01.nif imports now, although it has the same AttributeError: 'bhkRigidBodyT' object has no attribute 'rotation' error as the GOG/Steam version.

After import, the DVD version of anvilaltar01.nif has game set to UNKNOWN. After changing this to Oblivion, it successfully exported, with some warnings.

niftools:WARNING:Mopps for non-static objects may not function correctly in-game. You may wish to use simple primitives for collision.
niftools:INFO:Writing .nif file
WARNING:root:instance 0 is not a member of the <enum 'OblivionLayer'> class
WARNING:root:instance 1 is not a member of the <enum 'hkResponseType'> class
WARNING:root:instance 0 is not a member of the <enum 'OblivionLayer'> class
WARNING:root:instance 1 is not a member of the <enum 'hkResponseType'> class
WARNING:root:instance 1 is not a member of the <enum 'SpecularFlags'> class

The GOG/Steam .nif also imports (with the above-mentioned error), but the game is automatically set to Fallout 3. (I didn't notice if this happened with the develop branch code I tested earlier) I manually changed it to Oblivion and it exported successfully, with the same warnings as the DVD .nif.

Now time to go check the exported .nifs in-game.

[Allofich]

Both of the exported .nifs crash Oblivion when I try to load a saved game where I'm standing in front of the altar model. (I tried replacing the .nif with another .nif that comes with the game and that did not crash, so I think the crash is being caused by the exported .nif files)

Edit: Just to be sure I tried starting a new game (rather than loading a saved game) and warping to the cell with the .nif with the console, and the crash still happened.

[Candoran2]

Both of the exported .nifs crash Oblivion when I try to load a saved game where I'm standing in front of the altar model. (I tried replacing the .nif with another .nif that comes with the game and that did not crash, so I think the crash is being caused by the exported .nif files)

Edit: Just to be sure I tried starting a new game (rather than loading a saved game) and warping to the cell with the .nif with the console, and the crash still happened.

Probably best to start an issue at this point.

[Allofich]

Alright, I'll do so. Thanks again for helping me get things up and running so I could test the new code.

[Allofich]

Looks like you've made some fixes, so I'll re-build and re-test before making any issues.