tests: add feature tests for idalib #2742

williballenthin · 2025-10-29T19:22:07Z

TODO:

add GH workflow/CI/CD configuratin
triage failures
bug: extract alternative names (function names and API names)
bug: fix references to pma16-01-function=0x404356
bug: fix references to kernel32-64-function=0x1800202B0
bug: fix references to al-khaser x64-function=0x14004B4F0
use inf_get_ostype() to help OS detection for ELF files

Checklist

No CHANGELOG update needed

No documentation update needed

github-actions

Please add bug fixes, new features, breaking changes and anything else you think is worthwhile mentioning to the master (unreleased) section of CHANGELOG.md. If no CHANGELOG update is needed add the following to the PR description: [x] No CHANGELOG update needed

CHANGELOG updated or no update needed, thanks! 😄

williballenthin · 2025-10-29T19:25:10Z

Results (524.69s (0:08:44)):
     155 passed
      15 failed
         - tests/test_idalib_features.py:23 test_idalib_features[al-khaser x64-function=0x14004B4F0-api(__vcrt_GetModuleHandle)-True]
         - tests/test_idalib_features.py:23 test_idalib_features[kernel32-64-function=0x1800202B0-api(RtlCaptureContext)-True0]
         - tests/test_idalib_features.py:23 test_idalib_features[kernel32-64-function=0x1800202B0-api(RtlCaptureContext)-True1]
         - tests/test_idalib_features.py:23 test_idalib_features[pma12-04-file-characteristic(embedded pe)-True]
         - tests/test_idalib_features.py:23 test_idalib_features[pma16-01-function=0x404356-os(windows)-True]
         - tests/test_idalib_features.py:23 test_idalib_features[pma16-01-function=0x404356-arch(i386)-True]
         - tests/test_idalib_features.py:23 test_idalib_features[pma16-01-function=0x404356-format(pe)-True]
         - tests/test_idalib_features.py:23 test_idalib_features[pma16-01-function=0x404356,bb=0x4043B9-os(windows)-True]
         - tests/test_idalib_features.py:23 test_idalib_features[pma16-01-function=0x404356,bb=0x4043B9-arch(i386)-True]
         - tests/test_idalib_features.py:23 test_idalib_features[2bf18d-function=0x4027b3,bb=0x402861,insn=0x40286d-api(__GI_connect)-True]
         - tests/test_idalib_features.py:23 test_idalib_features[2bf18d-function=0x4027b3,bb=0x402861,insn=0x40286d-api(connect)-True]
         - tests/test_idalib_features.py:23 test_idalib_features[2bf18d-function=0x4027b3,bb=0x402861,insn=0x40286d-api(__libc_connect)-True]
         - tests/test_idalib_features.py:23 test_idalib_features[2bf18d-function=0x4088a4-function-name(__GI_connect)-True]
         - tests/test_idalib_features.py:23 test_idalib_features[2bf18d-function=0x4088a4-function-name(connect)-True]
         - tests/test_idalib_features.py:23 test_idalib_features[2bf18d-function=0x4088a4-function-name(__libc_connect)-True]

williballenthin · 2025-10-29T19:35:44Z

         - tests/test_idalib_features.py:23 test_idalib_features[2bf18d-function=0x4088a4-function-name(__GI_connect)-True]
         - tests/test_idalib_features.py:23 test_idalib_features[2bf18d-function=0x4088a4-function-name(connect)-True]
         - tests/test_idalib_features.py:23 test_idalib_features[2bf18d-function=0x4088a4-function-name(__libc_connect)-True]

need to figure out how to extract "alternative names":

and propagate to callers:

also, the connect function isn't recognized as a lib function, so its name isn't being extracted (the symbol comes from symtab, not FLIRT). the ida extractor should extract all names that don't look like sub_*.

williballenthin · 2025-10-29T19:36:57Z

       - tests/test_idalib_features.py:23 test_idalib_features[pma16-01-function=0x404356-os(windows)-True]
         - tests/test_idalib_features.py:23 test_idalib_features[pma16-01-function=0x404356-arch(i386)-True]
         - tests/test_idalib_features.py:23 test_idalib_features[pma16-01-function=0x404356-format(pe)-True]
         - tests/test_idalib_features.py:23 test_idalib_features[pma16-01-function=0x404356,bb=0x4043B9-os(windows)-True]
         - tests/test_idalib_features.py:23 test_idalib_features[pma16-01-function=0x404356,bb=0x4043B9-arch(i386)-True]

IDA recognizes this as a library function, so we'd better pick a different function: 0x401100 basic block 0x401130 for example.

williballenthin · 2025-10-29T19:37:55Z

         - tests/test_idalib_features.py:23 test_idalib_features[kernel32-64-function=0x1800202B0-api(RtlCaptureContext)-True0]
         - tests/test_idalib_features.py:23 test_idalib_features[kernel32-64-function=0x1800202B0-api(RtlCaptureContext)-True1]

this is due to recognizing the containing function as _report_gsfailure (a library function) and therefore skipping analysis of it, which is correct behavior.

note there's a dup here in the test cases, we should de-dup.

williballenthin · 2025-10-30T13:55:09Z

need to figure out how to extract "alternative names"

apparently "Alternative Names" are just stored in the function comment:

  def get_alternative_names(ea):
      """Get all alternative names for an address."""
      alt_names = []

      # Check indented comment
      cmt = ida_bytes.get_cmt(ea, False)  # False = non-repeatable
      if cmt:
          for line in cmt.split('\n'):
              if line.startswith("Alternative name is '") and line.endswith("'"):
                  name = line[21:-1]  # Extract name between quotes
                  alt_names.append(name)

      # Check function comment
      pfn = ida_funcs.get_func(ea)
      if pfn:
          func_cmt = ida_funcs.get_func_cmt(pfn, False)
          if func_cmt:
              for line in func_cmt.split('\n'):
                  if line.startswith("Alternative name is '") and line.endswith("'"):
                      name = line[21:-1]
                      alt_names.append(name)

      return alt_names

williballenthin · 2025-10-31T09:01:49Z

         - tests/test_idalib_features.py:23 test_idalib_features[al-khaser x64-function=0x14004B4F0-api(__vcrt_GetModuleHandle)-True]

this is due to IDA correctly identifying a library function and therefore not analyzing it:

mr-tz · 2025-12-11T11:20:21Z

current progress

     158 passed
      10 failed
         - tests/test_idalib_features.py:26 test_idalib_features[mimikatz-function=0x40B3C6-api(LocalFree)-True]
         - tests/test_idalib_features.py:26 test_idalib_features[mimikatz-function=0x4702FD-characteristic(calls from)-False]
         - tests/test_idalib_features.py:26 test_idalib_features[pma12-04-file-characteristic(embedded pe)-True]
         - tests/test_idalib_features.py:26 test_idalib_features[2bf18d-function=0x4027b3,bb=0x402861,insn=0x40286d-api(__GI_connect)-True]
         - tests/test_idalib_features.py:26 test_idalib_features[2bf18d-function=0x4027b3,bb=0x402861,insn=0x40286d-api(connect)-True]
         - tests/test_idalib_features.py:26 test_idalib_features[2bf18d-function=0x4027b3,bb=0x402861,insn=0x40286d-api(__libc_connect)-True]
         - tests/test_idalib_features.py:26 test_idalib_features[2bf18d-function=0x4088a4-function-name(__GI_connect)-True]
         - tests/test_idalib_features.py:26 test_idalib_features[2bf18d-function=0x4088a4-function-name(connect)-True]
         - tests/test_idalib_features.py:26 test_idalib_features[2bf18d-function=0x4088a4-function-name(__libc_connect)-True]
         - tests/test_idalib_features.py:43 test_idalib_feature_counts[mimikatz-function=0x4702FD-characteristic(calls from)-0]

mr-tz · 2025-12-15T15:10:02Z

currently I see:

FAILED tests/test_idalib_features.py::test_idalib_features[pma12-04-file-characteristic(embedded pe)-True] - AssertionError: characteristic(embedded pe) should be found in file
FAILED tests/test_idalib_features.py::test_idalib_features[2bf18d-function=0x4088a4-function-name(__libc_connect)-True] - AssertionError: function-name(__libc_connect) should be found in function=0x4088a4

Results (166.29s (0:02:46)):
     166 passed
       2 failed
         - tests/test_idalib_features.py:26 test_idalib_features[pma12-04-file-characteristic(embedded pe)-True]
         - tests/test_idalib_features.py:26 test_idalib_features[2bf18d-function=0x4088a4-function-name(__libc_connect)-True]

The first is due to not loading the resource, the second is to be investigated :)

williballenthin · 2025-12-16T12:22:22Z

FAILED tests/test_idalib_features.py::test_idalib_features[2bf18d-function=0x4088a4-function-name(__libc_connect)-True] - AssertionError: function-name(__libc_connect) should be found in function=0x4088a4

IDA recognizes:

connect
__GI_connect

but doesn't recognize __libc_connect:

from the symbol table:

$ objdump -x 2bf18d0403677378adad9001b1243211.elf_ | grep "8a4 "
00000000004088a4  w    F .text  000000000000002b connect
00000000004088a4  w    F .text  000000000000002b .hidden __GI_connect
00000000004088a4 g     F .text  000000000000002b __libc_connect

for which Claude explains:

Looking at these symbol table entries, the key difference is in the binding flags:

w = Weak symbol

g = Global symbol

The three entries:

Address	Flags	Name
0x4088a4	w (weak)	connect
0x4088a4	w (weak)	__GI_connect
0x4088a4	g (global)	__libc_connect

All three point to the same code at address 0x4088a4.

What this means:

Weak (w): Can be overridden by another symbol with the same name. If you define your
own connect function, the linker will use yours instead of this one.

Global (g): The "canonical" strong symbol. Cannot be overridden by weak symbols, only
by other strong symbols (which would cause a linker error if duplicated).

Why glibc does this:

This is a common glibc pattern:

__libc_connect - The internal "real" implementation (global/strong)
connect - The public API name (weak, so users can interpose/override it)
__GI_connect - Internal glibc alias (.hidden means not exported to dynamic linker)
used for internal calls that bypass PLT overhead

The .hidden on __GI_connect means glibc's internal code can call connect() via
__GI_connect without going through the PLT, improving performance for internal calls
while still allowing user code to override the public connect symbol.

so let me see if this is a bug with IDA. edit: its not immediately clear why this is happening. there's a good amount of logic to recover various names and store them all, and I don't immediately see why __libc_connect is ignored.

williballenthin · 2025-12-16T14:06:26Z

apparently __libc_connect is being overwritten by a name provided by Lumina! if i disable Lumina in IDA Pro, then __libc_connect is the primary name for the function. there's an existing bug report against IDA for good names being overrided by Lumina, so that'll be fixed... when its fixed.

i had no idea that idalib pulled from Lumina by default, so TIL
we should probably find a way to disable that, at least within the capa session

edit: we can disable Lumina on a per-session basis either by:

setting LUMINA_PRIMARY and LUMINA_SECONDARY to the empty strings, or
providing the "-Olumina:host=" argument to init_library or open_database, however the value can't be empty, so maybe it points to something non-existent (danger! maybe use local host not an unregistered domain.)

mr-tz · 2025-12-16T15:30:20Z

apparently __libc_connect is being overwritten by a name provided by Lumina!

🙈 oh oh...

good finds here, thanks!

see #2742 in which Lumina names overwrote names provided by debug info

tests/fixtures.py

williballenthin · 2025-12-17T20:36:34Z

i'll triage these final IDA 9.0 failures tomorrow. i wonder if argv isn't supported in idalib for 9.0.

williballenthin · 2025-12-17T20:44:46Z

sure enough 9.1 added support for argv for idalib: https://docs.hex-rays.com/release-notes/9_1

so we should xfail those tests on 9.0 (and/or use 9.1)

mr-tz · 2025-12-18T08:06:55Z

I'd say we use 9.1 instead of 9.0.

williballenthin · 2026-01-06T14:30:32Z

i think this is ready to merge @mr-tz

mr-tz

almost good to go IMHO

.github/workflows/tests.yml

capa/features/extractors/ida/function.py

capa/features/extractors/ida/helpers.py

tests/fixtures.py

see #2742 in which Lumina names overwrote names provided by debug info

somehow we were extracting alternate names but not function names

9.0 doesn't support disabling lumina (or loading resources, for that matter, too)

williballenthin · 2026-01-13T15:15:59Z

assuming tests pass (just rebased), this is ready to go.

williballenthin · 2026-01-13T15:16:32Z

i'll have a follow-up PR to migrate the code to use the Domain API, which is cleaner and more idiomatic, and probably worth using instead of the low level IDA Python SDK.

mr-tz · 2026-01-13T20:45:18Z

Great work. This is a huge improvement!

tests: add feature tests for idalib

eda53ab

williballenthin added the enhancement New feature or request label Oct 29, 2025

github-actions bot previously requested changes Oct 29, 2025

View reviewed changes

changelog

f252b6b

lints

b5e5840

williballenthin and others added 2 commits November 3, 2025 12:47

fixtures: remove dups

cf46367

Merge branch 'master' into idalib-tests

074f7c7

address idalib-based test fails

dc08843

mr-tz force-pushed the idalib-tests branch 2 times, most recently from f9d5b1d to 34488b3 Compare December 15, 2025 15:29

williballenthin force-pushed the idalib-tests branch from f6f9844 to 0d44fc5 Compare December 16, 2025 14:38

williballenthin added a commit that referenced this pull request Dec 16, 2025

loader: idalib: disable lumina

1e87080

see #2742 in which Lumina names overwrote names provided by debug info

williballenthin added a commit that referenced this pull request Dec 16, 2025

loader: idalib: disable lumina

28d107c

see #2742 in which Lumina names overwrote names provided by debug info

williballenthin force-pushed the idalib-tests branch from 1e87080 to 28d107c Compare December 16, 2025 15:58

williballenthin commented Dec 17, 2025

View reviewed changes

tests/fixtures.py Outdated Show resolved Hide resolved

mr-tz mentioned this pull request Jan 6, 2026

idalib: use -R for all open_database calls #2808

Open

mr-tz reviewed Jan 6, 2026

View reviewed changes

.github/workflows/tests.yml Outdated Show resolved Hide resolved

capa/features/extractors/ida/function.py Show resolved Hide resolved

capa/features/extractors/ida/helpers.py Outdated Show resolved Hide resolved

tests/fixtures.py Outdated Show resolved Hide resolved

williballenthin force-pushed the idalib-tests branch from 58af432 to 4bea577 Compare January 13, 2026 12:44

mr-tz and others added 19 commits January 13, 2026 16:15

Merge branch 'master' into idalib-tests

7bdd1f1

ci: add configuration for idalib tests

9c6db00

tests: idalib: better detect missing idapro package

132e64a

loader: idalib: disable lumina

82be20b

see #2742 in which Lumina names overwrote names provided by debug info

ida: function: extract function name

e1fd184

somehow we were extracting alternate names but not function names

load resource for test sample

87fb96d

add ida version to test matrix name

4fb6ac0

tests: fix logging message

200c803

tests: idalib: use 9.1 instead of 9.0 as min ver

c5808c4

9.0 doesn't support disabling lumina (or loading resources, for that matter, too)

tests: idalib: xfail two tests on 9.0 and 9.1

1d561bd

ci: enable testing of IDA 9.0

f72bd49

ida: pep8

412ab62

tests: idalib: xfail resource test on 9.0

03cc901

ci: pin setup-uv

c89871f

ida: skip function-name features for default names (sub_*)

ce67d99

ida: helpers: refactor discovery of alternative names

3646fce

ida: fix return value from open_database

8d6b878

ida: loader: load resource sections to help discovery of embedded files

0686305

pyproject: ida: silence SWIG related warnings from IDA bindings

d1fc844

williballenthin force-pushed the idalib-tests branch from 6961ee1 to d1fc844 Compare January 13, 2026 15:15

mr-tz approved these changes Jan 13, 2026

View reviewed changes

mr-tz merged commit 6ad4fbb into master Jan 13, 2026
36 of 38 checks passed

mr-tz deleted the idalib-tests branch January 13, 2026 20:48

tests: add feature tests for idalib #2742

tests: add feature tests for idalib #2742

Conversation

williballenthin commented Oct 29, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Checklist

Uh oh!

github-actions bot left a comment

Choose a reason for hiding this comment

Uh oh!

williballenthin commented Oct 29, 2025

Uh oh!

williballenthin commented Oct 29, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

williballenthin commented Oct 29, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

williballenthin commented Oct 29, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

williballenthin commented Oct 30, 2025

Uh oh!

williballenthin commented Oct 31, 2025

Uh oh!

mr-tz commented Dec 11, 2025

Uh oh!

mr-tz commented Dec 15, 2025

Uh oh!

williballenthin commented Dec 16, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

williballenthin commented Dec 16, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

mr-tz commented Dec 16, 2025

Uh oh!

Uh oh!

williballenthin commented Dec 17, 2025

Uh oh!

williballenthin commented Dec 17, 2025

Uh oh!

mr-tz commented Dec 18, 2025

Uh oh!

williballenthin commented Jan 6, 2026

Uh oh!

mr-tz left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

williballenthin commented Jan 13, 2026

Uh oh!

williballenthin commented Jan 13, 2026

Uh oh!

mr-tz commented Jan 13, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

williballenthin commented Oct 29, 2025 •

edited

Loading

williballenthin commented Oct 29, 2025 •

edited

Loading

williballenthin commented Oct 29, 2025 •

edited

Loading

williballenthin commented Oct 29, 2025 •

edited

Loading

williballenthin commented Dec 16, 2025 •

edited

Loading

williballenthin commented Dec 16, 2025 •

edited

Loading