RDF Ingestion Source #15520

stephengoldbaum · 2025-12-10T23:30:12Z

Summary

This PR introduces a new RDF ingestion source for DataHub, enabling ingestion of RDF/OWL ontologies (Turtle, RDF/XML, JSON-LD, N3, N-Triples) with a focus on business glossaries. The source extracts glossary terms, term hierarchies, and relationships from RDF files using standard vocabularies like SKOS, OWL, and RDFS.

What's New

Core Features

RDF Ingestion Source (type: rdf) - Native DataHub plugin for RDF/OWL ontologies
Multiple Format Support - Turtle, RDF/XML, JSON-LD, N3, N-Triples
Flexible Source Loading - Files, directories (with recursive option), URLs, and comma-separated file lists
Glossary Term Extraction - Converts skos:Concept and owl:Class to DataHub GlossaryTerms
Glossary Node Hierarchy - Auto-creates glossary nodes from IRI path hierarchies
Term Relationships - Extracts skos:broader and skos:narrower relationships as isRelatedTerms
Stateful Ingestion - Full support for stale entity removal via stateful_ingestion config
Platform Instance Support - Configurable platform instances via platform_instance config

Architecture

Modular Entity Processing - Clean separation with extractors, converters, and MCP builders
Dependency-Based Processing - Topological sort for correct entity processing order
Test Connection Support - Implements test_connection() for connection validation

Capabilities

The source supports the following DataHub capabilities:

Capability	Status	Notes
Glossary Terms	✅	Enabled by default
Glossary Nodes	✅	Auto-created from IRI path hierarchies
Term Relationships	✅	Supports `skos:broader` and `skos:narrower`
Detect Deleted Entities	✅	Requires `stateful_ingestion.enabled: true`
Platform Instance	✅	Supported via `platform_instance` config
Extract Descriptions	✅	Enabled by default (from `skos:definition` or `rdfs:comment`)
Data Domain	❌	Not applicable (domains used internally for hierarchy)
Dataset Profiling	❌	Not applicable
Extract Lineage	❌	Not in MVP
Extract Ownership	❌	Not supported
Extract Tags	❌	Not supported

Testing

Test Coverage

126 unit tests - Comprehensive coverage of core functionality, error handling, and edge cases
16 integration tests - End-to-end testing with golden file validation
Test scenarios include:
- Simple glossary ingestion
- Glossary with relationships
- Glossary with domains
- Multiple RDF formats (Turtle, RDF/XML, JSON-LD)
- Recursive directory ingestion
- Export filtering (export_only, skip_export)
- Stateful ingestion with stale entity removal
- Error handling (missing files, malformed RDF, invalid formats)
- Large file performance warnings
- Path traversal protection

Test Files

tests/unit/rdf/ - Unit tests for individual components
tests/integration/rdf/ - Integration tests with golden file validation
All tests passing ✅

Documentation

User Documentation

docs/sources/rdf/rdf.md - Comprehensive user guide (489 lines)
- Quickstart guide
- Configuration reference
- RDF format and source types
- Dialects and selective export
- Stateful ingestion guide
- Example RDF files
- IRI-to-URN mapping
- Glossary node hierarchy
- Supported vocabularies
- Limitations and troubleshooting

Recipe Examples

docs/sources/rdf/rdf_recipe.yml - Example recipes for basic and stateful ingestion

Integration Test Documentation

tests/integration/rdf/README.md - Detailed guide for running integration tests

Configuration Example

source:
type: rdf
config:
source: ./glossary.ttl
format: turtle
environment: PROD
stateful_ingestion:
enabled: true
remove_stale_metadata: true
export_only:
- glossary## Files Changed

Technical Notes

Security & Performance

URL Loading Security - Timeout limits, size limits, and redirect limits for safe URL loading
Path Traversal Protection - Configurable enforcement to prevent access outside intended directories
Memory Efficiency - Generator patterns for work unit generation and streaming for large URL downloads
Format Validation - Validates RDF formats before processing

Code Quality

Thread-Safe Registry - Entity registry uses double-checked locking pattern for thread safety
Component Validation - Validates registered components for entity type consistency
Type Safety - Complete type hints with proper forward references for MCP return types
Error Handling - Granular error reporting with structured logs and context
URN Generation - Standardized URN format using dot notation, proper encoding, and GUID fallback for non-ASCII characters

New Files

src/datahub/ingestion/source/rdf/ingestion/rdf_source.py - Main source implementation
src/datahub/ingestion/source/rdf/core/rdf_loader.py - RDF loading utilities with security
src/datahub/ingestion/source/rdf/core/urn_generator.py - URN generation with encoding
src/datahub/ingestion/source/rdf/entities/base.py - Base interfaces for entity processing
src/datahub/ingestion/source/rdf/entities/registry.py - Thread-safe entity registry
docs/sources/rdf/rdf.md - User documentation
docs/sources/rdf/rdf_recipe.yml - Recipe examples
tests/integration/rdf/test_rdf_source.py - Integration tests
tests/unit/rdf/ - Unit tests (multiple files)

Modified Files

setup.py - Added RDF source to entry points (line 862)

Breaking Changes

None - This is a new feature addition with no breaking changes to existing functionality.

Support Status

The RDF source is marked as INCUBATING (SupportStatus.INCUBATING), indicating it's ready for community adoption but may have minor version changes in future releases based on feedback.

Checklist

- Add RDF ingestion source for glossary terms, domains, and relationships - Streamlined architecture: extractors return DataHub AST directly - Removed unnecessary abstraction layers (RDF AST, converters where not needed) - Support for SKOS, OWL, and other RDF vocabularies - Comprehensive test coverage with 128 passing tests - UI integration for RDF source configuration

- Remove build_relationship_mcps() method from GlossaryTermMCPBuilder - Update tests to use RelationshipMCPBuilder directly - Clean separation: glossary_term handles terms, relationship handles relationships - Refactor _generate_workunits_from_ast to reduce complexity

…ance - Enhance error handling in RDFSource to provide actionable messages for missing files, malformed RDF, and invalid formats. - Implement unit tests to verify error handling behavior and ensure graceful degradation. - Update glossary term URN generation to use dot notation for hierarchical paths. - Improve logging for large file processing and ensure consistent URN formats across glossary nodes and terms. - Refactor methods to yield MCPs for memory efficiency during processing.

codecov · 2025-12-11T00:12:10Z

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ All tests successful. No failed tests found.

📢 Thoughts on this report? Let us know!

…uidance - Added helper text for various RDF source fields including source, format, extensions, recursive processing, environment, and dialect. - These enhancements aim to provide clearer instructions and examples for users configuring RDF ingestion settings.

- Introduced new RDF platform entry in capability_summary.json with detailed capabilities including deletion detection, tags, ownership, lineage, data profiling, domains, descriptions, and platform instance support. - Each capability includes a description and support status to enhance clarity for users configuring RDF ingestion.

codecov · 2025-12-11T23:40:35Z

Bundle Report

Changes will increase total bundle size by 2.94kB (0.01%) ⬆️. This is within the configured threshold ✅

Detailed changes

Bundle name	Size	Change
datahub-react-web-esm	28.86MB	2.94kB (0.01%) ⬆️

Affected Assets, Files, and Routes:

view changes for bundle: datahub-react-web-esm

Assets Changed:

Asset Name	Size Change	Total Size	Change (%)
`assets/index-*.js`	2.94kB	19.23MB	0.02%

Files in assets/index-*.js:

./src/app/ingestV2/source/builder/RecipeForm/rdf.ts → Total Size: 2.86kB
./src/app/ingestV2/source/builder/sources.json → Total Size: 37.23kB
./src/app/ingestV2/source/builder/constants.ts → Total Size: 6.06kB
./src/app/ingestV2/source/builder/RecipeForm/constants.ts → Total Size: 10.11kB

- Changed the warning filter to ignore specific SQLAlchemy warnings. - Added a new dependency for RDF support in the setup configuration.

…ters - Introduced a new documentation file for the RDF ingestion source. - Updated type hints across various classes to use `Optional` for context parameters, enhancing code clarity and type safety. - Adjusted method signatures in `EntityExtractor`, `EntityConverter`, `EntityMCPBuilder`, and related classes to reflect these changes.

- Included "rdf" in both base development and full test development requirements in setup.py to ensure proper support for RDF ingestion.

…erate capability_summary.json

…_summary.json

…erate capability_summary.json

This reverts commit 48a0118.

- Added unit tests for duplicate term definition handling, ensuring correct extraction behavior for same URIs and properties. - Implemented comprehensive validation tests for RDF source configuration, covering required fields, type checks, and value constraints. - Introduced connection testing unit tests to verify functionality and error handling for various scenarios, including file existence and RDF format validation. - Developed edge case tests to handle scenarios like empty files, circular relationships, and special characters in paths. - Enhanced error handling tests to ensure actionable feedback for file not found, invalid format, and unsupported extensions.

…dbaum/datahub into rdf-simplification-pr

stephengoldbaum added 4 commits December 9, 2025 15:17

Bringing RDF ingestion to DataHub standards compliance

170c558

github-actions bot added ingestion PR or Issue related to the ingestion of metadata product PR or Issue related to the DataHub UI/UX community-contribution PR or Issue raised by member(s) of DataHub Community labels Dec 10, 2025

datahub-cyborg bot added the needs-review Label for PRs that need review from a maintainer. label Dec 10, 2025

vercel bot had a problem deploying to Preview December 10, 2025 23:37 Failure

stephengoldbaum added 2 commits December 11, 2025 10:46

stephengoldbaum changed the title ~~Rdf simplification pr~~ RDF Ingestion Source Dec 11, 2025

vercel bot had a problem deploying to Preview December 11, 2025 16:40 Failure

Merge branch 'master' into rdf-simplification-pr

530aa88

vercel bot had a problem deploying to Preview December 11, 2025 23:10 Failure

fix(ingestion): update warning filters and add RDF dependency

dc640d8

- Changed the warning filter to ignore specific SQLAlchemy warnings. - Added a new dependency for RDF support in the setup configuration.

vercel bot had a problem deploying to Preview December 12, 2025 03:06 Failure

vercel bot had a problem deploying to Preview December 12, 2025 21:28 Failure

stephengoldbaum added 2 commits December 12, 2025 18:45

chore(setup): add RDF dependency to base and test requirements

7441f88

- Included "rdf" in both base development and full test development requirements in setup.py to ensure proper support for RDF ingestion.

Merge branch 'master' into rdf-simplification-pr

479a700

vercel bot had a problem deploying to Preview December 12, 2025 23:58 Failure

stephengoldbaum added 3 commits December 12, 2025 19:19

Update capability_summary.json with RDF plugin

5a95f69

Merge remote-tracking branch 'origin/rdf-simplification-pr' and regen…

9dad6fa

…erate capability_summary.json

Update capability_summary.json (auto-generated)

0e5a2c6

vercel bot had a problem deploying to Preview December 13, 2025 00:40 Failure

stephengoldbaum added 2 commits December 13, 2025 13:49

Update RDF documentation link and ensure newline at end of capability…

1f0108f

…_summary.json

Merge branch 'master' into rdf-simplification-pr

cb42706

vercel bot had a problem deploying to Preview December 13, 2025 19:02 Failure

stephengoldbaum added 2 commits December 13, 2025 14:04

Merge remote-tracking branch 'origin/rdf-simplification-pr' and regen…

b9a6038

…erate capability_summary.json

Fix capability_summary.py to ensure consistent plugin ordering

48a0118

vercel bot had a problem deploying to Preview December 13, 2025 19:19 Failure

stephengoldbaum added 2 commits December 13, 2025 15:34

Revert "Fix capability_summary.py to ensure consistent plugin ordering"

6e33396

This reverts commit 48a0118.

Fix broken markdown links in RDF documentation

fe23e45

vercel bot deployed to Preview December 13, 2025 20:53 View deployment

Merge branch 'master' into rdf-simplification-pr

631ebf6

vercel bot deployed to Preview December 15, 2025 13:57 View deployment

Merge branch 'master' into rdf-simplification-pr

182ba46

vercel bot deployed to Preview December 15, 2025 18:52 View deployment

stephengoldbaum added 2 commits December 15, 2025 16:51

Merge branch 'rdf-simplification-pr' of https://github.com/stephengol…

2948a9b

…dbaum/datahub into rdf-simplification-pr

vercel bot deployed to Preview December 15, 2025 22:09 View deployment

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

RDF Ingestion Source #15520

RDF Ingestion Source #15520

stephengoldbaum commented Dec 10, 2025

Uh oh!

codecov bot commented Dec 11, 2025 •

edited

Loading

Uh oh!

codecov bot commented Dec 11, 2025 •

edited

Loading

Assets Changed:

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

RDF Ingestion Source #15520

Are you sure you want to change the base?

RDF Ingestion Source #15520

Conversation

stephengoldbaum commented Dec 10, 2025

Summary

What's New

Core Features

Architecture

Capabilities

Testing

Test Coverage

Test Files

Documentation

User Documentation

Recipe Examples

Integration Test Documentation

Configuration Example

Technical Notes

Security & Performance

Code Quality

New Files

Modified Files

Breaking Changes

Support Status

Checklist

Uh oh!

codecov bot commented Dec 11, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

codecov bot commented Dec 11, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Bundle Report

Affected Assets, Files, and Routes:

Assets Changed:

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

codecov bot commented Dec 11, 2025 •

edited

Loading

codecov bot commented Dec 11, 2025 •

edited

Loading