feat(sdk): add input_type parameter for file type detection

[Zhangg7723]

Summary

• Add input_type argument to parsing methods to control file type detection mode
• Support modes: 'auto' (default), 'pdf', 'office', 'html', 'image' to bypass detection
• Enhance parse_to_md_with_binary to accept and use input_type
• Update server routes and parse service to handle input_type consistently
• Implement auto-detection fallback from filename extension to binary content
• Add detect_file_type util function to identify file format from binary data
• Extend and improve tests to cover input_type usage and auto-detection behaviors
• Add .env.example and environment variable loading in tests for API key configuration
• Modify form_data construction to include input_type in config JSON when not 'auto'
• Improve logging for input_type and format detection decisions in server and service layers
• Enhance parse_to_md_upload endpoint to accept 'file_url' for downloading and parsing files
• Allow either file upload or file_url parameter, reject providing both simultaneously
• Implement automatic filename extraction from URL or accept filename override in config
• Update SDK README with examples for file_url usage and binary parsing without extensions
• Add comprehensive tests covering valid, invalid, empty, and conflict scenarios for file_url parsing

Solution Description

[dosubot]

Related Documentation

1 document(s) may need updating based on files changed in this PR:

PowerRAG SDK

The PR introduces a new 'input_type' parameter for file type detection in document parsing methods, allowing explicit control over file type detection and bypassing auto-detection. The current documentation does not mention this parameter or its usage in the document parsing section or examples. To remain accurate and complete, the documentation should be updated to describe the 'input_type' parameter, its options, and provide usage examples reflecting the new capability.

View Suggested Changes

@@ -9,6 +9,7 @@
 - Create, update, list, and delete datasets.
 - Upload documents to datasets, supporting batch operations.
 - Parse documents to Markdown format, including direct binary parsing for PDF, Office documents, images, and HTML ([source](https://github.com/oceanbase/powerrag/pull/40)).
+- Explicit file type detection control via `input_type` parameter for parsing methods (see below).
 - Asynchronous and synchronous document parsing, with status polling and cancellation.
 - Manage document metadata, download content, and handle document chunks.
 
@@ -75,6 +76,40 @@
 - `enable_formula`: Recognize formulas (bool, default False)
 - `enable_table`: Recognize tables (bool, default True)
 - `from_page`, `to_page`: Page range for PDFs (int, default 0 and 100000)
+- `input_type`: File type detection mode (default: 'auto'). Controls how the SDK determines the file type for parsing. Options:
+    - `'auto'`: Try filename extension first, then auto-detect from binary if no extension or unsupported extension (default)
+    - `'pdf'`, `'office'`, `'html'`, `'image'`: Explicitly specify the file type to bypass detection
+
+#### Example Usage of `input_type`
+
+```python
+# Default: auto-detect using extension or binary content
+result = client.document_manager.parse_to_md_binary(
+    file_binary=open("document.pdf", "rb").read(),
+    filename="document.pdf"
+)
+
+# Auto-detect from binary when file has no extension
+result = client.document_manager.parse_to_md_binary(
+    file_binary=open("document_no_ext", "rb").read(),
+    filename="document_no_ext"
+)
+
+# Explicitly specify file type (bypass detection)
+result = client.document_manager.parse_to_md_binary(
+    file_binary=open("document", "rb").read(),
+    filename="document",
+    input_type="pdf"
+)
+
+# Using config and input_type together
+result = client.document_manager.parse_to_md_binary(
+    file_binary=open("document.html", "rb").read(),
+    filename="document.html",
+    config={"layout_recognize": "mineru"},
+    input_type="html"
+)
+```
 
 ### Chat Creation
 
@@ -115,13 +150,21 @@
 ### Parse Documents to Markdown (Binary)
 
 ```python
+# Default: auto-detect file type
 result = client.document_manager.parse_to_md_binary(
     file_binary=open("document.pdf", "rb").read(),
-    filename="document.pdf",
-    config={"layout_recognize": "mineru", "enable_ocr": True}
+    filename="document.pdf"
 )
 print(result['markdown'])
 print(f"Parsed {result['total_images']} images")
+
+# Explicitly specify file type
+result = client.document_manager.parse_to_md_binary(
+    file_binary=open("document", "rb").read(),
+    filename="document",
+    input_type="pdf"
+)
+print(result['markdown'])

(source)

</details>



<!-- Dosu Comment Footer -->
<sup>How did I do? Any feedback?</sup>&nbsp;&nbsp;[![Join Discord](https://img.shields.io/badge/join-5865F2?logo=discord&logoColor=white&label=)](https://go.dosu.dev/discord-bot)

[Copilot]

Pull request overview

This pull request adds support for automatic file type detection and URL-based file parsing to the PowerRAG SDK. The changes enable the system to handle files without extensions and download files directly from URLs for parsing.

Changes:

• Added input_type parameter ('auto', 'pdf', 'office', 'html', 'image', 'markdown') to control file type detection mode across parsing methods
• Implemented detect_file_type utility function to identify file formats from binary content using magic numbers
• Enhanced parse_to_md_upload endpoint to accept file_url parameter for downloading and parsing remote files
• Updated SDK methods to propagate and handle input_type consistently
• Added comprehensive test coverage for auto-detection and file_url functionality
• Updated test configuration to require API key via environment variables with .env file support

Reviewed changes

Copilot reviewed 9 out of 9 changed files in this pull request and generated 20 comments.

Show a summary per file

File	Description
powerrag/utils/file_utils.py	Adds detect_file_type function to identify file formats from binary data using magic numbers
powerrag/server/services/parse_service.py	Adds input_type parameter to parse_file_binary and implements auto-detection logic
powerrag/server/routes/powerrag_routes.py	Enhances /parse_to_md/upload endpoint to support file_url and input_type parameters
powerrag/sdk/modules/document_manager.py	Updates SDK methods to accept and propagate input_type parameter through API calls
powerrag/sdk/tests/conftest.py	Requires POWERRAG_API_KEY from environment, changes default port to 9390
powerrag/sdk/tests/.env.example	Provides template for test environment configuration
powerrag/sdk/tests/pytest.ini	Adds test timeout and configuration settings
powerrag/sdk/tests/test_document.py	Adds comprehensive tests for input_type auto-detection and file_url functionality
powerrag/sdk/README.md	Documents new input_type and file_url features with usage examples

Comments suppressed due to low confidence (1)

powerrag/server/routes/powerrag_routes.py:1160

• The route is calling the private method _parse_to_markdown directly instead of using the public parse_file_binary method. This bypasses the logic and validation in parse_file_binary and creates duplicate code for input_type handling. It would be better to call service.parse_file_binary(binary, filename, config, input_type) which already handles all the format detection logic, and then extract markdown and images from its return value. This would reduce code duplication and ensure consistent behavior across the codebase.

        # Create service and parse
        gotenberg_url = config.get("gotenberg_url", GOTENBERG_URL)
        service = PowerRAGParseService(gotenberg_url=gotenberg_url)
        
        # Parse to markdown
        md_content, images = service._parse_to_markdown(
            filename=filename,
            binary=binary,
            format_type=format_type,
            config=config
        )

[Copilot]

The test at line 653-666 similarly uses a minimal, potentially invalid PDF and only asserts that "filename" is in the result. This is a very weak assertion that doesn't verify whether the parsing actually succeeded or if the PDF content was correctly processed. Consider strengthening the assertion to check for "markdown" in result and verify markdown_length > 0, or use a valid PDF fixture.

[Copilot]

The valid input_type values shown in the error message at line 1147 don't include 'markdown', but the parse_file_binary method at line 190 in parse_service.py accepts 'markdown' as a valid input_type. This inconsistency means users will get an error if they try to use 'markdown' as input_type in this endpoint, even though the underlying service method supports it. Either add 'markdown' to the validation here, or remove it from parse_file_binary's validation.

[Copilot]

The file download functionality doesn't validate the file_url scheme or implement any URL allowlist. This could potentially allow Server-Side Request Forgery (SSRF) attacks where an attacker could make the server request internal network resources. Consider adding URL validation to only allow specific schemes (http/https) and optionally implement an allowlist or blocklist for domains/IPs, especially to prevent requests to internal/private network addresses.