Converters: WAV, BLUE, IQT & Others

[Teque5]

A long time ago we decided against doing this when the repos all lived in SigMF, but we should re-assess.

At GRCon23 we discussed adding converters for common formats (not metadata formats like issue #14) to include wav and other common time-series formats. Which formats would we like?

[777arc]

Yep, IQEngine was planning to add converters, so they could either live in this repo or in the IQEngine repo. It would be all python-based converters and IQEngine would provide a page where someone can upload their non-sigmf recording and it will download the sigmf-meta. Then for folks with sensitive recordings they can simply run the python converters manually.

[Teque5]

In my brain this will work like this(?):

# x is abc.wav, xyz.mp3, uvw.opus, qwe.flac
import sigmf
handle = sigmf.sigmffile.fromfile(x)
# handle now has interpreted metadata from the file, and checksum etc
samples = handle.read_samples()
handle.validate() # will pass correctly
# you can write this to sigmf archive or whatever
handle.toarchive('abc.sigmf')

If this looks good I could work up a demo pretty fast. I'm still unsure if we want to house such a converter in this repo, but it won't be that much code. There may be some lookup tables or other cumbersome things needed for the converters, but it might not be too bad.

[777arc]

Ideally there would be a python script someone can run, providing their original format as input and it generates the SigMF files

[Teque5]

Ideally there would be a python script someone can run, providing their original format as input and it generates the SigMF files

This is trivial by using the entry_points in the setup.py, but that kind of fully automated implementation would have to make some guesses that might be tricky.

Imagine if someone converts a 2-channel wav file. Do we assume this is I, Q or just a 2 channel real file. What about a 3 or 4 channel wav?

[777arc]

I think each converter is going to have some params

[Teque5]

feature/blue

@777arc & @KelseyCreekSoftware let's use this issue to discuss contributions staged within your repository discussed at our latest meeting. I have made a few passes over the bluefile converter and staged the changes in the feature/blue branch.

usage

Currently if you have the module checked out you can try the converter on any folder of .cdif files by doing something like:

export NONSIGMF_RECORDINGS_PATH=/some/location/with/blue/files
pytest -rA --log-level=DEBUG tests/test_convert.py

or by using the command-line utility:

pip install --editable .[apps]
sigmf_convert_blue some_bluefile.cdif
# you will then have some_bluefile.sigmf-meta & some_bluefile.sigmf-data in that same folder

what's working

• Currently it seems to read the headers, adjunct, and extended headers correctly.
• added support for CB format
• BLUE keys copied into SigMF keys
• raises SigMFConverversionError on problem
• This branch also contains a fix for Issue Create test for wav converter #50.

remaining tasks

• When I plot the PSDs of various formats I'm not certain that it's actually unpacking all the formats correctly. I'm going to have to set aside some time later to validate before actually creating the PR.
• Support annotations for "events". They would only have a sample start and length, no bandwidth.
• Decode Keywords within HCB

musings

It occurs to me we could not modify an existing .cdif and provide a non-conforming dataset metadata only SigMF file using simply by using core:header_bytes = 512 and core:trailing_bytes = extended_header_bytes. Perhaps after we finish this we should create sigmf_ncd or similar?

[777arc]

Good plan! As far as NCD, that's a good point, especially if it's a massive file it might be nice to not create a new .sigmf-data file and instead just make the meta in NCD format, we could have that behavior as a cmd line arg option, but I think we should get the basics (non-NCD) stuff worked in first.

[KelseyCreekSoftware]

@Teque5 and @777arc - The code consolidation / refactoring looks great! Let me know if I can help with testing or the docs - I will let you two proceed with the code review and further enhancement.

[Teque5]

@KelseyCreekSoftware I think what is now on feature/blue is just about ready. Please run some tests against any files you have and let me know. It seems to work and appear correct for the files I have seen. You can run the converter verbosely with python3 -m sigmf.convert.blue -i <filename> -vvvvv.

[KelseyCreekSoftware]

It may be my usage, but I'm seeing an issue with a missing file error, I think related to file extensions.

I'm running the utility like this:

PS C:\Data1\Ham_Radio\SDR\SigMF-MIDAS-Blue-File-Conversion\sigmf-python-feature-blue> python3 -m sigmf.convert.blue -i C:\Data1\Ham_Radio\SDR\SigMF-MIDAS-Blue-File-Conversion\SceptreTestFile1.cdif -vvvvv

The meta data file is not generated, and this missing file message is displayed.

FileNotFoundError: [WinError 2] The system cannot find the file specified: 'C:\\Data1\\Ham_Radio\\SDR\\SigMF-MIDAS-Blue-File-Conversion\\SceptreTestFile1.cdif.sigmf-data'

It does generate a SceptreTestFile1.sigmf-data file without the .cdif.

As a work around, if I copy the generated SceptreTestFile1.sigmf-data file to SceptreTestFile1.cdif.sigmf-data and re-run the utility, everything seems to work and a sigmf-meta data file that looks good, is created.

INFO:root:wrote C:\Data1\Ham_Radio\SDR\SigMF-MIDAS-Blue-File-Conversion\SceptreTestFile1.cdif.sigmf-meta

Similar behavior with a .tmp file extension provided as input, using the rust utility test resource files.

PS C:\Data1\Ham_Radio\SDR\SigMF-MIDAS-Blue-File-Conversion\sigmf-python-feature-blue> python3 -m sigmf.convert.blue -i C:\Data1\Ham_Radio\SDR\SigMF-MIDAS-Blue-File-Conversion\bluefile-main\bluefile-main\resources\test\pulse_cx.tmp -vvvvv

<snip>

FileNotFoundError: [WinError 2] The system cannot find the file specified: 'C:\\Data1\\Ham_Radio\\SDR\\SigMF-MIDAS-Blue-File-Conversion\\bluefile-main\\bluefile-main\\resources\\test\\pulse_cx.tmp.sigmf-data'

This might be related to the use of the data dictionary on about line 500?

        meta = SigMFFile(
            data_file=filenames["data_fn"],
            global_info=global_info,
        )

Additional error spew:

DEBUG:root:SCEPTRE_NOTES       :python3 meshtastic_gnuradio_RX.py -i ffffffff202f6033ef11d2c863080000302abbf12cd6917ba7d4
TEXT_MESSAGE_APP 33602f20 -> ffffffff Test
INFO:root:Using SigMF datatype: cf32_le for BLUE format CF
Traceback (most recent call last):
  File "<frozen runpy>", line 198, in _run_module_as_main
  File "<frozen runpy>", line 88, in _run_code
  File "C:\Data1\Ham_Radio\SDR\SigMF-MIDAS-Blue-File-Conversion\sigmf-python-feature-blue\sigmf\convert\blue.py", line 684, in <module>
    main()
  File "C:\Data1\Ham_Radio\SDR\SigMF-MIDAS-Blue-File-Conversion\sigmf-python-feature-blue\sigmf\convert\blue.py", line 680, in main
    _ = blue_to_sigmf(blue_path=args.input, out_path=args.output)
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Data1\Ham_Radio\SDR\SigMF-MIDAS-Blue-File-Conversion\sigmf-python-feature-blue\sigmf\convert\blue.py", line 657, in blue_to_sigmf
    meta = construct_sigmf(out_path, h_fixed, h_keywords, h_adjunct, h_extended, is_metadata_only)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Data1\Ham_Radio\SDR\SigMF-MIDAS-Blue-File-Conversion\sigmf-python-feature-blue\sigmf\convert\blue.py", line 499, in construct_sigmf
    meta = SigMFFile(
           ^^^^^^^^^^
  File "C:\Data1\Ham_Radio\SDR\SigMF-MIDAS-Blue-File-Conversion\sigmf-python-feature-blue\sigmf\sigmffile.py", line 187, in __init__
    self.set_data_file(data_file, skip_checksum=skip_checksum, map_readonly=map_readonly)
  File "C:\Data1\Ham_Radio\SDR\SigMF-MIDAS-Blue-File-Conversion\sigmf-python-feature-blue\sigmf\sigmffile.py", line 559, in set_data_file
    self._count_samples()
  File "C:\Data1\Ham_Radio\SDR\SigMF-MIDAS-Blue-File-Conversion\sigmf-python-feature-blue\sigmf\sigmffile.py", line 483, in _count_samples
    file_size = self.data_file.stat().st_size if self.data_size_bytes is None else self.data_size_bytes
                ^^^^^^^^^^^^^^^^^^^^^
  File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.11_3.11.2544.0_x64__qbz5n2kfra8p0\Lib\pathlib.py", line 1013, in stat
    return os.stat(self, follow_symlinks=follow_symlinks)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
FileNotFoundError: [WinError 2] The system cannot find the file specified: 'C:\\Data1\\Ham_Radio\\SDR\\SigMF-MIDAS-Blue-File-Conversion\\SceptreTestFile1.cdif.sigmf-data'
PS C:\Data1\Ham_Radio\SDR\SigMF-MIDAS-Blue-File-Conversion\sigmf-python-feature-blue>

Let me know if I should change my usage, or if I can provide any additional information.

[Teque5]

@KelseyCreekSoftware thanks for the windows debugging. I pushed some changes that should address the pathing issues you were having. I also rebased to main so you may need to do git fetch; git reset --hard origin/feature/blue. Please try your tests again and let me know.

This branch still has lots of debug logging but I'll delete that once we are happy with the functionality.

[KelseyCreekSoftware]

That is working much better for me under Windows 11 now!

I only have one file with real data from Sceptre. I viewed it on the IQ Engine Web site and the meta data, but not sure on the IQ content.

The other files I have from test resources for a RUST project, may be synthetic, but they all seemed to work. See screen shot from IQ Engine and a snip of debug output.

I find the debug output useful and even had a bit of the PYSDR plotting code, to try and look at the converted signal data as a cross check.

Sceptre Test File

RUST Test Files

...

DEBUG:root:parsing BLUE file data values
DEBUG:root:format: SD, data_size from header: 131072 bytes
DEBUG:root:elem_size: 8, elem_count: 16384, is_complex: False
INFO:root:wrote C:\Data1\Ham_Radio\SDR\SigMF-MIDAS-Blue-File-Conversion\bluefile-main\bluefile-main\resources\test\penny.prm.sigmf-data
INFO:root:Using SigMF datatype: rf64_le for BLUE format SD
WARNING:root:BLUE timecode is zero or missing; datetime metadata will be absent.
DEBUG:root:created SigMFFile({
    "global": {
        "blue:adjunct": {
            "subsize": 128,
            "xdelta": 1.0,
            "xstart": 0.0,
            "xunits": 0,
            "ydelta": 1.0,
            "ystart": 0.0,
            "yunits": 0
        },
        "blue:extended": {
            "COMMENT": "Demo data for XRTSURFACE/STAY",
            "COMMENT1": "XRTSURF/STAY/NOLAB/XC=5,PENNY,1.0,255.0,3,128,16",
            "COMMENT2": "XRTSURF/STAY/NOLAB/XC=5,PENNY,1.0,255.0,4,128,16",
            "COMMENT3": "XRTSURF/STAY/NOLAB/XC=5,PENNY,1.0,255.0,4,128,16,0,10,2",
            "COMMENT_1": "XRTSURF/STAY/NOLAB/XC=5,PENNY,1.0,255.0,4,128,16,0,10,2"
        },
        "blue:fixed": {
            "data_rep": "EEEI",
            "data_size": 131072.0,
            "data_start": 512.0,
            "detached": 0,
            "ext_size": 320,
            "ext_start": 257,
            "flagmask": 0,
            "format": "SD",
            "head_rep": "EEEI",
            "in_byte": 0.0,
            "inlet": 0,
            "keylength": 19,
            "keywords": "VER=1.1\u0000IO=X-Midas\u0000                                                                         ",
            "out_byte": 0.0,
            "outbytes": 0.0,
            "outlets": 0,
            "outmask": 0,
            "pipe": 0,
            "pipeloc": 0,
            "pipesize": 0,
            "protected": 1,
            "timecode": 0.0,
            "type": 2000,
            "version": "BLUE"
        },
        "blue:keywords": {
            "IO": "X-Midas",
            "VER": "1.1"
        },
        "core:author": "KE7AR",
        "core:datatype": "rf64_le",
        "core:extensions": [
            {
                "name": "blue",
                "optional": true,
                "version": "0.0.1"
            }
        ],
        "core:num_channels": 1,
        "core:recorder": "Official SigMF BLUE converter",
        "core:sample_rate": 1.0,
        "core:sha512": "25e962599bf9f542fa08d108ace7ba1c0c6af21dde656de237dd81081c9bb0ca62985a75f4fcf7855d00007b31d8feba5761baffb73ad324314c8dc223f27bde",
        "core:version": "1.2.5"
    },
    "captures": [
        {
            "core:sample_start": 0
        }
    ],
    "annotations": []
})
INFO:root:wrote C:\Data1\Ham_Radio\SDR\SigMF-MIDAS-Blue-File-Conversion\bluefile-main\bluefile-main\resources\test\penny.prm.sigmf-meta
DEBUG:root:>>>>>>>>> Fixed Header
DEBUG:root:version   : 'BLUE'  # Header version
DEBUG:root:head_rep  : 'EEEI'  # Header representation
DEBUG:root:data_rep  : 'EEEI'  # Data representation
DEBUG:root:detached  : 0  # Detached header
DEBUG:root:protected : 1  # Protected from overwrite
DEBUG:root:pipe      : 0  # Pipe mode (N/A)
DEBUG:root:ext_start : 257  # Extended header start (512-byte blocks)
DEBUG:root:ext_size  : 320  # Extended header size in bytes
DEBUG:root:data_start: 512.0  # Data start in bytes
DEBUG:root:data_size : 131072.0  # Data size in bytes
DEBUG:root:type      : 2000  # File type code
DEBUG:root:format    : 'SD'  # 2 Letter data format code
DEBUG:root:flagmask  : 0  # 16-bit flagmask
DEBUG:root:timecode  : 0.0  # Time code field
DEBUG:root:inlet     : 0  # Inlet owner
DEBUG:root:outlets   : 0  # Number of outlets
DEBUG:root:outmask   : 0  # Outlet async mask
DEBUG:root:pipeloc   : 0  # Pipe location
DEBUG:root:pipesize  : 0  # Pipe size in bytes
DEBUG:root:in_byte   : 0.0  # Next input byte
DEBUG:root:out_byte  : 0.0  # Next out byte (cumulative)
DEBUG:root:outbytes  : 0.0  # Next out byte (each outlet)
DEBUG:root:keylength : 19  # Length of keyword string
DEBUG:root:keywords  : 'VER=1.1\x00IO=X-Midas\x00                                                                         '  # User defined keyword string
DEBUG:root:>>>>>>>>> User Keywords
DEBUG:root:{'VER': '1.1', 'IO': 'X-Midas'}
DEBUG:root:>>>>>>>>> Adjunct Header
DEBUG:root:{'xstart': 0.0, 'xdelta': 1.0, 'xunits': 0, 'subsize': 128, 'ystart': 0.0, 'ydelta': 1.0, 'yunits': 0}
DEBUG:root:>>>>>>>>> Extended Header
DEBUG:root:COMMENT             :Demo data for XRTSURFACE/STAY
DEBUG:root:COMMENT             :XRTSURF/STAY/NOLAB/XC=5,PENNY,1.0,255.0,4,128,16,0,10,2
DEBUG:root:COMMENT1            :XRTSURF/STAY/NOLAB/XC=5,PENNY,1.0,255.0,3,128,16
DEBUG:root:COMMENT2            :XRTSURF/STAY/NOLAB/XC=5,PENNY,1.0,255.0,4,128,16
DEBUG:root:COMMENT3            :XRTSURF/STAY/NOLAB/XC=5,PENNY,1.0,255.0,4,128,16,0,10,2
PS C:\Data1\Ham_Radio\SDR\SigMF-MIDAS-Blue-File-Conversion\sigmf-python> python3 -m sigmf.convert.blue -i C:\Data1\Ham_Radio\SDR\SigMF-MIDAS-Blue-File-Conversion\bluefile-main\bluefile-main\resources\test\keyword_test_file.tmp -vvvvv
DEBUG:root:read C:\Data1\Ham_Radio\SDR\SigMF-MIDAS-Blue-File-Conversion\bluefile-main\bluefile-main\resources\test\keyword_test_file.tmp
INFO:root:Read BLUE type 1000 using BLUE 1.1 specification.
INFO:root:skipping data file creation for zero-sample BLUE file
INFO:root:Using SigMF datatype: ri8 for BLUE format SB
WARNING:root:BLUE timecode is zero or missing; datetime metadata will be absent.
DEBUG:root:created SigMFFile({
    "global": {
        "blue:adjunct": {
            "xdelta": 1.0,
            "xstart": 0.0,
            "xunits": 1
        },
        "blue:extended": {
            "B_TEST": 123,
            "B_TEST2": 99,
            "D_TEST": 9.87654321,
            "F_TEST": 0.12345000356435776,
            "I_TEST": 1337,
            "L_TEST": 113355,
            "O_TEST": "/w==",
            "STRING_TEST": "Hello World",
            "STRING_TEST_1": "Goodbye World",
            "X_TEST": 987654321
        },
        "blue:fixed": {
            "data_rep": "EEEI",
            "data_size": 0.0,
            "data_start": 512.0,
            "detached": 0,
            "ext_size": 224,
            "ext_start": 1,
            "flagmask": 0,
            "format": "SB",
            "head_rep": "EEEI",
            "in_byte": 0.0,
            "inlet": 0,
            "keylength": 19,
            "keywords": "VER=1.1\u0000IO=X-Midas\u0000                                                                         ",
            "out_byte": 0.0,
            "outbytes": 0.0,
            "outlets": 0,
            "outmask": 0,
            "pipe": 0,
            "pipeloc": 0,
            "pipesize": 0,
            "protected": 0,
            "timecode": 0.0,
            "type": 1000,
            "version": "BLUE"
        },
        "blue:keywords": {
            "IO": "X-Midas",
            "VER": "1.1"
        },
        "core:author": "KE7AR",
        "core:datatype": "ri8",
        "core:extensions": [
            {
                "name": "blue",
                "optional": true,
                "version": "0.0.1"
            }
        ],
        "core:metadata_only": true,
        "core:num_channels": 1,
        "core:recorder": "Official SigMF BLUE converter",
        "core:sample_rate": 1.0,
        "core:version": "1.2.5"
    },
    "captures": [
        {
            "core:sample_start": 0
        }
    ],
    "annotations": []
})
INFO:root:wrote C:\Data1\Ham_Radio\SDR\SigMF-MIDAS-Blue-File-Conversion\bluefile-main\bluefile-main\resources\test\keyword_test_file.tmp.sigmf-meta
DEBUG:root:>>>>>>>>> Fixed Header
DEBUG:root:version   : 'BLUE'  # Header version
DEBUG:root:head_rep  : 'EEEI'  # Header representation
DEBUG:root:data_rep  : 'EEEI'  # Data representation
DEBUG:root:detached  : 0  # Detached header
DEBUG:root:protected : 0  # Protected from overwrite
DEBUG:root:pipe      : 0  # Pipe mode (N/A)
DEBUG:root:ext_start : 1  # Extended header start (512-byte blocks)
DEBUG:root:ext_size  : 224  # Extended header size in bytes
DEBUG:root:data_start: 512.0  # Data start in bytes
DEBUG:root:data_size : 0.0  # Data size in bytes
DEBUG:root:type      : 1000  # File type code
DEBUG:root:format    : 'SB'  # 2 Letter data format code
DEBUG:root:flagmask  : 0  # 16-bit flagmask
DEBUG:root:timecode  : 0.0  # Time code field
DEBUG:root:inlet     : 0  # Inlet owner
DEBUG:root:outlets   : 0  # Number of outlets
DEBUG:root:outmask   : 0  # Outlet async mask
DEBUG:root:pipeloc   : 0  # Pipe location
DEBUG:root:pipesize  : 0  # Pipe size in bytes
DEBUG:root:in_byte   : 0.0  # Next input byte
DEBUG:root:out_byte  : 0.0  # Next out byte (cumulative)
DEBUG:root:outbytes  : 0.0  # Next out byte (each outlet)
DEBUG:root:keylength : 19  # Length of keyword string
DEBUG:root:keywords  : 'VER=1.1\x00IO=X-Midas\x00                                                                         '  # User defined keyword string
DEBUG:root:>>>>>>>>> User Keywords
DEBUG:root:{'VER': '1.1', 'IO': 'X-Midas'}
DEBUG:root:>>>>>>>>> Adjunct Header
DEBUG:root:{'xstart': 0.0, 'xdelta': 1.0, 'xunits': 1}
DEBUG:root:>>>>>>>>> Extended Header
DEBUG:root:B_TEST              :123
DEBUG:root:I_TEST              :1337
DEBUG:root:L_TEST              :113355
DEBUG:root:X_TEST              :987654321
DEBUG:root:F_TEST              :0.12345000356435776
DEBUG:root:D_TEST              :9.87654321
DEBUG:root:O_TEST              :/w==
DEBUG:root:STRING_TEST         :Hello World
DEBUG:root:B_TEST2             :99
DEBUG:root:STRING_TEST         :Goodbye World