[bug] @azure-tools/typespec-python `keyError: OperationGroups`

[l0lawrence]

With the recent typespec versioning updates to the rest-api-specs repo it looks like this error is back again for EventGrid System Events (a models only package): cmdout [automation_generate.sh] 2024-12-17 21:40:57 [ERROR] KeyError: 'operationGroups'

as seen here: https://github.com/Azure/azure-rest-api-specs/pull/31739/checks?check_run_id=34561185899

Was there a design change? I believe we ran into this before and resolved it.

[l0lawrence]

Ah looks like this still might be open? #2630. But see here it passed.

[swathipil]

I'm also seeing it after bumping from 0.35.1 to 0.37.1. Stack trace:

Error: Command failed: /home/swathip/repos/azure-sdk-tools/packages/python-packages/apiview-stub-generator/TempTypeSpecFiles/apiview-treestyle-parser-schema/node_modules/@typespec/http-client-python/venv/bin/python /home/swathip/repos/azure-sdk-tools/packages/python-packages/apiview-stub-generator/TempTypeSpecFiles/apiview-treestyle-parser-schema/node_modules/@typespec/http-client-python/eng/scripts/setup/run_tsp.py --output-folder=/home/swathip/repos/azure-sdk-tools/packages/python-packages/apiview-stub-generator --cadl-file=/tmp/cadl-codegen/python-yaml-path1cce935f-08a8-4ee4-82a0-c60f918da792.yaml --package-version=0.3.13 --generate-packaging-files=false --flavor=azure --package-dir=apiview-stub-generator --package-name=treestyle-parser --generate-test=false --generate-sample=false --models-mode=dpg --emit-cross-language-definition-file=true --from-typespec=true
Traceback (most recent call last):
  File "/home/swathip/repos/azure-sdk-tools/packages/python-packages/apiview-stub-generator/TempTypeSpecFiles/apiview-treestyle-parser-schema/node_modules/@typespec/http-client-python/eng/scripts/setup/run_tsp.py", line 41, in <module>
    codegen.CodeGenerator(output_folder=args.output_folder, cadl_file=args.cadl_file, **unknown_args).process()
  File "/home/swathip/repos/azure-sdk-tools/packages/python-packages/apiview-stub-generator/TempTypeSpecFiles/apiview-treestyle-parser-schema/node_modules/@typespec/http-client-python/generator/pygen/codegen/__init__.py", line 336, in process
    self.sort_exceptions(yaml_data)
  File "/home/swathip/repos/azure-sdk-tools/packages/python-packages/apiview-stub-generator/TempTypeSpecFiles/apiview-treestyle-parser-schema/node_modules/@typespec/http-client-python/generator/pygen/codegen/__init__.py", line 247, in sort_exceptions
    for group in client["operationGroups"]:
KeyError: 'operationGroups'

    at genericNodeError (node:internal/errors:983:15)
    at wrappedFn (node:internal/errors:537:14)
    at checkExecSyncError (node:child_process:882:11)
    at execSync (node:child_process:954:15)
    at $onEmit (file:///home/swathip/repos/azure-sdk-tools/packages/python-packages/apiview-stub-generator/TempTypeSpecFiles/apiview-treestyle-parser-schema/node_modules/@typespec/http-client-python/dist/emitter/emitter.js:110:9)
    at async Object.$onEmit [as emitFunction] (file:///home/swathip/repos/azure-sdk-tools/packages/python-packages/apiview-stub-generator/TempTypeSpecFiles/apiview-treestyle-parser-schema/node_modules/@azure-tools/typespec-python/dist/src/emitter.js:3:5)
    at async runEmitter (file:///home/swathip/repos/azure-sdk-tools/packages/python-packages/apiview-stub-generator/TempTypeSpecFiles/apiview-treestyle-parser-schema/node_modules/@typespec/compiler/dist/src/core/program.js:366:13)
    at async compile (file:///home/swathip/repos/azure-sdk-tools/packages/python-packages/apiview-stub-generator/TempTypeSpecFiles/apiview-treestyle-parser-schema/node_modules/@typespec/compiler/dist/src/core/program.js:123:9)
    at async compileTsp (file:///home/swathip/.nvm/versions/node/v23.4.0/lib/node_modules/@azure-tools/typespec-client-generator-cli/dist/typespec.js:97:21)
    at async generateCommand (file:///home/swathip/.nvm/versions/node/v23.4.0/lib/node_modules/@azure-tools/typespec-client-generator-cli/dist/commands.js:248:35)

--------------------------------------------------
Library Version                0.37.1
TypeSpec Compiler Version      0.63.0
--------------------------------------------------
    at runEmitter (file:///home/swathip/repos/azure-sdk-tools/packages/python-packages/apiview-stub-generator/TempTypeSpecFiles/apiview-treestyle-parser-schema/node_modules/@typespec/compiler/dist/src/core/program.js:369:19)
    at async compile (file:///home/swathip/repos/azure-sdk-tools/packages/python-packages/apiview-stub-generator/TempTypeSpecFiles/apiview-treestyle-parser-schema/node_modules/@typespec/compiler/dist/src/core/program.js:123:9)
    at async compileTsp (file:///home/swathip/.nvm/versions/node/v23.4.0/lib/node_modules/@azure-tools/typespec-client-generator-cli/dist/typespec.js:97:21)
    at async generateCommand (file:///home/swathip/.nvm/versions/node/v23.4.0/lib/node_modules/@azure-tools/typespec-client-generator-cli/dist/commands.js:248:35)
    at async updateCommand (file:///home/swathip/.nvm/versions/node/v23.4.0/lib/node_modules/@azure-tools/typespec-client-generator-cli/dist/commands.js:307:5)
    at async Object.handler (file:///home/swathip/.nvm/versions/node/v23.4.0/lib/node_modules/@azure-tools/typespec-client-generator-cli/dist/index.js:151:5) {
  info: {
    kind: 'emitter',
    metadata: {
      type: 'module',
      name: '@azure-tools/typespec-python',
      homepage: 'https://github.com/Azure/autorest.python',
      bugs: [Object],
      version: '0.37.1'
    },
    error: Error: Command failed: /home/swathip/repos/azure-sdk-tools/packages/python-packages/apiview-stub-generator/TempTypeSpecFiles/apiview-treestyle-parser-schema/node_modules/@typespec/http-client-python/venv/bin/python /home/swathip/repos/azure-sdk-tools/packages/python-packages/apiview-stub-generator/TempTypeSpecFiles/apiview-treestyle-parser-schema/node_modules/@typespec/http-client-python/eng/scripts/setup/run_tsp.py --output-folder=/home/swathip/repos/azure-sdk-tools/packages/python-packages/apiview-stub-generator --cadl-file=/tmp/cadl-codegen/python-yaml-path1cce935f-08a8-4ee4-82a0-c60f918da792.yaml --package-version=0.3.13 --generate-packaging-files=false --flavor=azure --package-dir=apiview-stub-generator --package-name=treestyle-parser --generate-test=false --generate-sample=false --models-mode=dpg --emit-cross-language-definition-file=true --from-typespec=true
    Traceback (most recent call last):
      File "/home/swathip/repos/azure-sdk-tools/packages/python-packages/apiview-stub-generator/TempTypeSpecFiles/apiview-treestyle-parser-schema/node_modules/@typespec/http-client-python/eng/scripts/setup/run_tsp.py", line 41, in <module>
        codegen.CodeGenerator(output_folder=args.output_folder, cadl_file=args.cadl_file, **unknown_args).process()
      File "/home/swathip/repos/azure-sdk-tools/packages/python-packages/apiview-stub-generator/TempTypeSpecFiles/apiview-treestyle-parser-schema/node_modules/@typespec/http-client-python/generator/pygen/codegen/__init__.py", line 336, in process
        self.sort_exceptions(yaml_data)
      File "/home/swathip/repos/azure-sdk-tools/packages/python-packages/apiview-stub-generator/TempTypeSpecFiles/apiview-treestyle-parser-schema/node_modules/@typespec/http-client-python/generator/pygen/codegen/__init__.py", line 247, in sort_exceptions
        for group in client["operationGroups"]:
    KeyError: 'operationGroups'

[msyyc]

Thanks for your feedback and let me take an investigation.

[msyyc]

Update: I have found the root cause and will fix it ASAP.

[msyyc]

@swathipil @l0lawrence The issue is fixed in latest emitter 0.37.2.

[iscai-msft]

Closing, please reopen if it's still an issue. Thanks @msyyc for your help!

[swathipil]

Hi @msyyc - I'm still seeing this issue after updating to the latest typespec-python version. Here's the trace:

Using output directory '/home/swathip/repos/azure-sdk-tools/packages/python-packages/apiview-stub-generator'

888                                      888 d8b                   888    
888                                      888 Y8P                   888    
888                                      888                       888    
888888 .d8888b  88888b.          .d8888b 888 888  .d88b.  88888b.  888888 
888    88K      888 "88b        d88P"    888 888 d8P  Y8b 888 "88b 888    
888    "Y8888b. 888  888 888888 888      888 888 88888888 888  888 888    
Y88b.       X88 888 d88P        Y88b.    888 888 Y8b.     888  888 Y88b.  
 "Y888  88888P' 88888P"          "Y8888P 888 888  "Y8888  888  888  "Y888 
                888                                                       
                888                                                       
                888                                                       

0.14.3
Found emitter package @azure-tools/typespec-python@latest
Installing dependencies from npm...

added 97 packages, and audited 98 packages in 21s

14 packages are looking for funding
  run `npm fund` for details

found 0 vulnerabilities
Resolved path: /home/swathip/repos/azure-sdk-tools/packages/python-packages/apiview-stub-generator/TempTypeSpecFiles/apiview-treestyle-parser-schema/node_modules/@typespec/compiler
Compiling tsp using @azure-tools/typespec-python...
Traceback (most recent call last):
  File "/home/swathip/repos/azure-sdk-tools/packages/python-packages/apiview-stub-generator/TempTypeSpecFiles/apiview-treestyle-parser-schema/node_modules/@typespec/http-client-python/eng/scripts/setup/run_tsp.py", line 41, in <module>
    codegen.CodeGenerator(output_folder=args.output_folder, cadl_file=args.cadl_file, **unknown_args).process()
  File "/home/swathip/repos/azure-sdk-tools/packages/python-packages/apiview-stub-generator/TempTypeSpecFiles/apiview-treestyle-parser-schema/node_modules/@typespec/http-client-python/generator/pygen/codegen/__init__.py", line 336, in process
    self.sort_exceptions(yaml_data)
  File "/home/swathip/repos/azure-sdk-tools/packages/python-packages/apiview-stub-generator/TempTypeSpecFiles/apiview-treestyle-parser-schema/node_modules/@typespec/http-client-python/generator/pygen/codegen/__init__.py", line 247, in sort_exceptions
    for group in client["operationGroups"]:
KeyError: 'operationGroups'
update

Sync and generate from a TypeSpec project

Options:
  -d, --debug            Enable debug logging                          [boolean]
  -o, --output-dir       Specify an alternate output directory for the generated
                         files.                          [string] [default: "."]
  -y, --no-prompt        Skip any interactive prompts.                 [boolean]
      --help             Show help                                     [boolean]
      --repo             Repository where the project is defined        [string]
      --commit           Commit hash to be used                         [string]
      --tsp-config       Path to tspconfig.yaml                         [string]
      --local-spec-repo  Path to local spec repo                        [string]
      --emitter-options  The options to pass to the emitter             [string]
      --save-inputs      Don't clean up the temp directory after generation
                                                                       [boolean]
  -v, --version          Show version number                           [boolean]

ExternalError: Emitter "@azure-tools/typespec-python" crashed! This is a bug.
Please file an issue at https://github.com/Azure/autorest.python/issues

Error: Command failed: /home/swathip/repos/azure-sdk-tools/packages/python-packages/apiview-stub-generator/TempTypeSpecFiles/apiview-treestyle-parser-schema/node_modules/@typespec/http-client-python/venv/bin/python /home/swathip/repos/azure-sdk-tools/packages/python-packages/apiview-stub-generator/TempTypeSpecFiles/apiview-treestyle-parser-schema/node_modules/@typespec/http-client-python/eng/scripts/setup/run_tsp.py --output-folder=/home/swathip/repos/azure-sdk-tools/packages/python-packages/apiview-stub-generator --cadl-file=/tmp/cadl-codegen/python-yaml-path3e7a3279-3ef7-4a45-9b82-59d8fad79e04.yaml --package-version=0.3.13 --generate-packaging-files=false --flavor=azure --package-dir=apiview-stub-generator --package-name=treestyle-parser --generate-test=false --generate-sample=false --models-mode=dpg --emit-cross-language-definition-file=true --from-typespec=true
Traceback (most recent call last):
  File "/home/swathip/repos/azure-sdk-tools/packages/python-packages/apiview-stub-generator/TempTypeSpecFiles/apiview-treestyle-parser-schema/node_modules/@typespec/http-client-python/eng/scripts/setup/run_tsp.py", line 41, in <module>
    codegen.CodeGenerator(output_folder=args.output_folder, cadl_file=args.cadl_file, **unknown_args).process()
  File "/home/swathip/repos/azure-sdk-tools/packages/python-packages/apiview-stub-generator/TempTypeSpecFiles/apiview-treestyle-parser-schema/node_modules/@typespec/http-client-python/generator/pygen/codegen/__init__.py", line 336, in process
    self.sort_exceptions(yaml_data)
  File "/home/swathip/repos/azure-sdk-tools/packages/python-packages/apiview-stub-generator/TempTypeSpecFiles/apiview-treestyle-parser-schema/node_modules/@typespec/http-client-python/generator/pygen/codegen/__init__.py", line 247, in sort_exceptions
    for group in client["operationGroups"]:
KeyError: 'operationGroups'

    at genericNodeError (node:internal/errors:983:15)
    at wrappedFn (node:internal/errors:537:14)
    at checkExecSyncError (node:child_process:882:11)
    at execSync (node:child_process:954:15)
    at $onEmit (file:///home/swathip/repos/azure-sdk-tools/packages/python-packages/apiview-stub-generator/TempTypeSpecFiles/apiview-treestyle-parser-schema/node_modules/@typespec/http-client-python/dist/emitter/emitter.js:110:9)
    at async Object.$onEmit [as emitFunction] (file:///home/swathip/repos/azure-sdk-tools/packages/python-packages/apiview-stub-generator/TempTypeSpecFiles/apiview-treestyle-parser-schema/node_modules/@azure-tools/typespec-python/dist/src/emitter.js:3:5)
    at async runEmitter (file:///home/swathip/repos/azure-sdk-tools/packages/python-packages/apiview-stub-generator/TempTypeSpecFiles/apiview-treestyle-parser-schema/node_modules/@typespec/compiler/dist/src/core/program.js:366:13)
    at async compile (file:///home/swathip/repos/azure-sdk-tools/packages/python-packages/apiview-stub-generator/TempTypeSpecFiles/apiview-treestyle-parser-schema/node_modules/@typespec/compiler/dist/src/core/program.js:123:9)
    at async compileTsp (file:///home/swathip/.nvm/versions/node/v23.5.0/lib/node_modules/@azure-tools/typespec-client-generator-cli/dist/typespec.js:97:21)
    at async generateCommand (file:///home/swathip/.nvm/versions/node/v23.5.0/lib/node_modules/@azure-tools/typespec-client-generator-cli/dist/commands.js:248:35)

--------------------------------------------------
Library Version                0.37.2
TypeSpec Compiler Version      0.63.0
--------------------------------------------------
    at runEmitter (file:///home/swathip/repos/azure-sdk-tools/packages/python-packages/apiview-stub-generator/TempTypeSpecFiles/apiview-treestyle-parser-schema/node_modules/@typespec/compiler/dist/src/core/program.js:369:19)
    at async compile (file:///home/swathip/repos/azure-sdk-tools/packages/python-packages/apiview-stub-generator/TempTypeSpecFiles/apiview-treestyle-parser-schema/node_modules/@typespec/compiler/dist/src/core/program.js:123:9)
    at async compileTsp (file:///home/swathip/.nvm/versions/node/v23.5.0/lib/node_modules/@azure-tools/typespec-client-generator-cli/dist/typespec.js:97:21)
    at async generateCommand (file:///home/swathip/.nvm/versions/node/v23.5.0/lib/node_modules/@azure-tools/typespec-client-generator-cli/dist/commands.js:248:35)
    at async updateCommand (file:///home/swathip/.nvm/versions/node/v23.5.0/lib/node_modules/@azure-tools/typespec-client-generator-cli/dist/commands.js:307:5)
    at async Object.handler (file:///home/swathip/.nvm/versions/node/v23.5.0/lib/node_modules/@azure-tools/typespec-client-generator-cli/dist/index.js:151:5) {
  info: {
    kind: 'emitter',
    metadata: {
      type: 'module',
      name: '@azure-tools/typespec-python',
      homepage: 'https://github.com/Azure/autorest.python',
      bugs: [Object],
      version: '0.37.2'
    },
    error: Error: Command failed: /home/swathip/repos/azure-sdk-tools/packages/python-packages/apiview-stub-generator/TempTypeSpecFiles/apiview-treestyle-parser-schema/node_modules/@typespec/http-client-python/venv/bin/python /home/swathip/repos/azure-sdk-tools/packages/python-packages/apiview-stub-generator/TempTypeSpecFiles/apiview-treestyle-parser-schema/node_modules/@typespec/http-client-python/eng/scripts/setup/run_tsp.py --output-folder=/home/swathip/repos/azure-sdk-tools/packages/python-packages/apiview-stub-generator --cadl-file=/tmp/cadl-codegen/python-yaml-path3e7a3279-3ef7-4a45-9b82-59d8fad79e04.yaml --package-version=0.3.13 --generate-packaging-files=false --flavor=azure --package-dir=apiview-stub-generator --package-name=treestyle-parser --generate-test=false --generate-sample=false --models-mode=dpg --emit-cross-language-definition-file=true --from-typespec=true
    Traceback (most recent call last):
      File "/home/swathip/repos/azure-sdk-tools/packages/python-packages/apiview-stub-generator/TempTypeSpecFiles/apiview-treestyle-parser-schema/node_modules/@typespec/http-client-python/eng/scripts/setup/run_tsp.py", line 41, in <module>
        codegen.CodeGenerator(output_folder=args.output_folder, cadl_file=args.cadl_file, **unknown_args).process()
      File "/home/swathip/repos/azure-sdk-tools/packages/python-packages/apiview-stub-generator/TempTypeSpecFiles/apiview-treestyle-parser-schema/node_modules/@typespec/http-client-python/generator/pygen/codegen/__init__.py", line 336, in process
        self.sort_exceptions(yaml_data)
      File "/home/swathip/repos/azure-sdk-tools/packages/python-packages/apiview-stub-generator/TempTypeSpecFiles/apiview-treestyle-parser-schema/node_modules/@typespec/http-client-python/generator/pygen/codegen/__init__.py", line 247, in sort_exceptions
        for group in client["operationGroups"]:
    KeyError: 'operationGroups'
    
        at genericNodeError (node:internal/errors:983:15)
        at wrappedFn (node:internal/errors:537:14)
        at checkExecSyncError (node:child_process:882:11)
        at execSync (node:child_process:954:15)
        at $onEmit (file:///home/swathip/repos/azure-sdk-tools/packages/python-packages/apiview-stub-generator/TempTypeSpecFiles/apiview-treestyle-parser-schema/node_modules/@typespec/http-client-python/dist/emitter/emitter.js:110:9)
        at async Object.$onEmit [as emitFunction] (file:///home/swathip/repos/azure-sdk-tools/packages/python-packages/apiview-stub-generator/TempTypeSpecFiles/apiview-treestyle-parser-schema/node_modules/@azure-tools/typespec-python/dist/src/emitter.js:3:5)
        at async runEmitter (file:///home/swathip/repos/azure-sdk-tools/packages/python-packages/apiview-stub-generator/TempTypeSpecFiles/apiview-treestyle-parser-schema/node_modules/@typespec/compiler/dist/src/core/program.js:366:13)
        at async compile (file:///home/swathip/repos/azure-sdk-tools/packages/python-packages/apiview-stub-generator/TempTypeSpecFiles/apiview-treestyle-parser-schema/node_modules/@typespec/compiler/dist/src/core/program.js:123:9)
        at async compileTsp (file:///home/swathip/.nvm/versions/node/v23.5.0/lib/node_modules/@azure-tools/typespec-client-generator-cli/dist/typespec.js:97:21)
        at async generateCommand (file:///home/swathip/.nvm/versions/node/v23.5.0/lib/node_modules/@azure-tools/typespec-client-generator-cli/dist/commands.js:248:35) {
      status: 1,
      signal: null,
      output: [Array],
      pid: 31159,
      stdout: <Buffer >,
      stderr: <Buffer 54 72 61 63 65 62 61 63 6b 20 28 6d 6f 73 74 20 72 65 63 65 6e 74 20 63 61 6c 6c 20 6c 61 73 74 29 3a 0a 20 20 46 69 6c 65 20 22 2f 68 6f 6d 65 2f 73 ... 935 more bytes>
    }
  }
}

Not sure if I'm missing any other package updates? Any help would be appreciated - thanks!

[msyyc]

After fix PR microsoft/typespec#5506 merged, we will release a new version.