Incorrect code suggestion

[alvin-r]

Environment details

• OS: macOS 14.5
• Node.js version: v22.7.0
• npm version: 10.8.2
• code-suggester version: 4.3.4

Steps to reproduce

You can recreate my PR at https://github.com/alvin-r/code-suggester-bug/pull/1/files.

1. Create a PR, adding a new file with the name 'common_tags.py' with:

from __future__ import annotations


def find_common_tags(articles: list[dict[str, list[str]]]) -> set[str]:
    if not articles:
        return set()

    common_tags = articles[0]["tags"]
    for article in articles[1:]:
        common_tags = [tag for tag in common_tags if tag in article["tags"]]
    return set(common_tags)

2. Use the reviewPullRequest. You can use this script:

import { Octokit } from "@octokit/rest"
import { reviewPullRequest } from "code-suggester"

async function reviewPR() {
  // Initialize authenticated Octokit instance
  const octokit = new Octokit({
    auth: <YOUR_GITHUB_TOKEN>,
  })

  // Define the files to review with their changes
  const diffContents = new Map()
  diffContents.set("common_tags.py", {
    oldContent: `from __future__ import annotations


def find_common_tags(articles: list[dict[str, list[str]]]) -> set[str]:
    if not articles:
        return set()

    common_tags = articles[0]["tags"]
    for article in articles[1:]:
        common_tags = [tag for tag in common_tags if tag in article["tags"]]
    return set(common_tags)`,

    newContent: `from __future__ import annotations


def find_common_tags(articles: list[dict[str, list[str]]]) -> set[str]:
    if not articles:
        return set()

    common_tags = set(articles[0]["tags"])
    for article in articles[1:]:
        common_tags.intersection_update(article["tags"])
    return common_tags`,
  })

  // Configure review options
  const options = {
    owner: "alvin-r",
    repo: "code-suggester-bug",
    pullNumber: 1,
    pageSize: 100,
  }

  try {
    const result = await reviewPullRequest(octokit, diffContents, options)
    console.log("Review completed:", result)
  } catch (error) {
    console.error("Review failed:", error)
  }
}

reviewPR()

3. When the review is created, the suggested code does not remove the 'return set(common_tags)' from the original code. You can see the comment created in https://github.com/alvin-r/code-suggester-bug/pull/1/files.

[misrasaurabh1]

Hi @chingor13 , we running into many other bugs with code-suggester as I am using it for codeflash.ai's github diff creation. Would appreciate a quick look here at the bug.

[misrasaurabh1]

@chingor13 if i can get your attention on this. code-suggester is unusable in a real application. We would need to move out of code-suggestor without these bugs being fixed.

[misrasaurabh1]

Here is the response of Google after I reached out directly -

Hi Saurabh,

Code-suggested is primarily used for our internal purposes and was only ever released as an early alpha. In particular, Jeff explained to me that the code diffing functionality has always been problematic. That particular function is not getting active investment from us and is probably not worth depending upon.

Thanks for your understanding,

[chingor13]

The code review feature never worked perfectly and "This library is considered to be in alpha".

[Saga4]

@chingor13 As this project still has 150k weekly download, could you please take a review on the fix for this issue:
#519