504 Gateway timeout on forge-pull of github.com/ocaml/ocaml

[NickBarnes]

Trying to get started with forge. I got as far as my first forge-pull and hit:

error in process sentinel: ghub--signal-error: HTTP Error: 504, "Gateway time-out", "/graphql", ((message . "
        ") (documentation_url . "https://github.com/magit/ghub/wiki/Github-Errors"))

I set forge.omitExpensive to true as suggested by the forge-pull docstring, but that doesn't change the error. Here's the backtrace:

Debugger entered--Lisp error: (ghub-http-error 504 "Gateway time-out" "/graphql" ((message . "\15\n        ") (documentation_url . "https://github.com/magit/ghub/wiki/Github-Errors")))
  signal(ghub-http-error (504 "Gateway time-out" "/graphql" ((message . "\15\n        ") (documentation_url . "https://github.com/magit/ghub/wiki/Github-Errors"))))
  ghub--signal-error((error http 504) ((message . "\15\n        ") (documentation_url . "https://github.com/magit/ghub/wiki/Github-Errors")) #s(ghub--graphql-req :url #s(url :type "https" :user nil :password nil :host "api.github.com" :portspec nil :filename "/graphql" :target nil :attributes nil :fullness t :silent nil :use-cookies t :asynchronous t) :forge nil :silent nil :method "POST" :headers #f(compiled-function () #<bytecode -0xf06d75e9e76a8a3>) :handler ghub--graphql-handle-response :unpaginate nil :noerror nil :reader nil :callback #f(compiled-function (data) #<bytecode -0xd176e3dd0635432>) :errorback nil :value nil :extra nil :query (query (repository [(owner $owner String!) (name $name String!)] name id createdAt updatedAt nameWithOwner (parent nameWithOwner) description homepageUrl (defaultBranchRef name) isArchived isFork isLocked isMirror isPrivate hasIssuesEnabled hasWikiEnabled (licenseInfo name) (stargazers totalCount) (watchers totalCount) (assignableUsers [(:edges t)] id login name) (issues [(:edges t) (:singular issue number) (orderBy (... ...))] number id state (author login) title createdAt updatedAt closedAt locked (milestone id) body (assignees [(:edges t)] id) (comments [(:edges t)] databaseId (author login) createdAt updatedAt body) (labels [(:edges t)] id)) (labels [(:edges t) (:singular label id)] id name color description) (milestones [(:edges t) (:singular milestone id)] id number title createdAt updatedAt dueOn closedAt description) (pullRequests [(:edges t) (:singular pullRequest number) (orderBy (... ...))] number id state (author login) title createdAt updatedAt closedAt mergedAt isDraft locked maintainerCanModify isCrossRepository (milestone id) body (baseRef name (repository nameWithOwner)) baseRefOid (headRef name (repository (owner login) nameWithOwner)) headRefOid (assignees [(:edges t)] id) (reviewRequests [(:edges t)] (requestedReviewer "... on User { id }\n")) (comments [(:edges t)] databaseId (author login) createdAt updatedAt body) (labels [(:edges t)] id)))) :query-str "query ($name: String!,$owner: String!) {\n  reposit..." :variables ((owner . "ocaml") (name . "ocaml")) :until ((issues-until) (pullRequests-until)) :buffer #<buffer magit: dev> :pages 1))
  ghub--handle-response-error((:error (error http 504)) ((message . "\15\n        ") (documentation_url . "https://github.com/magit/ghub/wiki/Github-Errors")) #s(ghub--graphql-req :url #s(url :type "https" :user nil :password nil :host "api.github.com" :portspec nil :filename "/graphql" :target nil :attributes nil :fullness t :silent nil :use-cookies t :asynchronous t) :forge nil :silent nil :method "POST" :headers #f(compiled-function () #<bytecode -0xf06d75e9e76a8a3>) :handler ghub--graphql-handle-response :unpaginate nil :noerror nil :reader nil :callback #f(compiled-function (data) #<bytecode -0xd176e3dd0635432>) :errorback nil :value nil :extra nil :query (query (repository [(owner $owner String!) (name $name String!)] name id createdAt updatedAt nameWithOwner (parent nameWithOwner) description homepageUrl (defaultBranchRef name) isArchived isFork isLocked isMirror isPrivate hasIssuesEnabled hasWikiEnabled (licenseInfo name) (stargazers totalCount) (watchers totalCount) (assignableUsers [(:edges t)] id login name) (issues [(:edges t) (:singular issue number) (orderBy (... ...))] number id state (author login) title createdAt updatedAt closedAt locked (milestone id) body (assignees [(:edges t)] id) (comments [(:edges t)] databaseId (author login) createdAt updatedAt body) (labels [(:edges t)] id)) (labels [(:edges t) (:singular label id)] id name color description) (milestones [(:edges t) (:singular milestone id)] id number title createdAt updatedAt dueOn closedAt description) (pullRequests [(:edges t) (:singular pullRequest number) (orderBy (... ...))] number id state (author login) title createdAt updatedAt closedAt mergedAt isDraft locked maintainerCanModify isCrossRepository (milestone id) body (baseRef name (repository nameWithOwner)) baseRefOid (headRef name (repository (owner login) nameWithOwner)) headRefOid (assignees [(:edges t)] id) (reviewRequests [(:edges t)] (requestedReviewer "... on User { id }\n")) (comments [(:edges t)] databaseId (author login) createdAt updatedAt body) (labels [(:edges t)] id)))) :query-str "query ($name: String!,$owner: String!) {\n  reposit..." :variables ((owner . "ocaml") (name . "ocaml")) :until ((issues-until) (pullRequests-until)) :buffer #<buffer magit: dev> :pages 1))
  ghub--graphql-handle-response((:error (error http 504)) #s(ghub--graphql-req :url #s(url :type "https" :user nil :password nil :host "api.github.com" :portspec nil :filename "/graphql" :target nil :attributes nil :fullness t :silent nil :use-cookies t :asynchronous t) :forge nil :silent nil :method "POST" :headers #f(compiled-function () #<bytecode -0xf06d75e9e76a8a3>) :handler ghub--graphql-handle-response :unpaginate nil :noerror nil :reader nil :callback #f(compiled-function (data) #<bytecode -0xd176e3dd0635432>) :errorback nil :value nil :extra nil :query (query (repository [(owner $owner String!) (name $name String!)] name id createdAt updatedAt nameWithOwner (parent nameWithOwner) description homepageUrl (defaultBranchRef name) isArchived isFork isLocked isMirror isPrivate hasIssuesEnabled hasWikiEnabled (licenseInfo name) (stargazers totalCount) (watchers totalCount) (assignableUsers [(:edges t)] id login name) (issues [(:edges t) (:singular issue number) (orderBy (... ...))] number id state (author login) title createdAt updatedAt closedAt locked (milestone id) body (assignees [(:edges t)] id) (comments [(:edges t)] databaseId (author login) createdAt updatedAt body) (labels [(:edges t)] id)) (labels [(:edges t) (:singular label id)] id name color description) (milestones [(:edges t) (:singular milestone id)] id number title createdAt updatedAt dueOn closedAt description) (pullRequests [(:edges t) (:singular pullRequest number) (orderBy (... ...))] number id state (author login) title createdAt updatedAt closedAt mergedAt isDraft locked maintainerCanModify isCrossRepository (milestone id) body (baseRef name (repository nameWithOwner)) baseRefOid (headRef name (repository (owner login) nameWithOwner)) headRefOid (assignees [(:edges t)] id) (reviewRequests [(:edges t)] (requestedReviewer "... on User { id }\n")) (comments [(:edges t)] databaseId (author login) createdAt updatedAt body) (labels [(:edges t)] id)))) :query-str "query ($name: String!,$owner: String!) {\n  reposit..." :variables ((owner . "ocaml") (name . "ocaml")) :until ((issues-until) (pullRequests-until)) :buffer #<buffer magit: dev> :pages 1))
  apply(ghub--graphql-handle-response ((:error (error http 504)) #s(ghub--graphql-req :url #s(url :type "https" :user nil :password nil :host "api.github.com" :portspec nil :filename "/graphql" :target nil :attributes nil :fullness t :silent nil :use-cookies t :asynchronous t) :forge nil :silent nil :method "POST" :headers #f(compiled-function () #<bytecode -0xf06d75e9e76a8a3>) :handler ghub--graphql-handle-response :unpaginate nil :noerror nil :reader nil :callback #f(compiled-function (data) #<bytecode -0xd176e3dd0635432>) :errorback nil :value nil :extra nil :query (query (repository [(owner $owner String!) (name $name String!)] name id createdAt updatedAt nameWithOwner (parent nameWithOwner) description homepageUrl (defaultBranchRef name) isArchived isFork isLocked isMirror isPrivate hasIssuesEnabled hasWikiEnabled (licenseInfo name) (stargazers totalCount) (watchers totalCount) (assignableUsers [(:edges t)] id login name) (issues [(:edges t) (:singular issue number) (orderBy ...)] number id state (author login) title createdAt updatedAt closedAt locked (milestone id) body (assignees [...] id) (comments [...] databaseId (author login) createdAt updatedAt body) (labels [...] id)) (labels [(:edges t) (:singular label id)] id name color description) (milestones [(:edges t) (:singular milestone id)] id number title createdAt updatedAt dueOn closedAt description) (pullRequests [(:edges t) (:singular pullRequest number) (orderBy ...)] number id state (author login) title createdAt updatedAt closedAt mergedAt isDraft locked maintainerCanModify isCrossRepository (milestone id) body (baseRef name (repository nameWithOwner)) baseRefOid (headRef name (repository ... nameWithOwner)) headRefOid (assignees [...] id) (reviewRequests [...] (requestedReviewer "... on User { id }\n")) (comments [...] databaseId (author login) createdAt updatedAt body) (labels [...] id)))) :query-str "query ($name: String!,$owner: String!) {\n  reposit..." :variables ((owner . "ocaml") (name . "ocaml")) :until ((issues-until) (pullRequests-until)) :buffer #<buffer magit: dev> :pages 1)))
  url-http-activate-callback()
  url-http-end-of-document-sentinel(#<process api.github.com> "connection broken by remote peer\n")
  url-http-async-sentinel(#<process api.github.com> "connection broken by remote peer\n")

[tarsius]

That used to happen quite frequently, so there is a FAQ entry about it. (Well about 502, but what I've written about that, should also apply to 504.)

Looking at that, I realized that it needs updating. We now know a bit more what is causing it (well at least one cause). The fix/workaround is mentioned at #20 (comment).

[NickBarnes]

That workaround (git config --local forge.graphqlItemLimit 50) changes my 504 to a 502. Various other numbers (20, 10) don't fix it.

[tarsius]

I'll delay looking into this until after the current round of major changes to Forge (i.e., until after the next release).

[tarsius]

I gave it a try a little while ago and the good news is that the changes since then were enough to prevent the server from getting overwhelmed. The bad news is that now your local machine is gonna run out of memory on such a huge repository.

I'll have to rethink the the whole "fetch everything before storing it in the database" approach (for other reasons as well). I do plan to start working on such things in a month or two, but it's going to be a lot of work.