Sentry: Error uploading sourcemap file - Error: connect EADDRNOTAVAIL 35.188.42.15:443 - Local

[igorescobar]

Hi,

A few days from now we started to get this error on our pipelines:

Error ---------------------------------------------------
 
  Error: Sentry: Error uploading sourcemap file - Error: connect EADDRNOTAVAIL 35.188.42.15:443 - Local (172.17.0.2:0)
      at SentryPlugin.<anonymous> (/builds/services/node_modules/serverless-sentry/dist/index.js:603:31)
      at step (/builds/services/node_modules/serverless-sentry/dist/index.js:44:23)
      at Object.throw (/builds/services/node_modules/serverless-sentry/dist/index.js:25:53)
      at rejected (/builds/services/node_modules/serverless-sentry/dist/index.js:17:65)
      at processTicksAndRejections (internal/process/task_queues.js:95:5)
 
     For debugging logs, run again after setting the "SLS_DEBUG=*" environment variable.
 
  Get Support --------------------------------------------
     Docs:          docs.serverless.com
     Bugs:          github.com/serverless/serverless/issues
     Issues:        forum.serverless.com
 
  Your Environment Information ---------------------------
     Operating System:          linux
     Node Version:              14.19.3
     Framework Version:         2.72.3 (local)
     Plugin Version:            5.5.4
     SDK Version:               4.3.2
     Components Version:        3.18.2

This is what our dependencies look like at the moment:

    "@sentry/integrations": "^7.2.0",
    "@sentry/node": "^7.2.0",
    "serverless-sentry-lib": "^2.5.0",
    "serverless-sentry": "^2.5.0",

We even tried to disable this functionality by adding the environment variable SENTRY_SOURCEMAPS or adding the custom config:

sentry:
    sourceMaps: false

But it runs regardless and ignores the flag. Do you have any idea what is happening and how to work around this?

[igorescobar]

Maybe is this related?
https://stackoverflow.com/questions/21859537/connect-eaddrnotavail-in-nodejs-under-high-load-how-to-faster-free-or-reuse-tc

[igorescobar]

Maybe we are being too aggressive here:
https://github.com/arabold/serverless-sentry-plugin/blob/master/src/index.ts#L476 and we are running out of TCP ports/sockets or something because of the large number of files being sent?

[jonmast]

Thanks for reporting @igorescobar, I'm sorry this is causing you problems. I did see that error myself but it went away on its own so I assumed it was just some sort of transient network issue.

It still uploading with sourceMaps: false is a bug, we should add a check to prevent that from happening.

As a temporary workaround you could remove your authToken config, that will disable releases entirely which will skip sourcemap upload.

My understanding is that there are around 30K ports available so I wouldn't have thought we'd be hitting that. Your setup is a bit different than mine though, I'm using serverless-webpack to bundle my files so there are only a few that get uploaded. If your setup is pulling in all of node_modules there will be many more and it's plausible you're hitting the limit. We should probably be ignoring that folder since most libraries don't have sourcemaps.

[igorescobar]

Yeah, I also noticed that after upgrading to there is no way to disable the source maps upload (besides doing what you described). the only way is rolling it back.

[arabold]

I'm literally on a plane right now but should have a quick fix in a bit.

[arabold]

No risk, no fun. Just published a fully untested 2.5.1 that (hopefully) fixes two problems:

• Fix #63: Upload source maps serially to avoid running out of sockets.
• Correctly disable uploading source maps if the config setting is false or unset.

🤞

[igorescobar]

Fix #63: Upload source maps serially to avoid running out of sockets.

Maybe we should do this in batches instead of calling all of them in parallel or in series.... creating batches of 50-100 requests might fix the issue. In our case it would take ages to submit all files I suppose.

[arabold]

The best would be to upload like 3 to 5 simultaneously for best performance. 50-100 parallel connections are a lot. But I currently lack the time and ability to test this properly, so I went for the quickest fix. It's slower but should work regardless of how large the project is and how many artifacts it has to upload.

[igorescobar]

The best would be to upload like 3 to 5 simultaneously for best performance. 50-100 parallel connections are a lot. But I currently lack the time and ability to test this properly, so I went for the quickest fix. It's slower but should work regardless of how large the project is and how many artifacts it has to upload.

I would also create a property to configure it if necessary. It will be hard to come up with a config that will be nice for everyone so be conservative but also allow people to be more aggressive if they have the computing power to do so 👍

[arabold]

I have another update 2.5.2. Can you give it a try? It will upload 5 in parallel which should not impact your performance anymore. 10, 20, 50 really shouldn't make a noticeable difference, but feel free to adjust it and try it out if you notice something. I'm open to any changes. Again, this is fully untested 🙄

[igorescobar]

Tests checklist v2.5.2

• Source code is now honoring the sourceMaps default false and it deployed the application with no issues. (Job finished in 7 minutes 37 seconds)
• sourceMaps: true. Job couldn't finish. I was taking too much time. I stop the job at 45min.

[igorescobar]

Already 20 minutes in and counting 😄 . No errors but as expected... it will take ages to finish 😄

[arabold]

😞 How long did it take before my changes to serialize the upload?

[igorescobar]

without sourcemaps it took about 7 minutes. with sourcemaps we are now 32 minutes in.