Possibility for caching mirrored repositories

[malinink]

Is there any possibility to cache mirrored repos zipballs directly in s3?

As for now, i have configured mirror:

packeton:
  mirrors:
    packagist:
      url: https://repo.packagist.org
      public_access: true
      sync_lazy: true

And s3 provider:

      STORAGE_SOURCE: s3
      STORAGE_AWS_BUCKET: test
      STORAGE_AWS_PREFIX: package
      STORAGE_AWS_ARTIFACT_PREFIX: artifact
      STORAGE_AWS_ARGS: '{"endpoint": "http://minio:9000", "accessKeyId": "minioadmin", "accessKeySecret": "minioadmin", "region": "ru-central1"}'

It work great, but i found that directory /data/composer contains package data and package folder in S3 is empty.

I need to setup packeton as stateless service, and my question is:
Is there any possibilities to store packages data that passes via packeton(mirror) into S3 to cache them?

[malinink]

I have research for a while, and found out that there are some cases that are not covered on S3 providing.
i.e.
https://github.com/vtsykun/packeton/blob/master/src/Mirror/RemoteProxyRepository.php#L39
https://github.com/vtsykun/packeton/blob/master/src/Mirror/Service/ZipballDownloadManager.php#L23

@vtsykun Am I right?

[vtsykun]

Hi @malinink

S3 for proxy is not support due to performance problem.

ZipballDownloadManager use Filesystem only for caching, not sure that we need add abstraction filesystem here, but it not problem.

Another problem with using Filesystem RemoteProxyRepository when it is lazy. I don't know how we can effective cache exists call $this->filesystem->exists() If сount of packages is large, for example 1000, it may very impact on performance. I'll want to avoide issue when proxy works slower than Packagist

[malinink]

@vtsykun
As for now, there is problem in stateless deployment of service. We have 4 upstreams of service. Problem occurs when I try to install packages, and they are not found on disk(for example after restart of service(where we loses all data, except: db, redis, S3) or after install packages and requested them in another upstream). We have a lot of 404, and composer uses original url for downloading packages.

Catch idea of slowly proxy. But if we will copy cache from S3 it will be single long call after service restart per each package. After populating local cache finished, it will respond as usual. Do you think that this behavior still affects speeding of proxy?

[vtsykun]

Yes, I understood problem. I'll investigate how we can add S3 storage support for proxies to avoid performance issue with a lot of S3 requests

[snezhkoigor]

@vtsykun Hi. There are some changes? This functionality is very important. I'm really looking forward to it :)

[y-batsianouski]

@vtsykun Hello! Are there any updates? We're also interested in this feature

[kuzaxak]

For general purpose S3 buckets (STANDARD, IA, etc.) AWS says:

your application can achieve at least 3,500 PUT/COPY/POST/DELETE or 5,500 GET/HEAD requests per second per partitioned Amazon S3 prefix. There are no limits to the number of prefixes in a bucket.

They also give the example:

If you create 10 prefixes in an S3 bucket to parallelize reads, you could scale your read performance to 55,000 read requests per second.

There’s also an overall S3 service quota per account (requests/s across prefixes), but that’s high and can be raised; the bottleneck you usually hit first is per-prefix throughput.

Internally S3 stores keys in partitions/shards:

1. A partitioned prefix is basically “a hot slice of the keyspace” that S3 has split out and given its own capacity.
2. A single prefix starts on one partition; as traffic grows, S3 automatically splits that prefix into more partitions and increases throughput.

If you want to avoid ratelimiting, use sharding as part of the path on S3:
mirror-cache/<shard>/<vendor>/<package>/<version>.zip

[kuzaxak]

I created a PR and tested it in S3-only mode (without a local cache) on a repository:

Everything works well. Let me know if you’d like to see some benchmarks. I’m open to suggestions on how to run them.

@vtsykun

[kuzaxak]

Running 2.8.6 (2025-02-25 13:03:50) with PHP 8.3.19 on Linux / 6.10.14-linuxkit

Pool optimizer completed in 1.439 seconds
Found 22,810 package versions referenced in your dependency graph. 20,278 (89%) were optimized away.
Updating dependencies
Generating rules
Resolving dependencies through SAT
Looking at all rules.
Something's changed, looking at all rules again (pass #3)
Dependency resolution completed in 0.037 seconds
Analyzed 2532 packages to resolve dependencies

No security vulnerability advisories found.

real	0m32.994s
user	0m7.781s
sys	0m1.332s