v1.4.0 slow "[4/8 Bodies] Downloading block bodies"

[luono]

System information

Erigon version: 1.4.0

OS & Version: Ubuntu 24.04

Erigon Command (with flags/config):

./erigon --datadir=/node/data --chain=bsc --port=30311 --http --http.api=eth,debug,net,trace,web3,erigon,bsc,admin,ots --http.addr="0.0.0.0" --http.port=11488 --authrpc.port=8551 --torrent.port=42069 --private.api.addr=127.0.0.1:9090 --torrent.download.rate=1024mb --db.size.limit=4T --db.pagesize=16k --batchSize=10240M --sync.loop.block.limit=100000 --prune.mode=archive --rpc.batch.concurrency=10000 --rpc.batch.limit=1000 --rpc.returndata.limit=100000000 --bodies.cache=42949672960 --maxpeers=200

Actual behaviour

around 20 blk/sec, not patch the "blk/sec" shows in log. My hardware is 12x3.5T NVMe and network is 10Gbps.

INFO[10-14|18:22:58.004] [4/8 Bodies] Downloading block bodies    block_num=63224989 delivery/sec=1.3MB wasted/sec=0B remaining=195970 delivered=5436 blk/sec=181 cache=35.2MB alloc=15.4GB sys=22.6GB
INFO[10-14|18:23:28.004] [4/8 Bodies] Downloading block bodies    block_num=63225568 delivery/sec=1.9MB wasted/sec=71.2KB remaining=195391 delivered=5848 blk/sec=194 cache=12.5MB alloc=17.4GB sys=22.6GB
INFO[10-14|18:23:58.004] [4/8 Bodies] Downloading block bodies    block_num=63225941 delivery/sec=2.1MB wasted/sec=0B remaining=195018 delivered=6302 blk/sec=210 cache=25.2MB alloc=11.8GB sys=22.6GB
INFO[10-14|18:24:22.078] [p2p] GoodPeers                          eth68=42
INFO[10-14|18:24:28.004] [4/8 Bodies] Downloading block bodies    block_num=63226410 delivery/sec=2.0MB wasted/sec=149.4KB remaining=194549 delivered=6745 blk/sec=224 cache=20.8MB alloc=14.0GB sys=22.6GB
INFO[10-14|18:24:58.004] [4/8 Bodies] Downloading block bodies    block_num=63226802 delivery/sec=1.7MB wasted/sec=0B remaining=194157 delivered=7111 blk/sec=237 cache=16.6MB alloc=15.9GB sys=22.6GB
INFO[10-14|18:25:28.004] [4/8 Bodies] Downloading block bodies    block_num=63226927 delivery/sec=919.2KB wasted/sec=0B remaining=194032 delivered=7320 blk/sec=244 cache=26.8MB alloc=17.6GB sys=22.6GB
INFO[10-14|18:25:52.078] [p2p] GoodPeers                          eth68=47
INFO[10-14|18:25:52.570] [mem] memory stats                       Rss=44.5GB Size=0B Pss=44.5GB SharedClean=24.8MB SharedDirty=0B PrivateClean=23.2GB PrivateDirty=21.2GB Referenced=38.6GB Anonymous=21.2GB Swap=0B alloc=19.5GB sys=22.6GB
INFO[10-14|18:25:52.710] [txpool] stat                            pending=0 baseFee=0 queued=0
INFO[10-14|18:25:58.004] [4/8 Bodies] Downloading block bodies    block_num=63227622 delivery/sec=2.7MB wasted/sec=0B remaining=193351 delivered=7949 blk/sec=264 cache=18.7MB alloc=11.8GB sys=22.6GB
INFO[10-14|18:26:28.004] [4/8 Bodies] Downloading block bodies    block_num=63227872 delivery/sec=1.4MB wasted/sec=0B remaining=193104 delivered=8270 blk/sec=275 cache=29.2MB alloc=13.8GB sys=22.6GB
INFO[10-14|18:26:58.004] [4/8 Bodies] Downloading block bodies    block_num=63228194 delivery/sec=1.7MB wasted/sec=0B remaining=192765 delivered=8663 blk/sec=288 cache=37.6MB alloc=15.7GB sys=22.6GB
INFO[10-14|18:27:22.078] [p2p] GoodPeers                          eth68=53
INFO[10-14|18:27:28.004] [4/8 Bodies] Downloading block bodies    block_num=63228993 delivery/sec=3.0MB wasted/sec=0B remaining=191966 delivered=9302 blk/sec=310 cache=18.8MB alloc=18.1GB sys=22.6GB
INFO[10-14|18:27:58.004] [4/8 Bodies] Downloading block bodies    block_num=63229488 delivery/sec=2.1MB wasted/sec=69.9KB remaining=191501 delivered=9745 blk/sec=324 cache=10.3MB alloc=11.9GB sys=22.6GB
INFO[10-14|18:28:28.004] [4/8 Bodies] Downloading block bodies    block_num=63229632 delivery/sec=1.3MB wasted/sec=72.8KB remaining=191327 delivered=10024 blk/sec=334 cache=29.4MB alloc=13.7GB sys=22.6GB
INFO[10-14|18:28:52.078] [p2p] GoodPeers                          eth68=59
INFO[10-14|18:28:52.173] [snapshots] Seeding                      peers=7 conns=56 upload=32.0MB/s alloc=15.5GB sys=22.6GB
INFO[10-14|18:28:52.612] [mem] memory stats                       Rss=46.6GB Size=0B Pss=46.6GB SharedClean=35.0MB SharedDirty=0B PrivateClean=25.4GB PrivateDirty=21.2GB Referenced=40.8GB Anonymous=21.2GB Swap=0B alloc=15.5GB sys=22.6GB
INFO[10-14|18:28:52.710] [txpool] stat                            pending=0 baseFee=0 queued=0
INFO[10-14|18:28:58.004] [4/8 Bodies] Downloading block bodies    block_num=63230201 delivery/sec=2.4MB wasted/sec=140.7KB remaining=190758 delivered=10529 blk/sec=350 cache=20.6MB alloc=16.0GB sys=22.6GB

[blxdyx]

Use latest main branch. Try add this in start command: --bodies.cache=4T .

[luono]

After I build the main branch and added the flag --bodies.cache=4T. It is still slow. About 10~20blk/sec

INFO[10-15|02:41:42.800] [4/8 Bodies] Downloading block bodies    block_num=63775273 delivery/sec=2.1MB wasted/sec=161.5KB remaining=263061 delivered=2843 blk/sec=94 cache=22.1MB alloc=12.9GB sys=26.0GB
INFO[10-15|02:42:12.800] [4/8 Bodies] Downloading block bodies    block_num=63775554 delivery/sec=2.5MB wasted/sec=211.6KB remaining=262780 delivered=3213 blk/sec=107 cache=39.4MB alloc=15.0GB sys=26.0GB
INFO[10-15|02:42:42.801] [4/8 Bodies] Downloading block bodies    block_num=63775935 delivery/sec=3.0MB wasted/sec=212.0KB remaining=262399 delivered=3552 blk/sec=118 cache=47.6MB alloc=17.3GB sys=26.0GB
INFO[10-15|02:42:56.576] [p2p] GoodPeers                          eth68=50
INFO[10-15|02:42:56.885] [txpool] stat                            pending=0 baseFee=0 queued=0
INFO[10-15|02:42:57.083] [mem] memory stats                       Rss=45.2GB Size=0B Pss=45.2GB SharedClean=13.8MB SharedDirty=0B PrivateClean=22.5GB PrivateDirty=22.7GB Referenced=36.7GB Anonymous=22.6GB Swap=0B alloc=18.3GB sys=26.0GB
INFO[10-15|02:43:12.800] [4/8 Bodies] Downloading block bodies    block_num=63776117 delivery/sec=2.7MB wasted/sec=74.1KB remaining=262217 delivered=3829 blk/sec=127 cache=67.2MB alloc=19.4GB sys=26.0GB
INFO[10-15|02:43:42.800] [4/8 Bodies] Downloading block bodies    block_num=63776548 delivery/sec=3.0MB wasted/sec=81.1KB remaining=261795 delivered=4196 blk/sec=139 cache=42.5MB alloc=13.1GB sys=26.0GB
INFO[10-15|02:44:12.800] [4/8 Bodies] Downloading block bodies    block_num=63776862 delivery/sec=3.2MB wasted/sec=188.9KB remaining=261472 delivered=4631 blk/sec=154 cache=67.1MB alloc=15.3GB sys=26.0GB

[AmazingAng]

I have the same problem:

INFO[10-19|09:40:29.069] [p2p] GoodPeers                          eth68=32
INFO[10-19|09:40:29.746] [4/8 Bodies] Downloading block bodies    block_num=63262195 delivery/sec=1.2MB wasted/sec=0B remaining=5737 delivered=399 blk/sec=13 cache=24.8MB alloc=27.3GB sys=41.0GB
INFO[10-19|09:40:59.745] [4/8 Bodies] Downloading block bodies    block_num=63262318 delivery/sec=1.7MB wasted/sec=0B remaining=5614 delivered=930 blk/sec=31 cache=64.3MB alloc=28.1GB sys=41.0GB
INFO[10-19|09:41:29.745] [4/8 Bodies] Downloading block bodies    block_num=63262366 delivery/sec=1.5MB wasted/sec=69.1KB remaining=5566 delivered=1380 blk/sec=46 cache=102.2MB alloc=16.7GB sys=41.0GB

[blxdyx]

Try --bodies.cache=4G

[luono]

Try --bodies.cache=4G

I run the main branch with "--bodies.cache=4G" flag and seems that about 30~50 blk/sec. It is still very slow in "Downloading block bodies" stage comparing with "reth-bsc".

INFO[10-20|22:00:36.256] [4/8 Bodies] Downloading block bodies    block_num=63014122 delivery/sec=2.3MB wasted/sec=257.2KB remaining=191667 delivered=13135 blk/sec=437 cache=2.0MB alloc=15.4GB sys=23.4GB
INFO[10-20|22:01:06.256] [4/8 Bodies] Downloading block bodies    block_num=63015240 delivery/sec=2.4MB wasted/sec=181.0KB remaining=190549 delivered=14323 blk/sec=477 cache=6.1MB alloc=17.1GB sys=23.4GB
INFO[10-20|22:01:36.256] [4/8 Bodies] Downloading block bodies    block_num=63016587 delivery/sec=2.6MB wasted/sec=354.2KB remaining=189340 delivered=15583 blk/sec=519 cache=1.1MB alloc=12.1GB sys=23.4GB
INFO[10-20|22:01:58.735] [p2p] GoodPeers                          eth68=26
INFO[10-20|22:02:06.256] [4/8 Bodies] Downloading block bodies    block_num=63017845 delivery/sec=3.1MB wasted/sec=291.9KB remaining=187944 delivered=17104 blk/sec=570 cache=20.6MB alloc=14.1GB sys=23.4GB
INFO[10-20|22:02:36.256] [4/8 Bodies] Downloading block bodies    block_num=63018859 delivery/sec=1.8MB wasted/sec=278.5KB remaining=186971 delivered=17885 blk/sec=596 cache=4.2MB alloc=16.1GB sys=23.4GB
INFO[10-20|22:03:06.256] [4/8 Bodies] Downloading block bodies    block_num=63020654 delivery/sec=4.1MB wasted/sec=365.0KB remaining=185135 delivered=19937 blk/sec=664 cache=17.7MB alloc=18.1GB sys=23.4GB
INFO[10-20|22:03:28.734] [p2p] GoodPeers                          eth68=31
INFO[10-20|22:03:29.058] [txpool] stat                            pending=0 baseFee=0 queued=0
INFO[10-20|22:03:29.461] [mem] memory stats                       Rss=54.9GB Size=0B Pss=54.9GB SharedClean=3.1MB SharedDirty=0B PrivateClean=34.0GB PrivateDirty=20.9GB Referenced=49.3GB Anonymous=20.9GB Swap=0B alloc=19.5GB sys=23.4GB
INFO[10-20|22:03:36.256] [4/8 Bodies] Downloading block bodies    block_num=63022604 delivery/sec=3.2MB wasted/sec=387.3KB remaining=183185 delivered=21676 blk/sec=722 cache=6.6MB alloc=12.4GB sys=23.4GB
INFO[10-20|22:04:06.256] [4/8 Bodies] Downloading block bodies    block_num=63024114 delivery/sec=3.0MB wasted/sec=217.4KB remaining=181675 delivered=23161 blk/sec=772 cache=4.1MB alloc=14.2GB sys=23.4GB
INFO[10-20|22:04:36.259] [4/8 Bodies] Writing bodies              block_num=63025433 remaining=180357 alloc=16.1GB sys=23.4GB
INFO[10-20|22:04:58.734] [p2p] GoodPeers                          eth68=35
INFO[10-20|22:05:06.256] [4/8 Bodies] Downloading block bodies    block_num=63027189 delivery/sec=5.9MB wasted/sec=604.0KB remaining=178600 delivered=26314 blk/sec=877 cache=9.1MB alloc=18.1GB sys=23.4GB
INFO[10-20|22:05:36.256] [4/8 Bodies] Downloading block bodies    block_num=63028917 delivery/sec=3.6MB wasted/sec=208.3KB remaining=176872 delivered=27994 blk/sec=933 cache=6.4MB alloc=12.2GB sys=23.4GB
INFO[10-20|22:06:06.257] [4/8 Bodies] Downloading block bodies    block_num=63029386 delivery/sec=1.4MB wasted/sec=142.8KB remaining=176403 delivered=28683 blk/sec=956 cache=17.3MB alloc=13.9GB sys=23.4GB
INFO[10-20|22:06:28.734] [p2p] GoodPeers                          eth68=41
INFO[10-20|22:06:28.851] [snapshots] Seeding                      peers=11 conns=148 upload=32.0MB/s alloc=15.4GB sys=23.4GB
INFO[10-20|22:06:29.058] [txpool] stat                            pending=0 baseFee=0 queued=0
INFO[10-20|22:06:29.429] [mem] memory stats                       Rss=56.7GB Size=0B Pss=56.7GB SharedClean=3.1MB SharedDirty=0B PrivateClean=35.8GB PrivateDirty=20.9GB Referenced=55.0GB Anonymous=20.9GB Swap=0B alloc=15.4GB sys=23.4GB
INFO[10-20|22:06:36.256] [4/8 Bodies] Downloading block bodies    block_num=63031205 delivery/sec=3.0MB wasted/sec=253.2KB remaining=174584 delivered=30258 blk/sec=1008 cache=6.1MB alloc=16.1GB sys=23.4GB
INFO[10-20|22:07:06.256] [4/8 Bodies] Downloading block bodies    block_num=63032398 delivery/sec=2.5MB wasted/sec=137.4KB remaining=173391 delivered=31458 blk/sec=1048 cache=6.3MB alloc=17.9GB sys=23.4GB
INFO[10-20|22:07:36.256] [4/8 Bodies] Downloading block bodies    block_num=63033213 delivery/sec=1.6MB wasted/sec=276.4KB remaining=172576 delivered=32369 blk/sec=1078 cache=10.2MB alloc=12.3GB sys=23.4GB
INFO[10-20|22:07:58.735] [p2p] GoodPeers                          eth68=43
INFO[10-20|22:08:06.256] [4/8 Bodies] Downloading block bodies    block_num=63034157 delivery/sec=1.8MB wasted/sec=69.7KB remaining=171632 delivered=33363 blk/sec=1112 cache=12.2MB alloc=14.1GB sys=23.4GB
INFO[10-20|22:08:36.256] [4/8 Bodies] Downloading block bodies    block_num=63034968 delivery/sec=1.6MB wasted/sec=142.9KB remaining=170821 delivered=34288 blk/sec=1142 cache=15.8MB alloc=15.9GB sys=23.4GB

[luono]

After running for a while, the speed dropped to 10 to 20 blocks per second.

INFO[10-21|16:32:00.426] [4/8 Bodies] Downloading block bodies    block_num=63456990 delivery/sec=92.6KB wasted/sec=0B remaining=75984 delivered=117956 blk/sec=3931 cache=2.7GB alloc=25.4GB sys=59.0GB
INFO[10-21|16:32:30.426] [4/8 Bodies] Downloading block bodies    block_num=63457057 delivery/sec=80.5KB wasted/sec=4.0KB remaining=75917 delivered=117965 blk/sec=3932 cache=2.6GB alloc=27.4GB sys=59.0GB
INFO[10-21|16:33:00.426] [4/8 Bodies] Downloading block bodies    block_num=63457402 delivery/sec=163.6KB wasted/sec=0B remaining=75572 delivered=117988 blk/sec=3932 cache=2.6GB alloc=21.0GB sys=59.0GB
INFO[10-21|16:33:28.734] [p2p] GoodPeers                          eth68=100
INFO[10-21|16:33:29.058] [txpool] stat                            pending=0 baseFee=0 queued=0
INFO[10-21|16:33:30.426] [4/8 Bodies] Downloading block bodies    block_num=63457809 delivery/sec=161.7KB wasted/sec=0B remaining=75165 delivered=118010 blk/sec=3933 cache=2.5GB alloc=23.3GB sys=59.0GB
INFO[10-21|16:33:34.671] [mem] memory stats                       Rss=52.9GB Size=0B Pss=52.9GB SharedClean=2.6MB SharedDirty=0B PrivateClean=16.9GB PrivateDirty=36.0GB Referenced=42.6GB Anonymous=36.0GB Swap=0B alloc=23.5GB sys=59.0GB
INFO[10-21|16:34:00.426] [4/8 Bodies] Downloading block bodies    block_num=63457957 delivery/sec=114.4KB wasted/sec=0B remaining=75017 delivered=118026 blk/sec=3934 cache=2.5GB alloc=25.3GB sys=59.0GB
INFO[10-21|16:34:30.426] [4/8 Bodies] Downloading block bodies    block_num=63458088 delivery/sec=111.7KB wasted/sec=0B remaining=74886 delivered=118040 blk/sec=3934 cache=2.5GB alloc=27.4GB sys=59.0GB
INFO[10-21|16:34:58.734] [p2p] GoodPeers                          eth68=100
INFO[10-21|16:35:00.426] [4/8 Bodies] Downloading block bodies    block_num=63458268 delivery/sec=101.2KB wasted/sec=0B remaining=74706 delivered=118054 blk/sec=3935 cache=2.5GB alloc=20.7GB sys=59.0GB
INFO[10-21|16:35:30.426] [4/8 Bodies] Downloading block bodies    block_num=63458609 delivery/sec=135.3KB wasted/sec=0B remaining=74365 delivered=118072 blk/sec=3935 cache=2.4GB alloc=22.9GB sys=59.0GB
INFO[10-21|16:36:00.426] [4/8 Bodies] Downloading block bodies    block_num=63458780 delivery/sec=186.7KB wasted/sec=0B remaining=74194 delivered=118099 blk/sec=3936 cache=2.4GB alloc=25.0GB sys=59.0GB
INFO[10-21|16:36:28.734] [p2p] GoodPeers                          eth68=100
INFO[10-21|16:36:29.058] [txpool] stat                            pending=0 baseFee=0 queued=0
INFO[10-21|16:36:30.426] [4/8 Bodies] Downloading block bodies    block_num=63459072 delivery/sec=134.0KB wasted/sec=8.6KB remaining=73902 delivered=118117 blk/sec=3937 cache=2.4GB alloc=27.2GB sys=59.0GB
INFO[10-21|16:36:34.587] [mem] memory stats                       Rss=52.3GB Size=0B Pss=52.3GB SharedClean=2.6MB SharedDirty=0B PrivateClean=16.3GB PrivateDirty=36.1GB Referenced=43.0GB Anonymous=36.1GB Swap=0B alloc=27.5GB sys=59.0GB
INFO[10-21|16:37:00.426] [4/8 Bodies] Downloading block bodies    block_num=63459241 delivery/sec=121.9KB wasted/sec=0B remaining=73733 delivered=118132 blk/sec=3937 cache=2.3GB alloc=20.7GB sys=59.0GB
INFO[10-21|16:37:30.426] [4/8 Bodies] Downloading block bodies    block_num=63459433 delivery/sec=110.3KB wasted/sec=0B remaining=73541 delivered=118146 blk/sec=3938 cache=2.3GB alloc=22.8GB sys=59.0GB
INFO[10-21|16:37:58.734] [p2p] GoodPeers                          eth68=100
INFO[10-21|16:38:00.426] [4/8 Bodies] Downloading block bodies    block_num=63459597 delivery/sec=105.2KB wasted/sec=0B remaining=73377 delivered=118160 blk/sec=3938 cache=2.3GB alloc=24.9GB sys=59.0GB
INFO[10-21|16:38:30.427] [4/8 Bodies] Downloading block bodies    block_num=63459949 delivery/sec=117.0KB wasted/sec=0B remaining=73025 delivered=118176 blk/sec=3939 cache=2.2GB alloc=27.1GB sys=59.0GB