Skip to content

Conversation

@mattbenjamin
Copy link

Add mode 'u' to do unordered bucket listing, using the
ListObjectsRequest mechanism.

Signed-off-by: Matt Benjamin mbenjamin@redhat.com

Add mode 'u' to do unordered bucket listing, using the
ListObjectsRequest mechanism.

Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>
@markhpc markhpc self-requested a review August 29, 2019 13:04
Copy link
Owner

@markhpc markhpc left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Tried this against my current test cluster but it didn't appear to be working vs the regular list:

2019/08/29 08:55:09 Running Loop 0 OBJECT PUT TEST
2019/08/29 08:55:10 Loop: 0, Int: 0, Dur(s): 1.0, Mode: PUT, Ops: 6439, MB/s: 25.15, IO/s: 6439, Lat(ms): [ min: 2.9, avg: 19.5, 99%: 50.2, max: 74.0 ], Slowdowns: 0
2019/08/29 08:55:11 Loop: 0, Int: 1, Dur(s): 1.0, Mode: PUT, Ops: 6384, MB/s: 24.94, IO/s: 6384, Lat(ms): [ min: 2.3, avg: 20.0, 99%: 61.8, max: 88.8 ], Slowdowns: 0
2019/08/29 08:55:12 Loop: 0, Int: 2, Dur(s): 1.0, Mode: PUT, Ops: 6888, MB/s: 26.91, IO/s: 6888, Lat(ms): [ min: 1.4, avg: 18.6, 99%: 52.8, max: 73.4 ], Slowdowns: 0
2019/08/29 08:55:13 Loop: 0, Int: 3, Dur(s): 1.0, Mode: PUT, Ops: 6716, MB/s: 26.23, IO/s: 6716, Lat(ms): [ min: 2.2, avg: 19.0, 99%: 54.1, max: 82.9 ], Slowdowns: 0
2019/08/29 08:55:14 Loop: 0, Int: 4, Dur(s): 1.0, Mode: PUT, Ops: 6646, MB/s: 25.96, IO/s: 6646, Lat(ms): [ min: 2.2, avg: 19.3, 99%: 56.5, max: 85.0 ], Slowdowns: 0
2019/08/29 08:55:15 Loop: 0, Int: 5, Dur(s): 1.0, Mode: PUT, Ops: 6622, MB/s: 25.87, IO/s: 6622, Lat(ms): [ min: 2.7, avg: 19.2, 99%: 55.7, max: 86.8 ], Slowdowns: 0
2019/08/29 08:55:16 Loop: 0, Int: 6, Dur(s): 1.0, Mode: PUT, Ops: 5784, MB/s: 22.59, IO/s: 5784, Lat(ms): [ min: 1.9, avg: 22.1, 99%: 119.9, max: 157.2 ], Slowdowns: 0
2019/08/29 08:55:17 Loop: 0, Int: 7, Dur(s): 1.0, Mode: PUT, Ops: 6594, MB/s: 25.76, IO/s: 6594, Lat(ms): [ min: 1.9, avg: 19.5, 99%: 66.8, max: 93.7 ], Slowdowns: 0
2019/08/29 08:55:18 Loop: 0, Int: 8, Dur(s): 1.0, Mode: PUT, Ops: 7050, MB/s: 27.54, IO/s: 7050, Lat(ms): [ min: 2.9, avg: 17.9, 99%: 122.5, max: 158.1 ], Slowdowns: 0
2019/08/29 08:55:19 Loop: 0, Int: 9, Dur(s): 1.0, Mode: PUT, Ops: 8782, MB/s: 34.30, IO/s: 8782, Lat(ms): [ min: 3.6, avg: 14.9, 99%: 29.9, max: 42.3 ], Slowdowns: 0
2019/08/29 08:55:20 Loop: 0, Int: 10, Dur(s): 1.0, Mode: PUT, Ops: 8563, MB/s: 33.45, IO/s: 8563, Lat(ms): [ min: 4.3, avg: 15.0, 99%: 25.4, max: 34.4 ], Slowdowns: 0
2019/08/29 08:55:21 Loop: 0, Int: 11, Dur(s): 1.0, Mode: PUT, Ops: 7862, MB/s: 30.71, IO/s: 7862, Lat(ms): [ min: 4.9, avg: 16.2, 99%: 78.7, max: 88.8 ], Slowdowns: 0
2019/08/29 08:55:22 Loop: 0, Int: 12, Dur(s): 1.0, Mode: PUT, Ops: 8206, MB/s: 32.05, IO/s: 8206, Lat(ms): [ min: 3.0, avg: 15.6, 99%: 54.1, max: 85.6 ], Slowdowns: 0
2019/08/29 08:55:23 Loop: 0, Int: TOTAL, Dur(s): 13.9, Mode: PUT, Ops: 100000, MB/s: 28.17, IO/s: 7211, Lat(ms): [ min: 1.4, avg: 17.7, 99%: 57.2, max: 158.1 ], Slowdowns: 0
2019/08/29 08:55:23 Running Loop 0 BUCKET LIST TEST
2019/08/29 08:55:29 Loop: 0, Int: TOTAL, Dur(s): 5.5, Mode: LIST, Ops: 100, MB/s: 0.00, IO/s: 18, Lat(ms): [ min: 41.5, avg: 55.0, 99%: 65.4, max: 65.9 ], Slowdowns: 0
2019/08/29 08:55:29 Running Loop 0 BUCKET UNORDERED LIST TEST
2019/08/29 08:55:29 Loop: 0, Int: TOTAL, Dur(s): 0.1, Mode: UNORDERED LIST, Ops: 1, MB/s: 0.00, IO/s: 20, Lat(ms): [ min: 50.3, avg: 50.3, 99%: 50.3, max: 50.3 ], Slowdowns: 0

}
case 'u':
log.Printf("Running Loop %d BUCKET UNORDERED LIST TEST", loop)
stats = makeStats(loop, "UNORDERED LIST", threads, intervalNano)
Copy link
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: trying to keep these short since the log lines are so big. :) Can we change it from "UNORDERED LIST" to "ULIST"?

input := &s3.ListObjectsInput{
Bucket: &buckets[bucket_num],
MaxKeys: &max_keys,
Marker: &next_marker,
Copy link
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I remember there was some issue with using marker with the V1 ListObjects which is ultimately why I switched to ListObjectsPages. Not sure if this is related to the issue I hit above?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants