Of Apples and Oranges: Fair Comparisons in Heterogenous Systems Evaluation

Hugo Sadok; Aurojit Panda; Justine Sherry

doi:10.1145/3626111.3628186

Of Apples and Oranges: Fair Comparisons in Heterogenous Systems Evaluation

Hugo Sadok, Aurojit Panda, Justine Sherry

Computer Science

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

Abstract

Accelerators, such as GPUs, SmartNICs and FPGAs, are common components of research systems today. This paper focuses on the question of how to fairly compare these systems. This is challenging because it requires comparing systems that use different hardware, e.g., two systems that use two different types of accelerators, or comparing a system that uses an accelerator with one that does not. We argue that fair evaluation in this case requires reporting not just performance, but also the cost of competing systems. We discuss what cost metrics should be used, and propose general principles for incorporating cost in research evaluations.

Original language	English (US)
Title of host publication	HotNets 2023 - Proceedings of the 22nd ACM Workshop on Hot Topics in Networks
Publisher	Association for Computing Machinery, Inc
Pages	1-8
Number of pages	8
ISBN (Electronic)	9798400704154
DOIs	https://doi.org/10.1145/3626111.3628186
State	Published - Nov 28 2023
Event	22nd ACM Workshop on Hot Topics in Networks, HotNets 2023 - Cambridge, United States Duration: Nov 28 2023 → Nov 29 2023

Publication series

Name	HotNets 2023 - Proceedings of the 22nd ACM Workshop on Hot Topics in Networks

Conference

Conference	22nd ACM Workshop on Hot Topics in Networks, HotNets 2023
Country/Territory	United States
City	Cambridge
Period	11/28/23 → 11/29/23

Keywords

Accelerator
Fair Comparison
Heterogenous Hardware

ASJC Scopus subject areas

Computer Networks and Communications

Access to Document

10.1145/3626111.3628186

Cite this

Sadok, H., Panda, A., & Sherry, J. (2023). Of Apples and Oranges: Fair Comparisons in Heterogenous Systems Evaluation. In HotNets 2023 - Proceedings of the 22nd ACM Workshop on Hot Topics in Networks (pp. 1-8). (HotNets 2023 - Proceedings of the 22nd ACM Workshop on Hot Topics in Networks). Association for Computing Machinery, Inc. https://doi.org/10.1145/3626111.3628186

Of Apples and Oranges: Fair Comparisons in Heterogenous Systems Evaluation. / Sadok, Hugo; Panda, Aurojit; Sherry, Justine.
HotNets 2023 - Proceedings of the 22nd ACM Workshop on Hot Topics in Networks. Association for Computing Machinery, Inc, 2023. p. 1-8 (HotNets 2023 - Proceedings of the 22nd ACM Workshop on Hot Topics in Networks).

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

Sadok, H, Panda, A & Sherry, J 2023, Of Apples and Oranges: Fair Comparisons in Heterogenous Systems Evaluation. in HotNets 2023 - Proceedings of the 22nd ACM Workshop on Hot Topics in Networks. HotNets 2023 - Proceedings of the 22nd ACM Workshop on Hot Topics in Networks, Association for Computing Machinery, Inc, pp. 1-8, 22nd ACM Workshop on Hot Topics in Networks, HotNets 2023, Cambridge, United States, 11/28/23. https://doi.org/10.1145/3626111.3628186

@inproceedings{cc8347618ba74e99b70ea1e7362d894e,

title = "Of Apples and Oranges: Fair Comparisons in Heterogenous Systems Evaluation",

abstract = "Accelerators, such as GPUs, SmartNICs and FPGAs, are common components of research systems today. This paper focuses on the question of how to fairly compare these systems. This is challenging because it requires comparing systems that use different hardware, e.g., two systems that use two different types of accelerators, or comparing a system that uses an accelerator with one that does not. We argue that fair evaluation in this case requires reporting not just performance, but also the cost of competing systems. We discuss what cost metrics should be used, and propose general principles for incorporating cost in research evaluations.",

keywords = "Accelerator, Fair Comparison, Heterogenous Hardware",

author = "Hugo Sadok and Aurojit Panda and Justine Sherry",

note = "Publisher Copyright: {\textcopyright} 2023 Owner/Author.; 22nd ACM Workshop on Hot Topics in Networks, HotNets 2023 ; Conference date: 28-11-2023 Through 29-11-2023",

year = "2023",

month = nov,

day = "28",

doi = "10.1145/3626111.3628186",

language = "English (US)",

series = "HotNets 2023 - Proceedings of the 22nd ACM Workshop on Hot Topics in Networks",

publisher = "Association for Computing Machinery, Inc",

pages = "1--8",

booktitle = "HotNets 2023 - Proceedings of the 22nd ACM Workshop on Hot Topics in Networks",

}

TY - GEN

T1 - Of Apples and Oranges

T2 - 22nd ACM Workshop on Hot Topics in Networks, HotNets 2023

AU - Sadok, Hugo

AU - Panda, Aurojit

AU - Sherry, Justine

PY - 2023/11/28

Y1 - 2023/11/28

N2 - Accelerators, such as GPUs, SmartNICs and FPGAs, are common components of research systems today. This paper focuses on the question of how to fairly compare these systems. This is challenging because it requires comparing systems that use different hardware, e.g., two systems that use two different types of accelerators, or comparing a system that uses an accelerator with one that does not. We argue that fair evaluation in this case requires reporting not just performance, but also the cost of competing systems. We discuss what cost metrics should be used, and propose general principles for incorporating cost in research evaluations.

AB - Accelerators, such as GPUs, SmartNICs and FPGAs, are common components of research systems today. This paper focuses on the question of how to fairly compare these systems. This is challenging because it requires comparing systems that use different hardware, e.g., two systems that use two different types of accelerators, or comparing a system that uses an accelerator with one that does not. We argue that fair evaluation in this case requires reporting not just performance, but also the cost of competing systems. We discuss what cost metrics should be used, and propose general principles for incorporating cost in research evaluations.

KW - Accelerator

KW - Fair Comparison

KW - Heterogenous Hardware

UR - http://www.scopus.com/inward/record.url?scp=85179846215&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=85179846215&partnerID=8YFLogxK

U2 - 10.1145/3626111.3628186

DO - 10.1145/3626111.3628186

M3 - Conference contribution

AN - SCOPUS:85179846215

T3 - HotNets 2023 - Proceedings of the 22nd ACM Workshop on Hot Topics in Networks

SP - 1

EP - 8

BT - HotNets 2023 - Proceedings of the 22nd ACM Workshop on Hot Topics in Networks

PB - Association for Computing Machinery, Inc

Y2 - 28 November 2023 through 29 November 2023

ER -

Of Apples and Oranges: Fair Comparisons in Heterogenous Systems Evaluation

Abstract

Publication series

Conference

Keywords

ASJC Scopus subject areas

Access to Document

Other files and links

Fingerprint

Cite this