# Ceph cluster benchmark result ## Config and result * Deployment on 4 nodes * Node 0 * Storage * Spark slave * Spark master * Ceph mon * Ceph mgr * Ceph mds * Ceph osd * Mount cephfs:/theflash-0 * Node 1 * Storage * Spark slave * Ceph mon * Ceph mgr * Ceph mds * Ceph osd * Mount cephfs:/theflash-1 * Node 2 * Storage * Spark slave * Ceph mon * Ceph mgr * Ceph mds * Ceph osd * Mount cephfs:/theflash-2 * Node 3 * Storage * Spark slave * Ceph osd * Mount cephfs:/theflash-3 * Network between nodes * Ping avg: 0.25ms * Brandwidth: * Node 0 ~ Node 1: 2GB/s * Node 2 ~ Node 3: 2GB/s * (Node 0 or Node 1) ~ (Node 2 or Node 3): 300MB/s - 1GB/s unstable speed * SSDs on each node: * Max: RandWrite: IOPS=6.5k BW=980MiB/s, RandRead: IOPS=56k BW=2650MiB/s * 64K: Write: IOPS=3.5k BW=230MiB/s, Read: IOPS=40k BW=2550MiB/s * Mounted cephfs on each node: * Max: RandWrite: IOPS=3k BW=250MiB/s, RandRead: IOPS=25.8k BW=1150MiB/s * 64K: Write: IOPS=160 BW=10MiB/s, Read: IOPS=14k BW=920MiB/s * Config * TheFlash * Partitions per split: 1 * Storage engine: MutableMergeTree * Storage partitions: 16 * Ceph * cephfs pg: 64 + 64 ## Result | Query | SSD | Ceph | | ----- | ------: | ------: | | Q01 | 11.4 | 52.6 | | Q02 | 23.8 | 30.3 | | Q03 | 31.8 | 41.6 | | Q04 | 104.2 | 245.3 | | Q05 | 65.1 | 74.2 | | Q06 | 5.9 | 7.4 | | Q07 | 46.4 | 46.7 | | Q08 | 71.7 | 76.7 | | Q09 | 80.0 | 100.4 | | Q10 | 27.9 | 28.6 | | Q11 | 12.3 | 13.8 | | Q12 | 19.2 | 21.4 | | Q13 | 25.2 | 25.5 | | Q14 | 11.0 | 11.0 | | Q15 | 11.8 | 11.7 | | Q16 | 45.2 | 54.9 | | Q17 | 80.7 | 79.3 | | Q18 | 96.3 | 97.1 | | Q19 | 15.3 | 16.4 | | Q20 | 42.5 | 40.3 | | Q21 | 641.4 | 737.8 | | Q22 | 46.3 | 97.9 | ## Raw result data * SSD: partitionsPerSplit=1 ``` Q01, avg: 11.4, detail: [10.0, 14.8, 9.8, 10.1, 10.4, 11.9, 12.8, 13.8, 9.7, 11.0] Q02, avg: 23.8, detail: [23.2, 23.8, 24.5, 23.2, 24.0, 24.3, 23.6, 23.8, 24.0, 23.9] Q03, avg: 31.8, detail: [31.1, 31.3, 32.2, 31.3, 30.6, 30.9, 32.6, 34.5, 30.7, 32.7] Q04, avg: 104.2, detail: [90.7, 110.0, 107.6, 108.5, 98.6, 109.0, 110.1, 106.0, 100.9, 100.5] Q05, avg: 65.1, detail: [49.2, 60.7, 71.6, 59.9, 55.7, 78.2, 94.5, 67.3, 61.9, 51.9] Q06, avg: 5.9, detail: [5.8, 5.7, 6.0, 5.9, 5.7, 5.9, 6.1, 5.9, 6.0, 6.0] Q07, avg: 46.4, detail: [40.6, 42.2, 48.1, 46.1, 49.0, 48.6, 46.9, 53.2, 42.5, 46.3] Q08, avg: 71.7, detail: [58.6, 66.3, 79.5, 73.3, 73.0, 93.8, 72.3, 75.0, 59.8, 65.7] Q09, avg: 80.0, detail: [75.3, 72.4, 98.1, 69.9, 82.0, 89.7, 79.0, 79.4, 78.8, 75.5] Q10, avg: 27.9, detail: [25.1, 30.8, 24.6, 30.6, 24.9, 31.0, 26.2, 26.8, 29.4, 29.6] Q11, avg: 12.3, detail: [12.6, 12.4, 12.3, 12.2, 11.9, 11.9, 12.5, 12.3, 12.4, 12.5] Q12, avg: 19.2, detail: [19.5, 19.1, 18.8, 18.7, 19.6, 19.3, 19.2, 20.0, 19.2, 18.7] Q13, avg: 25.2, detail: [24.8, 24.2, 25.9, 25.3, 24.8, 25.7, 25.0, 26.6, 24.8] Q14, avg: 11.0, detail: [10.7, 11.7, 10.9, 11.1, 10.9, 10.8, 11.2, 10.8, 11.1] Q15, avg: 11.8, detail: [11.8, 11.7, 11.8, 11.9, 11.9, 12.1, 11.8, 11.9, 11.6] Q16, avg: 45.2, detail: [69.3, 38.0, 38.8, 42.2, 54.7, 41.0, 38.4, 38.1, 46.2] Q17, avg: 80.7, detail: [78.2, 77.8, 83.9, 80.8, 86.9, 85.0, 77.7, 79.0, 76.8] Q18, avg: 96.3, detail: [100.5, 88.1, 90.7, 110.8, 91.2, 101.3, 90.4, 104.5, 89.5] Q19, avg: 15.3, detail: [14.9, 15.6, 15.2, 15.3, 15.1, 15.1, 15.3, 14.9, 16.5] Q20, avg: 42.5, detail: [45.9, 37.7, 39.8, 39.6, 37.2, 41.8, 38.3, 50.8, 51.1] Q21, avg: 641.4, detail: [624.3, 642.8, 645.2, 623.0, 682.0, 620.2, 678.3, 618.7, 637.8] Q22, avg: 46.3, detail: [47.9, 47.6, 46.1, 47.1, 44.5, 44.1, 42.9, 42.8, 53.9] ``` * Ceph: partitionsPerSplit=1 ``` Q01, avg: 52.6, detail: [51.8, 54.9, 61.4, 47.7, 48.3, 45.7, 42.9, 61.4, 58.9] Q02, avg: 30.3, detail: [23.9, 33.4, 29.9, 31.2, 33.2, 28.2, 32.0, 32.3, 29.6, 29.1] Q03, avg: 41.6, detail: [38.1, 53.3, 39.6, 50.3, 42.1, 41.0, 37.2, 36.1, 43.9, 34.8] Q04, avg: 245.3, detail: [290.0, 236.5, 255.5, 252.1, 261.7, 237.6, 214.2, 225.0, 265.3, 215.3] Q05, avg: 74.2, detail: [137.9, 78.5, 79.5, 64.4, 59.3, 64.8, 64.3, 55.0, 71.5, 66.8] Q06, avg: 7.4, detail: [20.6, 6.0, 5.9, 6.0, 6.1, 6.0, 6.1, 5.9, 5.7, 5.9] Q07, avg: 46.7, detail: [55.9, 53.1, 43.5, 40.7, 43.9, 43.4, 45.9, 45.2, 46.7, 48.7] Q08, avg: 76.7, detail: [120.1, 80.2, 75.6, 73.4, 81.0, 59.7, 61.2, 65.5, 81.2, 69.4] Q09, avg: 100.4, detail: [114.4, 91.9, 110.0, 75.6, 104.1, 98.6, 72.6, 101.1, 113.4, 122.3] Q10, avg: 28.6, detail: [36.0, 29.8, 26.0, 27.2, 30.1, 25.7, 24.4, 27.5, 28.6, 30.9] Q11, avg: 13.8, detail: [13.7, 14.8, 13.4, 14.6, 14.0, 13.9, 12.3, 14.2, 14.0, 13.6] Q12, avg: 21.4, detail: [38.3, 19.6, 19.8, 19.9, 20.0, 19.4, 19.8, 19.1, 19.2, 19.4] Q13, avg: 25.5, detail: [29.9, 25.1, 24.4, 24.9, 24.8, 24.9, 24.0, 25.3, 24.4, 27.0] Q14, avg: 11.0, detail: [11.2, 10.9, 10.9, 11.2, 10.8, 10.9, 11.3, 11.2, 10.6, 10.7] Q15, avg: 11.7, detail: [11.7, 12.0, 11.6, 11.8, 11.9, 11.7, 11.5, 11.6, 11.6, 11.5] Q16, avg: 54.9, detail: [51.8, 56.6, 76.2, 56.7, 56.5, 49.6, 45.6, 54.0, 50.1, 52.1] Q17, avg: 79.3, detail: [81.4, 78.8, 80.4, 79.2, 78.0, 77.1, 77.7, 79.3, 83.0, 77.7] Q18, avg: 97.1, detail: [104.0, 94.0, 87.2, 118.7, 100.4, 91.2, 89.5, 89.1, 104.2, 92.6] Q19, avg: 16.4, detail: [23.9, 15.9, 15.3, 16.4, 15.0, 15.2, 15.0, 16.3, 15.6, 14.9] Q20, avg: 40.3, detail: [43.2, 38.3, 44.1, 37.4, 43.3, 37.2, 38.8, 37.1, 46.1, 37.3] Q21, avg: 737.8, detail: [959.3, 763.9, 808.4, 682.2, 708.4, 661.2, 672.3, 704.9, 717.4, 699.9] Q22, avg: 97.9, detail: [99.8, 90.7, 103.5, 102.8, 92.5, 101.3, 90.8, 109.0, 90.9, 97.5] ``` ## Environment ### Hardware * Each node: * CPU: 16 Cores, Intel Core Processor (Haswell, IBRS) @ 2399.996 MHz * Memory: 56G * SSD `*` 2 (one for ceph): 200MB/s on write, 450MB/s on read ### Software * Centos-release-7-5 * Spark 2.1.1 * TheFlash 85932438a30bfcdb53f5c7153f63a6c87cf0968c ### Data Set * TPCH-100 100G data scala