# TheFlash TPCH on 1 node, use idempotent-read, different RDD partition count ## Config and result * All * MutableMergeTree engine * Result columns * CH-0001: idempotent-read, partitionsPerSplit=16 * CH-0002: idempotent-read, partitionsPerSplit=8 * CH-0003: idempotent-read, partitionsPerSplit=4 * CH-0004: idempotent-read, partitionsPerSplit=2 * CH-0005: idempotent-read, partitionsPerSplit=1 * Mutable: session-read, partitions=16 | Query | CH-0001 | CH-0002 | CH-0003 | CH-0004 | CH-0005 | Mutable | | ----- | ------: | ------: | ------: | ------: | ------: | ------: | | Q01 | 23.7 | 24.8 | 25.3 | 25.1 | 25.9 | 22.3 | | Q02 | 70.2 | 53.5 | 47.8 | 43.9 | 45.1 | 51.3 | | Q03 | 192.5 | 122.0 | 88.8 | 69.0 | 66.3 | 86.7 | | Q04 | 753.8 | 456.2 | 303.1 | 232.4 | 222.4 | 344.3 | | Q05 | 348.4 | 218.7 | 149.4 | 122.0 | 118.1 | 140.1 | | Q06 | 12.4 | 13.0 | 13.9 | 14.3 | 14.2 | 12.2 | | Q07 | 204.1 | 137.6 | 105.2 | 93.6 | 94.0 | 114.3 | | Q08 | 391.8 | 244.2 | 166.2 | 126.1 | 118.2 | 146.9 | | Q09 | 442.8 | 280.5 | 209.9 | 167.5 | 158.8 | 191.0 | | Q10 | 101.6 | 78.2 | 60.3 | 56.3 | 56.4 | 79.7 | | Q11 | 31.0 | 24.4 | 20.2 | 19.0 | 18.3 | 25.2 | | Q12 | 97.3 | 64.7 | 49.5 | 43.8 | 55.8 | 64.4 | | Q13 | 303.3 | 169.7 | 99.3 | 74.1 | 56.7 | 66.0 | | Q14 | 27.6 | 21.9 | 21.3 | 24.5 | 23.7 | 35.4 | | Q16 | 1434.0 | 582.7 | 441.4 | 1345.6 | 150.7 | | | Q19 | 40.6 | 37.9 | 41.7 | 40.5 | 39.6 | 66.4 | | Q22 | 243.8 | 159.9 | 107.8 | 77.6 | 70.6 | 102.6 | ## Raw result data * CH-0001: partitionsPerSplit=16 ``` Q01, avg: 23.7, detail: [23.7, 23.0, 24.5, 23.6, 23.6] Q02, avg: 70.2, detail: [69.5, 69.8, 73.6, 69.5, 68.5] Q03, avg: 192.5, detail: [201.3, 203.7, 200.0, 182.9, 174.5] Q04, avg: 753.8, detail: [766.3, 759.9, 734.9, 751.5, 756.3] Q05, avg: 348.4, detail: [365.6, 344.8, 345.4, 347.1, 338.9] Q06, avg: 12.4, detail: [12.3, 13.0, 12.4, 11.8, 12.3] Q07, avg: 204.1, detail: [200.2, 201.0, 211.3, 195.3, 212.5] Q08, avg: 391.8, detail: [392.9, 395.9, 387.4, 396.1, 386.8] Q09, avg: 442.8, detail: [455.6, 444.0, 447.3, 430.8, 436.3] Q10, avg: 101.6, detail: [104.6, 102.5, 92.7, 104.3, 104.1] Q11, avg: 31.0, detail: [31.9, 31.2, 28.9, 32.8, 30.3] Q12, avg: 97.3, detail: [96.9, 91.7, 104.2, 95.6, 98.3] Q13, avg: 303.3, detail: [302.7, 304.3, 302.6, 303.3, 303.5] Q14, avg: 27.6, detail: [25.5, 28.0, 28.5, 29.6, 26.2] Q16, avg: 1434.0, detail: [799.5, 782.3, 1869.7, 1860.2, 1858.5] Q19, avg: 40.6, detail: [41.0, 32.4, 43.2, 45.6, 41.0] Q22, avg: 243.8, detail: [247.4, 244.4, 243.9, 237.9, 245.6] ``` * CH-0002: partitionsPerSplit=8 ``` Q01, avg: 24.8, detail: [26.6, 25.3, 23.3, 23.5, 25.3] Q02, avg: 53.5, detail: [53.9, 51.9, 58.1, 50.4, 53.1] Q03, avg: 122.0, detail: [122.2, 126.9, 112.4, 122.0, 126.6] Q04, avg: 456.2, detail: [452.6, 453.5, 450.8, 453.8, 470.3] Q05, avg: 218.7, detail: [224.5, 227.9, 209.5, 219.4, 212.4] Q06, avg: 13.0, detail: [12.7, 14.0, 12.7, 13.0, 12.6] Q07, avg: 137.6, detail: [133.1, 139.0, 132.5, 144.7, 138.6] Q08, avg: 244.2, detail: [243.5, 242.1, 255.0, 238.8, 241.8] Q09, avg: 280.5, detail: [274.6, 276.9, 283.9, 286.8, 280.3] Q10, avg: 78.2, detail: [86.5, 87.8, 66.8, 76.4, 73.3] Q11, avg: 24.4, detail: [24.4, 24.4, 24.7, 24.5, 23.8] Q12, avg: 64.7, detail: [63.8, 64.0, 63.2, 67.2, 65.3] Q13, avg: 169.7, detail: [163.6, 168.9, 184.1, 168.6, 163.1] Q14, avg: 21.9, detail: [22.0, 20.9, 22.4, 22.0, 22.0] Q16, avg: 582.7, detail: [537.7, 567.9, 962.3, 443.4, 402.3] Q19, avg: 37.9, detail: [38.7, 37.5, 38.6, 32.9, 42.0] Q22, avg: 159.9, detail: [166.0, 147.7, 146.9, 156.2, 182.5] ``` * CH-0003: partitionsPerSplit=4 ``` Q01, avg: 25.3, detail: [25.6, 25.4, 21.9, 27.1, 26.3] Q02, avg: 47.8, detail: [45.9, 47.2, 48.4, 48.8, 48.5] Q03, avg: 88.8, detail: [86.4, 86.2, 82.3, 88.0, 101.3] Q04, avg: 303.1, detail: [306.4, 305.1, 305.4, 295.9, 302.7] Q05, avg: 149.4, detail: [149.0, 151.7, 151.5, 146.3, 148.6] Q06, avg: 13.9, detail: [14.2, 12.3, 15.2, 14.0, 13.7] Q07, avg: 105.2, detail: [106.7, 101.5, 105.3, 102.0, 110.6] Q08, avg: 166.2, detail: [162.7, 166.0, 162.5, 163.1, 176.9] Q09, avg: 209.9, detail: [223.2, 199.1, 217.9, 200.9, 208.2] Q10, avg: 60.3, detail: [61.8, 62.0, 60.2, 57.4, 60.3] Q11, avg: 20.2, detail: [21.2, 19.0, 19.6, 21.8, 19.6] Q12, avg: 49.5, detail: [50.2, 44.6, 49.7, 53.2, 49.8] Q13, avg: 99.3, detail: [98.2, 97.8, 102.3, 99.8, 98.2] Q14, avg: 21.3, detail: [22.6, 19.5, 22.0, 20.5, 21.8] Q16, avg: 441.4, detail: [294.6, 651.0, 259.1, 450.8, 551.3] Q19, avg: 41.7, detail: [46.6, 40.3, 37.3, 39.5, 45.0] Q22, avg: 107.8, detail: [98.2, 107.6, 121.5, 117.3, 94.5] ``` * CH-0004: partitionsPerSplit=2 ``` Q01, avg: 25.1, detail: [24.8, 25.0, 25.1, 23.8, 26.9] Q02, avg: 43.9, detail: [43.9, 43.7, 43.2, 44.8, 44.0] Q03, avg: 69.0, detail: [65.4, 65.6, 66.9, 81.2, 65.7] Q04, avg: 232.4, detail: [230.0, 230.2, 233.8, 236.9, 231.1] Q05, avg: 122.0, detail: [121.1, 123.0, 119.4, 123.7, 122.8] Q06, avg: 14.3, detail: [14.9, 14.4, 14.9, 14.7, 12.7] Q07, avg: 93.6, detail: [91.2, 99.9, 93.3, 92.4, 91.4] Q08, avg: 126.1, detail: [129.6, 123.8, 124.0, 131.1, 121.8] Q09, avg: 167.5, detail: [160.4, 178.0, 158.7, 173.4, 167.2] Q10, avg: 56.3, detail: [54.9, 56.9, 57.2, 53.4, 59.0] Q11, avg: 19.0, detail: [20.0, 19.4, 18.7, 18.6, 18.1] Q12, avg: 43.8, detail: [47.4, 43.5, 45.1, 42.0, 41.0] Q13, avg: 74.1, detail: [76.5, 68.4, 79.0, 64.8, 81.9] Q14, avg: 24.5, detail: [22.6, 20.3, 25.8, 22.2, 31.4] Q16, avg: 1345.6, detail: [2558.4, 536.7, 1780.8, 978.7, 873.5] Q19, avg: 40.5, detail: [42.8, 39.5, 42.8, 35.3, 42.3] Q22, avg: 77.6, detail: [72.8, 72.0, 94.2, 75.1, 74.0] ``` * CH-0005: partitionsPerSplit=1 ``` Q01, avg: 25.9, detail: [27.2, 26.2, 26.1, 23.7, 26.4] Q02, avg: 45.1, detail: [44.6, 42.7, 46.5, 45.9, 45.6] Q03, avg: 66.3, detail: [64.7, 71.9, 67.0, 63.8, 64.0] Q04, avg: 222.4, detail: [240.6, 216.4, 219.4, 219.2, 216.5] Q05, avg: 118.1, detail: [113.0, 139.4, 110.3, 111.1, 116.6] Q06, avg: 14.2, detail: [13.3, 14.5, 15.0, 13.1, 15.0] Q07, avg: 94.0, detail: [94.9, 97.1, 94.5, 91.9, 91.8] Q08, avg: 118.2, detail: [117.8, 114.7, 115.9, 117.4, 125.0] Q09, avg: 158.8, detail: [156.1, 158.9, 162.6, 160.2, 156.2] Q10, avg: 56.4, detail: [58.1, 54.3, 56.9, 54.2, 58.3] Q11, avg: 18.3, detail: [18.6, 18.9, 17.4, 19.0, 17.5] Q12, avg: 55.8, detail: [54.1, 75.7, 44.2, 43.3, 61.8] Q13, avg: 56.7, detail: [53.1, 57.2, 53.5, 60.2, 59.5] Q14, avg: 23.7, detail: [22.4, 23.5, 24.3, 22.5, 25.9] Q16, avg: 150.7, detail: [173.3, 133.7, 87.5, 134.4, 224.4] Q19, avg: 39.6, detail: [43.3, 34.6, 52.5, 33.4, 34.1] Q22, avg: 70.6, detail: [77.3, 68.0, 68.2, 67.9, 71.5] ``` ## Environment ### Hardware * Intel Xeon E312xx 16 cores 2.4Ghz. * 16G Memory 1600MHz . * SSD 2.0GB+/s. ### Software * CentOS Linux release 7.2.1511 * Spark version 2.1.1 * TheFlash git hash be63fa911b1b4a1efd473a6670f0f0dec3f28a11 ### Data Set * TPCH-100 100G data scala