{"payload":{"feedbackUrl":"https://github.com/orgs/community/discussions/53140","repo":{"id":29795757,"defaultBranch":"master","name":"spark","ownerLogin":"sunchao","currentUserCanPush":false,"isFork":true,"isEmpty":false,"createdAt":"2015-01-24T23:30:38.000Z","ownerAvatar":"https://avatars.githubusercontent.com/u/506679?v=4","public":true,"private":false,"isOrgOwned":false},"refInfo":{"name":"","listCacheKey":"v0:1716417459.0","currentOid":""},"activityList":{"items":[{"before":null,"after":"6c5861b55db604920fcdd447076c831be3c9f7da","ref":"refs/heads/SPARK-48392","pushedAt":"2024-05-22T22:37:39.000Z","pushType":"branch_creation","commitsCount":0,"pusher":{"login":"sunchao","name":"Chao Sun","path":"/sunchao","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/506679?s=80&v=4"},"commit":{"message":"initial commit","shortMessageHtmlLink":"initial commit"}},{"before":null,"after":"f7c15aa4a9c0462f17a8994102a1c707b07629d6","ref":"refs/heads/3.4.2-diff","pushedAt":"2024-03-07T20:26:55.000Z","pushType":"branch_creation","commitsCount":0,"pusher":{"login":"sunchao","name":"Chao Sun","path":"/sunchao","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/506679?s=80&v=4"},"commit":{"message":"Apply Comet diff for Spark 3.4.2","shortMessageHtmlLink":"Apply Comet diff for Spark 3.4.2"}},{"before":"0f41c4bb064d6816fc3e635eb8f57a5883fe108a","after":"00effe76528f7b668f730dcce282f5f6d27e6256","ref":"refs/heads/SPARK-46947","pushedAt":"2024-02-26T00:53:48.000Z","pushType":"push","commitsCount":1,"pusher":{"login":"sunchao","name":"Chao Sun","path":"/sunchao","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/506679?s=80&v=4"},"commit":{"message":"comments","shortMessageHtmlLink":"comments"}},{"before":"af68ece323401d06f7e036c3b0782bc80a0f4c93","after":"0f41c4bb064d6816fc3e635eb8f57a5883fe108a","ref":"refs/heads/SPARK-46947","pushedAt":"2024-02-26T00:51:42.000Z","pushType":"push","commitsCount":2,"pusher":{"login":"sunchao","name":"Chao Sun","path":"/sunchao","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/506679?s=80&v=4"},"commit":{"message":"cleanup block manager test","shortMessageHtmlLink":"cleanup block manager test"}},{"before":"a980e0716ade545f69fd68e38d1017173bacc622","after":"af68ece323401d06f7e036c3b0782bc80a0f4c93","ref":"refs/heads/SPARK-46947","pushedAt":"2024-02-24T00:32:15.000Z","pushType":"push","commitsCount":1,"pusher":{"login":"sunchao","name":"Chao Sun","path":"/sunchao","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/506679?s=80&v=4"},"commit":{"message":"try use BlockManager","shortMessageHtmlLink":"try use BlockManager"}},{"before":"13f89142176f6418af3660f3b92b5093ca07ce51","after":"a980e0716ade545f69fd68e38d1017173bacc622","ref":"refs/heads/SPARK-46947","pushedAt":"2024-02-23T21:57:47.000Z","pushType":"force_push","commitsCount":0,"pusher":{"login":"sunchao","name":"Chao Sun","path":"/sunchao","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/506679?s=80&v=4"},"commit":{"message":"clean up","shortMessageHtmlLink":"clean up"}},{"before":"0022fcb911a878126e659a9823d73c61605d377e","after":"13f89142176f6418af3660f3b92b5093ca07ce51","ref":"refs/heads/SPARK-46947","pushedAt":"2024-02-20T19:06:24.000Z","pushType":"push","commitsCount":0,"pusher":{"login":"sunchao","name":"Chao Sun","path":"/sunchao","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/506679?s=80&v=4"}},{"before":"3e201f32a83ba7f8009c595f992c94180c774107","after":"0022fcb911a878126e659a9823d73c61605d377e","ref":"refs/heads/SPARK-46947","pushedAt":"2024-02-16T22:28:53.000Z","pushType":"force_push","commitsCount":0,"pusher":{"login":"sunchao","name":"Chao Sun","path":"/sunchao","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/506679?s=80&v=4"}},{"before":"a6502e06ef2bf68654f1d3c8bb5c483a071d8770","after":"3e201f32a83ba7f8009c595f992c94180c774107","ref":"refs/heads/SPARK-46947","pushedAt":"2024-02-16T19:50:07.000Z","pushType":"push","commitsCount":0,"pusher":{"login":"sunchao","name":"Chao Sun","path":"/sunchao","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/506679?s=80&v=4"}},{"before":"b7f790cd6c63b37eb9ae99345ad0dd9861013ea0","after":"a6502e06ef2bf68654f1d3c8bb5c483a071d8770","ref":"refs/heads/SPARK-46947","pushedAt":"2024-02-15T23:12:04.000Z","pushType":"push","commitsCount":1,"pusher":{"login":"sunchao","name":"Chao Sun","path":"/sunchao","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/506679?s=80&v=4"},"commit":{"message":"stop SparkContext","shortMessageHtmlLink":"stop SparkContext"}},{"before":"638bb65ffc836b5b64d3de5b8df7cb269b9e8f2c","after":"b7f790cd6c63b37eb9ae99345ad0dd9861013ea0","ref":"refs/heads/SPARK-46947","pushedAt":"2024-02-15T20:59:17.000Z","pushType":"push","commitsCount":2,"pusher":{"login":"sunchao","name":"Chao Sun","path":"/sunchao","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/506679?s=80&v=4"},"commit":{"message":"add tests","shortMessageHtmlLink":"add tests"}},{"before":"1fd880700afaedefacae043d7eaa12041c4f91ff","after":"638bb65ffc836b5b64d3de5b8df7cb269b9e8f2c","ref":"refs/heads/SPARK-46947","pushedAt":"2024-02-15T07:41:01.000Z","pushType":"push","commitsCount":1,"pusher":{"login":"sunchao","name":"Chao Sun","path":"/sunchao","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/506679?s=80&v=4"},"commit":{"message":"fix","shortMessageHtmlLink":"fix"}},{"before":"48b40886c2abfdf38855f371f760719cea9afbb7","after":"1fd880700afaedefacae043d7eaa12041c4f91ff","ref":"refs/heads/SPARK-46947","pushedAt":"2024-02-07T16:47:31.000Z","pushType":"force_push","commitsCount":0,"pusher":{"login":"sunchao","name":"Chao Sun","path":"/sunchao","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/506679?s=80&v=4"},"commit":{"message":"fix","shortMessageHtmlLink":"fix"}},{"before":"23c66afe8a26a3ac4ca556da53bc97479ad77072","after":"48b40886c2abfdf38855f371f760719cea9afbb7","ref":"refs/heads/SPARK-46947","pushedAt":"2024-02-07T08:57:41.000Z","pushType":"push","commitsCount":0,"pusher":{"login":"sunchao","name":"Chao Sun","path":"/sunchao","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/506679?s=80&v=4"}},{"before":"b38a321b02f408e28b3ae1cb9f1eece4b6700b80","after":"23c66afe8a26a3ac4ca556da53bc97479ad77072","ref":"refs/heads/SPARK-46947","pushedAt":"2024-02-07T06:05:58.000Z","pushType":"push","commitsCount":1,"pusher":{"login":"sunchao","name":"Chao Sun","path":"/sunchao","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/506679?s=80&v=4"},"commit":{"message":"update","shortMessageHtmlLink":"update"}},{"before":null,"after":"b38a321b02f408e28b3ae1cb9f1eece4b6700b80","ref":"refs/heads/SPARK-46947","pushedAt":"2024-02-06T23:33:27.000Z","pushType":"branch_creation","commitsCount":0,"pusher":{"login":"sunchao","name":"Chao Sun","path":"/sunchao","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/506679?s=80&v=4"},"commit":{"message":"initial commit","shortMessageHtmlLink":"initial commit"}},{"before":"6dab45bcd2ea4f66661fbdfdf4eaa544561b17e5","after":null,"ref":"refs/heads/SPARK-45731","pushedAt":"2023-11-09T23:57:16.000Z","pushType":"branch_deletion","commitsCount":0,"pusher":{"login":"sunchao","name":"Chao Sun","path":"/sunchao","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/506679?s=80&v=4"}},{"before":"c31f21d106ddf1706b0e313e9744b3acacd7047f","after":"6dab45bcd2ea4f66661fbdfdf4eaa544561b17e5","ref":"refs/heads/SPARK-45731","pushedAt":"2023-11-09T21:04:03.000Z","pushType":"push","commitsCount":1,"pusher":{"login":"sunchao","name":"Chao Sun","path":"/sunchao","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/506679?s=80&v=4"},"commit":{"message":"fix indentation","shortMessageHtmlLink":"fix indentation"}},{"before":"9e2d874769fd5380316b07fdd9ba5c51182f847f","after":"c31f21d106ddf1706b0e313e9744b3acacd7047f","ref":"refs/heads/SPARK-45731","pushedAt":"2023-11-09T16:49:48.000Z","pushType":"push","commitsCount":1,"pusher":{"login":"sunchao","name":"Chao Sun","path":"/sunchao","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/506679?s=80&v=4"},"commit":{"message":"fix bigint issue","shortMessageHtmlLink":"fix bigint issue"}},{"before":"e623a332479a34639430f76927b913220d7b5ca7","after":"9e2d874769fd5380316b07fdd9ba5c51182f847f","ref":"refs/heads/SPARK-45731","pushedAt":"2023-11-09T08:41:55.000Z","pushType":"push","commitsCount":1,"pusher":{"login":"sunchao","name":"Chao Sun","path":"/sunchao","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/506679?s=80&v=4"},"commit":{"message":"fix test","shortMessageHtmlLink":"fix test"}},{"before":"1b6b0a581f4fdfa8f8988427adc0f31eff70f3d9","after":"e623a332479a34639430f76927b913220d7b5ca7","ref":"refs/heads/SPARK-45731","pushedAt":"2023-11-09T05:34:34.000Z","pushType":"push","commitsCount":1,"pusher":{"login":"sunchao","name":"Chao Sun","path":"/sunchao","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/506679?s=80&v=4"},"commit":{"message":"update test","shortMessageHtmlLink":"update test"}},{"before":"5c1ba710d4e1d51829e31f79a7eece0dacec7c46","after":"1b6b0a581f4fdfa8f8988427adc0f31eff70f3d9","ref":"refs/heads/SPARK-45731","pushedAt":"2023-11-09T05:29:33.000Z","pushType":"push","commitsCount":1,"pusher":{"login":"sunchao","name":"Chao Sun","path":"/sunchao","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/506679?s=80&v=4"},"commit":{"message":"address comments","shortMessageHtmlLink":"address comments"}},{"before":"9768527ec82ae0957ee029ea3184dd4a2ad8ebbf","after":"5c1ba710d4e1d51829e31f79a7eece0dacec7c46","ref":"refs/heads/SPARK-45731","pushedAt":"2023-11-09T00:54:50.000Z","pushType":"push","commitsCount":1,"pusher":{"login":"sunchao","name":"Chao Sun","path":"/sunchao","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/506679?s=80&v=4"},"commit":{"message":"update config name","shortMessageHtmlLink":"update config name"}},{"before":"ed1f3a8b607db327032d48484314a2be5f528b47","after":"9768527ec82ae0957ee029ea3184dd4a2ad8ebbf","ref":"refs/heads/SPARK-45731","pushedAt":"2023-11-08T23:53:01.000Z","pushType":"force_push","commitsCount":0,"pusher":{"login":"sunchao","name":"Chao Sun","path":"/sunchao","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/506679?s=80&v=4"},"commit":{"message":"introduce a config","shortMessageHtmlLink":"introduce a config"}},{"before":"b16f72523f2140802ecd93ccabe688e72faefd97","after":"ed1f3a8b607db327032d48484314a2be5f528b47","ref":"refs/heads/SPARK-45731","pushedAt":"2023-11-08T23:49:39.000Z","pushType":"push","commitsCount":0,"pusher":{"login":"sunchao","name":"Chao Sun","path":"/sunchao","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/506679?s=80&v=4"}},{"before":"7ae76c11bf2eaa825d604ca1e894f5a1e147ff0a","after":"b16f72523f2140802ecd93ccabe688e72faefd97","ref":"refs/heads/SPARK-45731","pushedAt":"2023-11-01T20:08:10.000Z","pushType":"push","commitsCount":2,"pusher":{"login":"sunchao","name":"Chao Sun","path":"/sunchao","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/506679?s=80&v=4"},"commit":{"message":"remove duplicated code","shortMessageHtmlLink":"remove duplicated code"}},{"before":null,"after":"7ae76c11bf2eaa825d604ca1e894f5a1e147ff0a","ref":"refs/heads/SPARK-45731","pushedAt":"2023-11-01T18:04:43.000Z","pushType":"branch_creation","commitsCount":0,"pusher":{"login":"sunchao","name":"Chao Sun","path":"/sunchao","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/506679?s=80&v=4"},"commit":{"message":"initial commit","shortMessageHtmlLink":"initial commit"}},{"before":null,"after":"63e3f0323601760718fa86cdb2d6e08f700a3d92","ref":"refs/heads/SPARK-45652-branch-3.5","pushedAt":"2023-10-26T16:15:23.000Z","pushType":"branch_creation","commitsCount":0,"pusher":{"login":"sunchao","name":"Chao Sun","path":"/sunchao","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/506679?s=80&v=4"},"commit":{"message":"[SPARK-45652][SQL] SPJ: Handle empty input partitions after dynamic filtering\n\nHandle the case when input partitions become empty after V2 dynamic filtering, when SPJ is enabled.\n\nCurrent in the situation when all input partitions are filtered out via dynamic filtering, SPJ doesn't work but instead will panic:\n```\njava.util.NoSuchElementException: None.get\n\tat scala.None$.get(Option.scala:529)\n\tat scala.None$.get(Option.scala:527)\n\tat org.apache.spark.sql.execution.datasources.v2.BatchScanExec.filteredPartitions$lzycompute(BatchScanExec.scala:108)\n\tat org.apache.spark.sql.execution.datasources.v2.BatchScanExec.filteredPartitions(BatchScanExec.scala:65)\n\tat org.apache.spark.sql.execution.datasources.v2.BatchScanExec.inputRDD$lzycompute(BatchScanExec.scala:136)\n\tat org.apache.spark.sql.execution.datasources.v2.BatchScanExec.inputRDD(BatchScanExec.scala:135)\n\tat org.apache.spark.sql.boson.BosonBatchScanExec.inputRDD$lzycompute(BosonBatchScanExec.scala:28)\n```\n\nThis is because the `groupPartitions` method will return `None` in this scenario. We should handle the case.\n\nNo\n\nAdded a test case for this.\n\nNo\n\nCloses #43531 from sunchao/SPARK-45652.\n\nAuthored-by: Chao Sun <sunchao@apple.com>\nSigned-off-by: Chao Sun <sunchao@apple.com>","shortMessageHtmlLink":"[SPARK-45652][SQL] SPJ: Handle empty input partitions after dynamic f…"}},{"before":null,"after":"7fe82f20310f5a5f1ace0bb7ad9a0fac6e2abe5e","ref":"refs/heads/SPARK-45652-branch-3.4","pushedAt":"2023-10-26T16:02:47.000Z","pushType":"branch_creation","commitsCount":0,"pusher":{"login":"sunchao","name":"Chao Sun","path":"/sunchao","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/506679?s=80&v=4"},"commit":{"message":"[SPARK-45652][SQL] SPJ: Handle empty input partitions after dynamic filtering\n\nHandle the case when input partitions become empty after V2 dynamic filtering, when SPJ is enabled.\n\nCurrent in the situation when all input partitions are filtered out via dynamic filtering, SPJ doesn't work but instead will panic:\n```\njava.util.NoSuchElementException: None.get\n\tat scala.None$.get(Option.scala:529)\n\tat scala.None$.get(Option.scala:527)\n\tat org.apache.spark.sql.execution.datasources.v2.BatchScanExec.filteredPartitions$lzycompute(BatchScanExec.scala:108)\n\tat org.apache.spark.sql.execution.datasources.v2.BatchScanExec.filteredPartitions(BatchScanExec.scala:65)\n\tat org.apache.spark.sql.execution.datasources.v2.BatchScanExec.inputRDD$lzycompute(BatchScanExec.scala:136)\n\tat org.apache.spark.sql.execution.datasources.v2.BatchScanExec.inputRDD(BatchScanExec.scala:135)\n\tat org.apache.spark.sql.boson.BosonBatchScanExec.inputRDD$lzycompute(BosonBatchScanExec.scala:28)\n```\n\nThis is because the `groupPartitions` method will return `None` in this scenario. We should handle the case.\n\nNo\n\nAdded a test case for this.\n\nNo\n\nCloses #43531 from sunchao/SPARK-45652.\n\nAuthored-by: Chao Sun <sunchao@apple.com>\nSigned-off-by: Chao Sun <sunchao@apple.com>","shortMessageHtmlLink":"[SPARK-45652][SQL] SPJ: Handle empty input partitions after dynamic f…"}},{"before":null,"after":"7f27791c6e56fc9c651c114e2042d938d2d9729e","ref":"refs/heads/SPARK-45652","pushedAt":"2023-10-25T21:49:53.000Z","pushType":"branch_creation","commitsCount":0,"pusher":{"login":"sunchao","name":"Chao Sun","path":"/sunchao","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/506679?s=80&v=4"},"commit":{"message":"SPARK-45652: SPJ: Handle empty input partitions after dynamic filtering","shortMessageHtmlLink":"SPARK-45652: SPJ: Handle empty input partitions after dynamic filtering"}}],"hasNextPage":true,"hasPreviousPage":false,"activityType":"all","actor":null,"timePeriod":"all","sort":"DESC","perPage":30,"cursor":"djE6ks8AAAAEUW5QxAA","startCursor":null,"endCursor":null}},"title":"Activity · sunchao/spark"}