{"payload":{"feedbackUrl":"https://github.com/orgs/community/discussions/53140","repo":{"id":686994048,"defaultBranch":"main","name":"airflow","ownerLogin":"zeotuan","currentUserCanPush":false,"isFork":true,"isEmpty":false,"createdAt":"2023-09-04T11:39:43.000Z","ownerAvatar":"https://avatars.githubusercontent.com/u/48720253?v=4","public":true,"private":false,"isOrgOwned":false},"refInfo":{"name":"","listCacheKey":"v0:1698833763.0","currentOid":""},"activityList":{"items":[{"before":"3724a029dadf4678d1cc89049b247f6d2bc233e2","after":"48c98bc1d4371a990bfadbfdd1478254e0098fa6","ref":"refs/heads/main","pushedAt":"2024-04-27T07:06:47.000Z","pushType":"push","commitsCount":2340,"pusher":{"login":"zeotuan","name":null,"path":"/zeotuan","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/48720253?s=80&v=4"},"commit":{"message":"Update Hashicorp AWS assume role auth docs (#39287)","shortMessageHtmlLink":"Update Hashicorp AWS assume role auth docs (apache#39287)"}},{"before":"3339470650ee60259c06db6f71e9d90d57aa6cd6","after":"aa16843c93a46f568f9586ad3729104cc359b6c5","ref":"refs/heads/tuan/beam-xcom","pushedAt":"2023-11-01T10:16:52.000Z","pushType":"force_push","commitsCount":0,"pusher":{"login":"zeotuan","name":null,"path":"/zeotuan","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/48720253?s=80&v=4"},"commit":{"message":"push_xcom right before starting pipeline","shortMessageHtmlLink":"push_xcom right before starting pipeline"}},{"before":"c3a42197d67261d0370af21b19b1efacf1f76510","after":null,"ref":"refs/heads/tuan/spark-op","pushedAt":"2023-11-01T10:16:03.000Z","pushType":"branch_deletion","commitsCount":0,"pusher":{"login":"zeotuan","name":null,"path":"/zeotuan","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/48720253?s=80&v=4"}},{"before":"6f3d294645153db914be69cd2b2a49f12a18280c","after":"3724a029dadf4678d1cc89049b247f6d2bc233e2","ref":"refs/heads/main","pushedAt":"2023-11-01T10:15:04.000Z","pushType":"push","commitsCount":65,"pusher":{"login":"zeotuan","name":null,"path":"/zeotuan","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/48720253?s=80&v=4"},"commit":{"message":"Improve clear_not_launched_queued_tasks call duration (#34985)\n\n* Improve clear_not_launched_queued_tasks call duration\r\n\r\n* Apply suggestions from code review\r\n\r\nCo-authored-by: Tzu-ping Chung \r\n\r\n---------\r\n\r\nCo-authored-by: gopal \r\nCo-authored-by: Tzu-ping Chung \r\nCo-authored-by: Hussein Awala \r\nCo-authored-by: Elad Kalif <45845474+eladkal@users.noreply.github.com>","shortMessageHtmlLink":"Improve clear_not_launched_queued_tasks call duration (apache#34985)"}},{"before":"1f38d6c2d9c54b988d62edcfcd76778bc1a36e2a","after":"c3a42197d67261d0370af21b19b1efacf1f76510","ref":"refs/heads/tuan/spark-op","pushedAt":"2023-11-01T04:44:42.000Z","pushType":"push","commitsCount":49,"pusher":{"login":"zeotuan","name":null,"path":"/zeotuan","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/48720253?s=80&v=4"},"commit":{"message":"Merge branch 'apache:main' into tuan/spark-op","shortMessageHtmlLink":"Merge branch 'apache:main' into tuan/spark-op"}},{"before":"7ee8dfa373dd882b2872636fd5dac2202d06b6b0","after":"1f38d6c2d9c54b988d62edcfcd76778bc1a36e2a","ref":"refs/heads/tuan/spark-op","pushedAt":"2023-10-31T22:59:33.000Z","pushType":"push","commitsCount":1,"pusher":{"login":"zeotuan","name":null,"path":"/zeotuan","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/48720253?s=80&v=4"},"commit":{"message":"test use_krb5ccache on sparkjdbc operator","shortMessageHtmlLink":"test use_krb5ccache on sparkjdbc operator"}},{"before":"da25da9b2e25b08e32da22bdb52bf79b784e3eec","after":"7ee8dfa373dd882b2872636fd5dac2202d06b6b0","ref":"refs/heads/tuan/spark-op","pushedAt":"2023-10-31T11:03:36.000Z","pushType":"push","commitsCount":1,"pusher":{"login":"zeotuan","name":null,"path":"/zeotuan","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/48720253?s=80&v=4"},"commit":{"message":"add use_krb5ccache to spark submit operators","shortMessageHtmlLink":"add use_krb5ccache to spark submit operators"}},{"before":"988d42b492b69f63d67b6165a7fadad8f8575265","after":"da25da9b2e25b08e32da22bdb52bf79b784e3eec","ref":"refs/heads/tuan/spark-op","pushedAt":"2023-10-28T05:45:38.000Z","pushType":"push","commitsCount":1,"pusher":{"login":"zeotuan","name":null,"path":"/zeotuan","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/48720253?s=80&v=4"},"commit":{"message":"add use_krb5ccache to operator test","shortMessageHtmlLink":"add use_krb5ccache to operator test"}},{"before":"6f3d294645153db914be69cd2b2a49f12a18280c","after":"988d42b492b69f63d67b6165a7fadad8f8575265","ref":"refs/heads/tuan/spark-op","pushedAt":"2023-10-28T03:24:21.000Z","pushType":"push","commitsCount":1,"pusher":{"login":"zeotuan","name":null,"path":"/zeotuan","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/48720253?s=80&v=4"},"commit":{"message":"Add use_krb5ccach option to sparkSubmitOperator","shortMessageHtmlLink":"Add use_krb5ccach option to sparkSubmitOperator"}},{"before":null,"after":"6f3d294645153db914be69cd2b2a49f12a18280c","ref":"refs/heads/tuan/spark-op","pushedAt":"2023-10-28T03:15:15.000Z","pushType":"branch_creation","commitsCount":0,"pusher":{"login":"zeotuan","name":null,"path":"/zeotuan","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/48720253?s=80&v=4"},"commit":{"message":"Fix subtle bug in mocking processor_agent in our tests (#35221)\n\nSome of the scheduler tests tried to prevent DAG processor processing DAGs\r\nfrom \"tests/dags\" directory by setting processor_agent to Mock object:\r\n\r\n```python\r\n self.job_runner.processor_agent = mock.MagicMock()\r\n```\r\n\r\nThis, in connection with scheduler job cleaning all the tables and\r\napproach similar to:\r\n\r\n```python\r\n dag = self.dagbag.get_dag(\"test_retry_handling_job\")\r\n dag_task1 = dag.get_task(\"test_retry_handling_op\")\r\n dag.clear()\r\n dag.sync_to_db()\r\n```\r\n\r\nAllowed the test to run in isolated space where only one or few\r\nDAGS were present in the DB.\r\n\r\nThis probably worked perfectly in the past, but after some changes\r\nin how DAGFileProcessor works this did not prevent DAGFileProcessor\r\nfrom running when _execute method in scheduler_job_runner has been\r\nexecuted, and standalone dag processor was not running, the\r\nprocessor_agent has been overwritten by a new DagFileProcessor\r\nin the `_execute` method of scheduler_job_runner.\r\n\r\n```python\r\n if not self._standalone_dag_processor:\r\n self.processor_agent = DagFileProcessorAgent(\r\n dag_directory=Path(self.subdir),\r\n max_runs=self.num_times_parse_dags,\r\n processor_timeout=processor_timeout,\r\n dag_ids=[],\r\n pickle_dags=pickle_dags,\r\n async_mode=async_mode,\r\n )\r\n```\r\n\r\nThis led to a very subtle race condition which was more likely on\r\nmachines with multiple cores/faster disk (so for example it\r\nled to #35204 which appeared on self-hosted (8 core) runners and\r\ndid not appear on Public (2-core runners) or it could appear on\r\nan 8 core ARM Mac but not appear on 6 core Intel Mac (only on\r\nsqlite)\r\n\r\nIf the DAGFileProcessor managed to start and spawn some\r\nparsing processes and grab the DB write access for sqlite and those\r\nprocesses managed to parse some of the DAG files from tests/dags/\r\nfolder, those DAGs could have polutted the DAGs in the DB - leading\r\nto undesired effects (for example with test hanging while the\r\nscheduler job run attempted to process an unwanted subdag and\r\ngot deadlocked in case of #35204.\r\n\r\nThe solution to that is to only set the processor_agent if not\r\nset already. This can only happen in unit tests when the\r\n`processor_agent` sets it to Mock object. For \"production\" the\r\nagent is only set once in the `_execute` methods so there is no\r\nrisk involved in checking if it is not set already.\r\n\r\nFixes: #35204","shortMessageHtmlLink":"Fix subtle bug in mocking processor_agent in our tests (apache#35221)"}},{"before":"3b0ea05796e618d0761f30575070a188c719422f","after":null,"ref":"refs/heads/tuan/spark_kerberos_ticket_cache","pushedAt":"2023-10-28T03:15:02.000Z","pushType":"branch_deletion","commitsCount":0,"pusher":{"login":"zeotuan","name":null,"path":"/zeotuan","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/48720253?s=80&v=4"}},{"before":"c69d1c831f72d11124d7c0eb12fd6e3c35548add","after":null,"ref":"refs/heads/tuan/11246","pushedAt":"2023-10-28T03:14:13.000Z","pushType":"branch_deletion","commitsCount":0,"pusher":{"login":"zeotuan","name":null,"path":"/zeotuan","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/48720253?s=80&v=4"}},{"before":"543db7004ee593605e250265b0722917cef296d3","after":"6f3d294645153db914be69cd2b2a49f12a18280c","ref":"refs/heads/main","pushedAt":"2023-10-28T03:11:18.000Z","pushType":"push","commitsCount":27,"pusher":{"login":"zeotuan","name":null,"path":"/zeotuan","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/48720253?s=80&v=4"},"commit":{"message":"Fix subtle bug in mocking processor_agent in our tests (#35221)\n\nSome of the scheduler tests tried to prevent DAG processor processing DAGs\r\nfrom \"tests/dags\" directory by setting processor_agent to Mock object:\r\n\r\n```python\r\n self.job_runner.processor_agent = mock.MagicMock()\r\n```\r\n\r\nThis, in connection with scheduler job cleaning all the tables and\r\napproach similar to:\r\n\r\n```python\r\n dag = self.dagbag.get_dag(\"test_retry_handling_job\")\r\n dag_task1 = dag.get_task(\"test_retry_handling_op\")\r\n dag.clear()\r\n dag.sync_to_db()\r\n```\r\n\r\nAllowed the test to run in isolated space where only one or few\r\nDAGS were present in the DB.\r\n\r\nThis probably worked perfectly in the past, but after some changes\r\nin how DAGFileProcessor works this did not prevent DAGFileProcessor\r\nfrom running when _execute method in scheduler_job_runner has been\r\nexecuted, and standalone dag processor was not running, the\r\nprocessor_agent has been overwritten by a new DagFileProcessor\r\nin the `_execute` method of scheduler_job_runner.\r\n\r\n```python\r\n if not self._standalone_dag_processor:\r\n self.processor_agent = DagFileProcessorAgent(\r\n dag_directory=Path(self.subdir),\r\n max_runs=self.num_times_parse_dags,\r\n processor_timeout=processor_timeout,\r\n dag_ids=[],\r\n pickle_dags=pickle_dags,\r\n async_mode=async_mode,\r\n )\r\n```\r\n\r\nThis led to a very subtle race condition which was more likely on\r\nmachines with multiple cores/faster disk (so for example it\r\nled to #35204 which appeared on self-hosted (8 core) runners and\r\ndid not appear on Public (2-core runners) or it could appear on\r\nan 8 core ARM Mac but not appear on 6 core Intel Mac (only on\r\nsqlite)\r\n\r\nIf the DAGFileProcessor managed to start and spawn some\r\nparsing processes and grab the DB write access for sqlite and those\r\nprocesses managed to parse some of the DAG files from tests/dags/\r\nfolder, those DAGs could have polutted the DAGs in the DB - leading\r\nto undesired effects (for example with test hanging while the\r\nscheduler job run attempted to process an unwanted subdag and\r\ngot deadlocked in case of #35204.\r\n\r\nThe solution to that is to only set the processor_agent if not\r\nset already. This can only happen in unit tests when the\r\n`processor_agent` sets it to Mock object. For \"production\" the\r\nagent is only set once in the `_execute` methods so there is no\r\nrisk involved in checking if it is not set already.\r\n\r\nFixes: #35204","shortMessageHtmlLink":"Fix subtle bug in mocking processor_agent in our tests (apache#35221)"}},{"before":"1b1f881ad08d87395035c69d6152137204a462f2","after":"543db7004ee593605e250265b0722917cef296d3","ref":"refs/heads/main","pushedAt":"2023-10-25T05:42:14.000Z","pushType":"force_push","commitsCount":0,"pusher":{"login":"zeotuan","name":null,"path":"/zeotuan","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/48720253?s=80&v=4"},"commit":{"message":"Consolidate stacklevel in ssh operator warning (#35151)\n\nCo-authored-by: Shubham ","shortMessageHtmlLink":"Consolidate stacklevel in ssh operator warning (apache#35151)"}},{"before":"3898c09f7c0c932275d9176df24dfba5b2343456","after":"1b1f881ad08d87395035c69d6152137204a462f2","ref":"refs/heads/main","pushedAt":"2023-10-22T03:17:01.000Z","pushType":"push","commitsCount":0,"pusher":{"login":"zeotuan","name":null,"path":"/zeotuan","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/48720253?s=80&v=4"}},{"before":"eb7611e845e22e7b1088412c0b2bc0f36411be57","after":null,"ref":"refs/heads/tuan/r","pushedAt":"2023-10-19T21:46:43.000Z","pushType":"branch_deletion","commitsCount":0,"pusher":{"login":"zeotuan","name":null,"path":"/zeotuan","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/48720253?s=80&v=4"}},{"before":"cea590f0710f2c7ff0522268e0e8f4ed4a941ec9","after":"3898c09f7c0c932275d9176df24dfba5b2343456","ref":"refs/heads/main","pushedAt":"2023-10-19T21:30:01.000Z","pushType":"force_push","commitsCount":0,"pusher":{"login":"zeotuan","name":null,"path":"/zeotuan","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/48720253?s=80&v=4"},"commit":{"message":"Add use_krb5ccache arg","shortMessageHtmlLink":"Add use_krb5ccache arg"}},{"before":"0c75ce706ad9e4fa41b78f24e16f1d2530611f0c","after":"cea590f0710f2c7ff0522268e0e8f4ed4a941ec9","ref":"refs/heads/main","pushedAt":"2023-10-19T21:29:12.000Z","pushType":"force_push","commitsCount":0,"pusher":{"login":"zeotuan","name":null,"path":"/zeotuan","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/48720253?s=80&v=4"}},{"before":"cea590f0710f2c7ff0522268e0e8f4ed4a941ec9","after":null,"ref":"refs/heads/zeotuan/main","pushedAt":"2023-10-19T21:26:43.000Z","pushType":"branch_deletion","commitsCount":0,"pusher":{"login":"zeotuan","name":null,"path":"/zeotuan","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/48720253?s=80&v=4"}},{"before":null,"after":"cea590f0710f2c7ff0522268e0e8f4ed4a941ec9","ref":"refs/heads/zeotuan/main","pushedAt":"2023-10-19T21:25:41.000Z","pushType":"branch_creation","commitsCount":0,"pusher":{"login":"zeotuan","name":null,"path":"/zeotuan","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/48720253?s=80&v=4"}},{"before":"cea590f0710f2c7ff0522268e0e8f4ed4a941ec9","after":"eb7611e845e22e7b1088412c0b2bc0f36411be57","ref":"refs/heads/tuan/r","pushedAt":"2023-10-19T21:21:43.000Z","pushType":"push","commitsCount":0,"pusher":{"login":"zeotuan","name":null,"path":"/zeotuan","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/48720253?s=80&v=4"}},{"before":"0c75ce706ad9e4fa41b78f24e16f1d2530611f0c","after":"cea590f0710f2c7ff0522268e0e8f4ed4a941ec9","ref":"refs/heads/tuan/r","pushedAt":"2023-10-19T21:21:02.000Z","pushType":"force_push","commitsCount":0,"pusher":{"login":"zeotuan","name":null,"path":"/zeotuan","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/48720253?s=80&v=4"}},{"before":null,"after":"0c75ce706ad9e4fa41b78f24e16f1d2530611f0c","ref":"refs/heads/tuan/r","pushedAt":"2023-10-19T20:46:14.000Z","pushType":"branch_creation","commitsCount":0,"pusher":{"login":"zeotuan","name":null,"path":"/zeotuan","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/48720253?s=80&v=4"}},{"before":"c364a25e8c00b6b7db6eb4e286967fb9f36bda97","after":"c69d1c831f72d11124d7c0eb12fd6e3c35548add","ref":"refs/heads/tuan/11246","pushedAt":"2023-10-18T19:28:50.000Z","pushType":"push","commitsCount":54,"pusher":{"login":"eladkal","name":"Elad Kalif","path":"/eladkal","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/45845474?s=80&v=4"},"commit":{"message":"Merge branch 'main' into tuan/11246","shortMessageHtmlLink":"Merge branch 'main' into tuan/11246"}},{"before":"a9df69b86af540fd2e904eecca647773888fd034","after":"c364a25e8c00b6b7db6eb4e286967fb9f36bda97","ref":"refs/heads/tuan/11246","pushedAt":"2023-10-17T23:47:08.000Z","pushType":"push","commitsCount":1,"pusher":{"login":"zeotuan","name":null,"path":"/zeotuan","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/48720253?s=80&v=4"},"commit":{"message":"Add doc String","shortMessageHtmlLink":"Add doc String"}},{"before":"37483cf8f33be27ba1e19b4f5a5e15e0afe06660","after":"a9df69b86af540fd2e904eecca647773888fd034","ref":"refs/heads/tuan/11246","pushedAt":"2023-10-16T05:08:13.000Z","pushType":"push","commitsCount":3,"pusher":{"login":"zeotuan","name":null,"path":"/zeotuan","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/48720253?s=80&v=4"},"commit":{"message":"Merge branch 'tuan/11246' of https://github.com/zeotuan/airflow into tuan/11246","shortMessageHtmlLink":"Merge branch 'tuan/11246' of https://github.com/zeotuan/airflow into …"}},{"before":"08a18d258fc91d50726a6ca3456aeefa53ea0238","after":"37483cf8f33be27ba1e19b4f5a5e15e0afe06660","ref":"refs/heads/tuan/11246","pushedAt":"2023-10-14T06:21:32.000Z","pushType":"force_push","commitsCount":0,"pusher":{"login":"zeotuan","name":null,"path":"/zeotuan","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/48720253?s=80&v=4"},"commit":{"message":"Refactor: Extract get_kerberos_principle func","shortMessageHtmlLink":"Refactor: Extract get_kerberos_principle func"}},{"before":"4c1808bb8991c990ef2b7ef3554a5a4a5fb6078f","after":"08a18d258fc91d50726a6ca3456aeefa53ea0238","ref":"refs/heads/tuan/11246","pushedAt":"2023-10-14T06:14:42.000Z","pushType":"push","commitsCount":1,"pusher":{"login":"zeotuan","name":null,"path":"/zeotuan","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/48720253?s=80&v=4"},"commit":{"message":"Refactor: Extract get_kerberos_principle func","shortMessageHtmlLink":"Refactor: Extract get_kerberos_principle func"}},{"before":null,"after":"4c1808bb8991c990ef2b7ef3554a5a4a5fb6078f","ref":"refs/heads/tuan/11246","pushedAt":"2023-10-14T06:07:58.000Z","pushType":"branch_creation","commitsCount":0,"pusher":{"login":"zeotuan","name":null,"path":"/zeotuan","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/48720253?s=80&v=4"},"commit":{"message":"Remove hacktoberfest label from project description (#34697)","shortMessageHtmlLink":"Remove hacktoberfest label from project description (apache#34697)"}},{"before":"589c2454f003fb5c0ff31a34a2c89e8ace874e00","after":"0c75ce706ad9e4fa41b78f24e16f1d2530611f0c","ref":"refs/heads/main","pushedAt":"2023-10-06T01:46:54.000Z","pushType":"push","commitsCount":0,"pusher":{"login":"zeotuan","name":null,"path":"/zeotuan","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/48720253?s=80&v=4"}}],"hasNextPage":true,"hasPreviousPage":false,"activityType":"all","actor":null,"timePeriod":"all","sort":"DESC","perPage":30,"cursor":"djE6ks8AAAAEO7iXswA","startCursor":null,"endCursor":null}},"title":"Activity · zeotuan/airflow"}