You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
<>.. Why is it doing.. Can you please help.. Here is the yaml and dag code..
Note: When i was using 2.7.1, i used to redirect the yaml file contents to a string variable and use to pass that string variable the sparkoperator.. Its no longer working in 2.8.4 .. so i am passing the file path of the yaml.. (in the code thats the reason i commented.. # yaml_file = yaml.dump(yaml.load(open(args_dict["application_yaml_file"]), Loader=yaml.FullLoader))
# print(yaml_file)... i wish i can still pass that as a string (not to divert the discussion)
reacted with thumbs up emoji reacted with thumbs down emoji reacted with laugh emoji reacted with hooray emoji reacted with confused emoji reacted with heart emoji reacted with rocket emoji reacted with eyes emoji
-
I am getting this error when using SparkKubernetesOperator operator in 2.8.4
File "/usr/local/lib/python3.11/site-packages/airflow/providers/cncf/kubernetes/operators/spark_kubernetes.py", line 127, in manage_template_specs
template_body = _load_body_to_dict(open(self.application_file))
^^^^^^^^^^^^^^^^^^^^^^^^^^^
FileNotFoundError: [Errno 2] No such file or directory: "apiVersion: sparkoperator.k8s.io/v1beta2\nkind: SparkApplication\nmetadata:\n name: demo-pyspark-app\n namespace: test-bdp-dev-001\n annotations:\n sidecar.istio.io/inject: 'false'\nspec:\n type: Scala\n mode: cluster\n image: 'test
.azurecr.io/test/bdp/bdp-spark:3.3.0-108'\n imagePullPolicy: IfNotPresent\n mainClass: org.apache.spark.examples.SparkPi\n mainApplicationFile: 'local:///opt/spark/examples/jars/spark-examples_2.12-3.3.0.jar'\n arguments:\n - '200000'\n sparkVersion: 3.3.0\n sparkConf:\n spark.eventLog.dir: /tmp/\n spark.eventLog.enabled: 'true'\n spark.eventLog.logStageExecutorMetrics: 'true'\n spark.jar.ivy: /tmp\n spark.kubernetes.driver.pod.name: demo-pyspark-app\n spark.kubernetes.executor.podNamePrefix: demo-pyspark-app\n spark.dynamicAllocation.enabled: 'true'\n spark.dynamicAllocation.shuffleTracking.enabled: 'true'\n spark.dynamicAllocation.minExecutors: '1'\n spark.dynamicAllocation.maxExecutors: '15'\n spark.kubernetes.executor.volumes.persistentVolumeClaim.data.mount.path: /var/log/app_logs\n spark.kubernetes.executor.volumes.persistentVolumeClaim.data.options.claimName: OnDemand\n spark.kubernetes.executor.volumes.persistentVolumeClaim.data.options.sizeLimit: 2Gi\n spark.kubernetes.executor.volumes.persistentVolumeClaim.data.options.storageClass: cps-managed-disk-delete\n driver:\n annotations:\n sidecar.istio.io/inject: 'false'\n coreLimit: 1000m\n coreRequest: 55m\n memory: 512m\n serviceAccount: dev-bdp\n executor:\n annotations:\n sidecar.istio.io/inject: 'false'\n coreLimit: 1000m\n coreRequest: 55m\n memory: 512m\n labels:\n version: 3.3.0"
<>.. Why is it doing.. Can you please help..
Here is the yaml and dag code..
Note: When i was using 2.7.1, i used to redirect the yaml file contents to a string variable and use to pass that string variable the sparkoperator.. Its no longer working in 2.8.4 .. so i am passing the file path of the yaml.. (in the code thats the reason i commented.. # yaml_file = yaml.dump(yaml.load(open(args_dict["application_yaml_file"]), Loader=yaml.FullLoader))
# print(yaml_file)... i wish i can still pass that as a string (not to divert the discussion)
Beta Was this translation helpful? Give feedback.
All reactions