Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

use splineAgent as a Jar file #761

Open
zacayd opened this issue Oct 30, 2023 · 12 comments
Open

use splineAgent as a Jar file #761

zacayd opened this issue Oct 30, 2023 · 12 comments

Comments

@zacayd
Copy link

zacayd commented Oct 30, 2023 •
edited by wajda

Hi
i tried to install the Agent Spline without Maven - but Upload of the a Jar FIile
i cloned the code from git and build it to a spline-spark-agent_2.12.jar
and installed library using Upload to Databricks
when i run the code

import scala.util.parsing.json.JSON
import za.co.absa.spline.harvester.SparkLineageInitializer._
import za.co.absa.spline.agent.AgentConfig
import za.co.absa.spline.harvester.postprocessing.AbstractPostProcessingFilter
import za.co.absa.spline.harvester.postprocessing.PostProcessingFilter
import org.apache.commons.configuration.Configuration
import za.co.absa.spline.harvester.conf.StandardSplineConfigurationStack
import za.co.absa.spline.harvester.HarvestingContext
import za.co.absa.spline.producer.model.ExecutionPlan
import za.co.absa.spline.producer.model.ExecutionEvent
import za.co.absa.spline.producer.model.ReadOperation
import za.co.absa.spline.producer.model.WriteOperation
import za.co.absa.spline.producer.model.DataOperation
import za.co.absa.spline.harvester.ExtraMetadataImplicits._
import za.co.absa.spline.harvester.SparkLineageInitializer._

I get error that it doesnt recognise the packages
can you please advise what is missing?
@cerveada
Copy link
Contributor

Hi, share the error, what jar are you uploading?

@zacayd
Copy link
Author

zacayd commented Nov 1, 2023

HI
i have attached the screen shot of the installation of the jar
the jar is the artifact of build the project
https://github.com/AbsaOSS/spline-spark-agent
Screenshot 2023-11-01 130220

the errors are

command-4378807525288898:2: error: not found: value za
import za.co.absa.spline.harvester.SparkLineageInitializer._
^
command-4378807525288898:3: error: not found: value za
import za.co.absa.spline.agent.AgentConfig
^
command-4378807525288898:4: error: not found: value za
import za.co.absa.spline.harvester.postprocessing.AbstractPostProcessingFilter
^
command-4378807525288898:5: error: not found: value za
import za.co.absa.spline.harvester.postprocessing.PostProcessingFilter
^
command-4378807525288898:6: error: object configuration is not a member of package org.apache.commons
import org.apache.commons.configuration.Configuration
^
command-4378807525288898:7: error: not found: value za
import za.co.absa.spline.harvester.conf.StandardSplineConfigurationStack
^
command-4378807525288898:8: error: not found: value za
import za.co.absa.spline.harvester.HarvestingContext
^
command-4378807525288898:9: error: not found: value za
import za.co.absa.spline.producer.model.ExecutionPlan
^
command-4378807525288898:10: error: not found: value za
import za.co.absa.spline.producer.model.ExecutionEvent
^
command-4378807525288898:11: error: not found: value za
import za.co.absa.spline.producer.model.ReadOperation
^
command-4378807525288898:12: error: not found: value za
import za.co.absa.spline.producer.model.WriteOperation
^
command-4378807525288898:13: error: not found: value za
import za.co.absa.spline.producer.model.DataOperation
^
command-4378807525288898:14: error: not found: value za
import za.co.absa.spline.harvester.ExtraMetadataImplicits._
^
command-4378807525288898:15: error: not found: value za
import za.co.absa.spline.harvester.SparkLineageInitializer._
^

@cerveada
Copy link
Contributor

cerveada commented Nov 1, 2023

If you are modifying the code, you must build and use your own spark-3.3-spline-agent-bundle.jar not the one from maven.

@wajda
Copy link
Contributor

wajda commented Nov 1, 2023

error: not found: value za

It looks like the Jar is either not seen by the Spark, or is empty/does not contain any Spline related classes. Please check your Jar content and the way you attach it to Spark.

@zacayd
Copy link
Author

zacayd commented Nov 2, 2023

the jar is 5K - it was created when i made a build on the Intelij
the file is called spline-spark-agent_2.12.jar
i took the content of the POM.xml from Bundle 3.3
I then uploaded the jar as "Install New" on the Databricks Cluster
Do you have a working solution of spline for Databricks which is not from Maven Central ?

@wajda
Copy link
Contributor

wajda commented Nov 2, 2023 •
edited

Please follow the building procedure from this section in the README - https://github.com/AbsaOSS/spline-spark-agent#building
Then take the bundle jar from the bundle-3.3/target folder.

@zacayd
Copy link
Author

zacayd commented Nov 6, 2023 via email

@wajda
Copy link
Contributor

wajda commented Nov 6, 2023

I've never seen this error. Something might be wrong with your POM files. Try it on a clean project. call the Maven command in the project root directory.

Here's how it should look like:

 ~/Projects/spline-spark-agent │ develop                                                                                                                                                      
❯ mvn scala-cross-build:change-version -Pscala-2.11
[INFO] Scanning for projects...
[INFO] ------------------------------------------------------------------------
[INFO] Reactor Build Order:
[INFO]
[INFO] Spline Spark Agent                                                 [pom]
[INFO] agent-commons_2.12                                                 [jar]
[INFO] agent-core_2.12                                                    [jar]
[INFO] examples_2.12                                                      [jar]
[INFO] integration-tests_2.12                                             [jar]
[INFO] spark-2.2-spline-agent-bundle_2.11                                 [jar]
[INFO] spark-2.3-spline-agent-bundle_2.11                                 [jar]
[INFO] spark-2.4-spline-agent-bundle_2.12                                 [jar]
[INFO]
[INFO] -------< za.co.absa.spline.agent.spark:spline-spark-agent_2.12 >--------
[INFO] Building Spline Spark Agent 2.1.0-SNAPSHOT                         [1/8]
[INFO]   from pom.xml
[INFO] --------------------------------[ pom ]---------------------------------
[INFO]
[INFO] --- scala-cross-build:0.2.2:change-version (default-cli) @ spline-spark-agent_2.12 ---
[INFO]
[INFO] ----------< za.co.absa.spline.agent.spark:agent-commons_2.12 >----------
[INFO] Building agent-commons_2.12 2.1.0-SNAPSHOT                         [2/8]
[INFO]   from commons/pom.xml
[INFO] --------------------------------[ jar ]---------------------------------
[INFO]
[INFO] --- scala-cross-build:0.2.2:change-version (default-cli) @ agent-commons_2.12 ---
[INFO]
[INFO] -----------< za.co.absa.spline.agent.spark:agent-core_2.12 >------------
[INFO] Building agent-core_2.12 2.1.0-SNAPSHOT                            [3/8]
[INFO]   from core/pom.xml
[INFO] --------------------------------[ jar ]---------------------------------
[INFO]
[INFO] --- scala-cross-build:0.2.2:change-version (default-cli) @ agent-core_2.12 ---
[INFO]
[INFO] ------------< za.co.absa.spline.agent.spark:examples_2.12 >-------------
[INFO] Building examples_2.12 2.1.0-SNAPSHOT                              [4/8]
[INFO]   from examples/pom.xml
[INFO] --------------------------------[ jar ]---------------------------------
[INFO]
[INFO] --- scala-cross-build:0.2.2:change-version (default-cli) @ examples_2.12 ---
[INFO]
[INFO] --------< za.co.absa.spline.agent.spark:integration-tests_2.12 >--------
[INFO] Building integration-tests_2.12 2.1.0-SNAPSHOT                     [5/8]
[INFO]   from integration-tests/pom.xml
[INFO] --------------------------------[ jar ]---------------------------------
[INFO]
[INFO] --- scala-cross-build:0.2.2:change-version (default-cli) @ integration-tests_2.12 ---
[INFO]
[INFO] --< za.co.absa.spline.agent.spark:spark-2.2-spline-agent-bundle_2.11 >--
[INFO] Building spark-2.2-spline-agent-bundle_2.11 2.1.0-SNAPSHOT         [6/8]
[INFO]   from bundle-2.2/pom.xml
[INFO] --------------------------------[ jar ]---------------------------------
[INFO]
[INFO] --- scala-cross-build:0.2.2:change-version (default-cli) @ spark-2.2-spline-agent-bundle_2.11 ---
[INFO]
[INFO] --< za.co.absa.spline.agent.spark:spark-2.3-spline-agent-bundle_2.11 >--
[INFO] Building spark-2.3-spline-agent-bundle_2.11 2.1.0-SNAPSHOT         [7/8]
[INFO]   from bundle-2.3/pom.xml
[INFO] --------------------------------[ jar ]---------------------------------
[INFO]
[INFO] --- scala-cross-build:0.2.2:change-version (default-cli) @ spark-2.3-spline-agent-bundle_2.11 ---
[INFO]
[INFO] --< za.co.absa.spline.agent.spark:spark-2.4-spline-agent-bundle_2.12 >--
[INFO] Building spark-2.4-spline-agent-bundle_2.12 2.1.0-SNAPSHOT         [8/8]
[INFO]   from bundle-2.4/pom.xml
[INFO] --------------------------------[ jar ]---------------------------------
[INFO]
[INFO] --- scala-cross-build:0.2.2:change-version (default-cli) @ spark-2.4-spline-agent-bundle_2.12 ---
[INFO] ------------------------------------------------------------------------
[INFO] Reactor Summary for Spline Spark Agent 2.1.0-SNAPSHOT:
[INFO]
[INFO] Spline Spark Agent ................................. SUCCESS [  0.078 s]
[INFO] agent-commons_2.12 ................................. SUCCESS [  0.009 s]
[INFO] agent-core_2.12 .................................... SUCCESS [  0.009 s]
[INFO] examples_2.12 ...................................... SUCCESS [  0.009 s]
[INFO] integration-tests_2.12 ............................. SUCCESS [  0.009 s]
[INFO] spark-2.2-spline-agent-bundle_2.11 ................. SUCCESS [  0.007 s]
[INFO] spark-2.3-spline-agent-bundle_2.11 ................. SUCCESS [  0.008 s]
[INFO] spark-2.4-spline-agent-bundle_2.12 ................. SUCCESS [  0.009 s]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time:  0.279 s
[INFO] Finished at: 2023-11-06T10:58:25+01:00
[INFO] ------------------------------------------------------------------------

@zacayd
Copy link
Author

zacayd commented Nov 6, 2023 via email

@wajda
Copy link
Contributor

wajda commented Nov 13, 2023

There seems to be a compilation error. Are you compiling the original source code, or you changed something in it?

@zacayd
Copy link
Author

zacayd commented Nov 14, 2023

the origanl code

@wajda
Copy link
Contributor

wajda commented Nov 14, 2023

Sorry I cannot reproduce it. It builds fine on all our environments including CI and personal PCs. Please try to build it on another env. It must work.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
Spline
Status: New
Milestone
No milestone
Development

No branches or pull requests
3 participants
@wajda @cerveada @zacayd