Domain-Specific Language (DSL) for defining Big Data pipeplines. The developed DSL supports
- a separation of concerns between design and run-time issues;
- reuse of previously developed steps and pipelines in designing new pipelines;
- a flexible and smooth data transfer between steps; and
- applying containerization for nodes and pipeline descriptions.
DEF-PIPE provides tooling that allows you to define Big Data pipelines according to the DSL.
In order to load the DSL project you will need to install the Eclipse IDE.
We recommend using release 2021-12 of the Eclipse Installer. During the installation, select the bundle Eclipse Modeling Tools as this contains the Eclipse Modeling Framework (EMF) package that is used to define DSL.
The EMF project is a modeling framework and code generation facility for building tools and other applications based on a structured data model. In this project the core metamodel is technically represented as an ECore model, which is organised into several smaller packages. Each of these packages reflects a specific aspect of our DSL. The DSL is developed using EMF version 2.28 which comes installed with recommended 2021-12 release of the Eclipse Modeling Tools bundle. Other versions are not guaranteed to work correctly.
Follow these steps to load the EMF project and open the EMF diagrams:
-
Clone the DEF-PIPE-DSL repository from GitHub.
-
Start the Eclipse IDE and select the cloned folder as the workspace.
-
Select "File" -> "Open projects from file system" and import the following projects.
-
Select "Window" -> "Perspective" -> "Open Perspective" -> "Other" and select the "Modeling (default)".
-
Navigate to the file "workflow/model/generated/WfDsl.aird" and double click on it. After this you will then be able to navigate to the sub-folder "Representations per category" where you will be able to open the EMF diagrams, i.e., the "wfDsl class diagram" as shown below.
The grammar of the DSL is defined according to Xtext and is located at