xtext-maven-plugin generate does not respect order of languages in pom.xml

[cosminpolifronie]

Hello everyone! I was trying to use my languages to generate code in another project, and I found out that the order in the pom.xml is not respected.

Instead, all grammars to be parsed are gathered together, some deltas are computed, and the end result is an arbitrary run of the languages.

For example, in my case:

<plugin>
  <groupId>org.eclipse.xtext</groupId>
  <artifactId>xtext-maven-plugin</artifactId>
  <version>${xtext-version}</version>
  <executions>
    <execution>
      <id>my-generate-prepare</id>
      <goals>
        <goal>generate</goal>
      </goals>
      <configuration>
        <languages>
          <language>
            <setup>my.generator.lang1StandaloneSetup</setup>
            <outputConfigurations>
              <outputConfiguration>
                <outputDirectory>${my-grammar-files}</outputDirectory>
              </outputConfiguration>
            </outputConfigurations>
          </language>
          <language>
            <setup>my.generator.lang2StandaloneSetup</setup>
            <outputConfigurations>
              <outputConfiguration>
                <outputDirectory>${my-grammar-files}</outputDirectory>
              </outputConfiguration>
            </outputConfigurations>
          </language>
          <language>
            <setup>my.generator.lang3StandaloneSetup</setup>
            <outputConfigurations>
              <outputConfiguration>
                <outputDirectory>${my-grammar-files}</outputDirectory>
              </outputConfiguration>
            </outputConfigurations>
          </language>
          <language>
            <setup>my.generator.lang4StandaloneSetup</setup>
            <outputConfigurations>
              <outputConfiguration>
                <outputDirectory>${my-grammar-files}</outputDirectory>
              </outputConfiguration>
            </outputConfigurations>
          </language>
          <language>
            <setup>my.generator.lang5StandaloneSetup</setup>
            <outputConfigurations>
              <outputConfiguration>
                <outputDirectory>${my-grammar-files}</outputDirectory>
              </outputConfiguration>
            </outputConfigurations>
          </language>
        </languages>
      </configuration>
    </execution>
  </executions>
  <dependencies>
    <dependency>
      <groupId>my.generator</groupId>
      <artifactId>my.generator</artifactId>
      <version>1.0.0-SNAPSHOT</version>
    </dependency>
  </dependencies>
</plugin>

The pom above will validate and generate in this order:
lang3 lang1 lang5 lang2 lang4

Expected run:
lang1 lang2 lang3 lang4 lang5

This is necessary in my project as the first 2 languages are initializing some states/variables inside the generator which will be used by langs3-5.

I found the culprit code to be here:

xtext/org.eclipse.xtext.builder.standalone/src/org/eclipse/xtext/builder/standalone/StandaloneBuilder.java

Lines 250 to 273 in 1d83f89

	LOG.info("Validate and generate.");
	while (!changedSourceFiles.isEmpty() || !allDeltas.isEmpty()) {
	installSourceLevelURIs(resourceSet, changedSourceFiles);
	Iterator<URI> sourceResourceIterator = changedSourceFiles.iterator();
	while (sourceResourceIterator.hasNext()) {
	List<Resource> resources = new ArrayList<>();
	int clusterIndex = 0;
	boolean canContinue = true;
	while (sourceResourceIterator.hasNext() && canContinue) {
	URI uri = sourceResourceIterator.next();
	Resource resource = resourceSet.getResource(uri, true);
	resources.add(resource);
	resource.getContents(); // fully initialize
	EcoreUtil2.resolveLazyCrossReferences(resource, CancelIndicator.NullImpl);
	IResourceDescription.Manager manager = resourceDescriptionManager(resource);
	IResourceDescription oldDescription = index.getResourceDescription(uri);
	IResourceDescription newDescription = SerializableResourceDescription
	.createCopy(manager.getResourceDescription(resource));
	index.addDescription(uri, newDescription);
	aggregateDelta(manager.createDelta(oldDescription, newDescription), allDeltas);
	// TODO adjust to handle validations that need an up-to-date index
	hasValidationErrors = validate(resource) || hasValidationErrors;

This code should be modified to use the languages list to determine the parsing order of the files.
I tried to do the changes myself but I am having trouble building xtext locally.

Would there be another way to do this? I cannot use multiple execution entries as this would reinitialize the language model.

This language works perfectly fine inside Eclipse, I just have to manually run the generator on each file individually, in the order that I need.

[LorenzoBettini]

You could have different executions in the plugin configuration: the first one with the first two languages and the second one with the other ones

[cdietrich]

you also could have circular dependencies between the two files. thus the behaviour is intentional.
also i eclipse the builder would not make generated files by one dsl visible to another.

can you give some insights what

are initializing some states/variables inside the generator actually resembles to

[cosminpolifronie]

You could have different executions in the plugin configuration: the first one with the first two languages and the second one with the other ones

@LorenzoBettini Sadly I already tried that and it doesn't work: there will be 2 different runs, one for each execution, so the generator state is reset between the runs.

you also could have circular dependencies between the two files. thus the behaviour is intentional. also i eclipse the builder would not make generated files by one dsl visible to another.

can you give some insights what

are initializing some states/variables inside the generator actually resembles to

@cdietrich I will have to look into the code and come back to you. I am not that intimately familiar with the implemented DSL, I was working on just porting the DSL to Maven.

Currently I just know that two files (each with their own grammar) need to be parsed first before being able to parse number 3, then 4, and then 5. Any other order does not work.

Will be back with updates.

[cosminpolifronie]

Okay, so I took a look over what is implemented.

Let's assume this is the implementation of Lang1Generator class (that processes .lang1 files):

class Lang1Generator extends AbstractGenerator {

	public static var List<Variable> list1 = new ArrayList()
	public static var List<Variable> list2 = new ArrayList()
	public static var List<String> list3 = new ArrayList()
        ...
	
	override void doGenerate(org.eclipse.emf.ecore.resource.Resource input, IFileSystemAccess2 fsa, IGeneratorContext context) {
		// populate list1-3 based on what has been read from .lang1 file
        }
}	

And then, let's assume this is the implementation of Lang5Generator class (that processes .lang5 files):

class Lang5Generator extends AbstractGenerator {
	override void doGenerate(org.eclipse.emf.ecore.resource.Resource input, IFileSystemAccess2 fsa, IGeneratorContext context) {
		// function calls using Lang1Generator::list1
        }
}	

If we try to first generate .lang5 files, Lang1Generator::list1 will be null, and the generation will error out. We would need to first parse the .lang1 files to populate Lang1Generator::list1, and then parse .lang5 files to be able to use what has been read previously.

I'm not sure how orthodox this implementation is.

How would I create a dependency with .lang1 files in the .lang5 code? Or how should I go around this?

Inside Eclipse, we would manually save .lang1 file first, then .lang2, and then the others. As such we are making sure all variables are populated before generation. I haven't implemented it like this, I just took this project over.

[cdietrich]

and what do you store in the lists?
shouldnt you be able to calculate that from anywhere
based on the resourceset/xtext index
maybe even store the result in the xtext index

also the manual eclipse order seems wrong

[cosminpolifronie]

So, the .lang1 and .lang2 files contain some product definitions that have their own grammar. Things like:

var Persistierung as setPersistencyBehaviour
[
	nein		: "Notification::PersistencyBehaviour::NONE",
	shutdown	: "Notification::PersistencyBehaviour::LAZY",
	sofort		: "Notification::PersistencyBehaviour::IMMEDIATELY"
]

service dummyservice, ID: 0x1234, ServiceType: required ("28.03.2024", Version "1.2.3")
{
     resource dummyresource1 / Create = No, Update = No, Read = Yes, Get = No, Set = No, Subscribed = Yes, Polled = No, PollingInterval = 0, PollingTimeout = 0
     {
          type  Enum:UInt32  ID  LogicalValues: 1 = definitionof1;2 = definitionof2;3 = definitionof3
          value  UInt32   
     }

     resource dummyresource2 / Create = No, Update = No, Read = Yes, Get = No, Set = No, Subscribed = Yes, Polled = No, PollingInterval = 0, PollingTimeout = 0
     {
          elementId  UInt32  ID  
          timeGap  UInt32   
          dummyObjectId  UInt32   
          dummyObjectIdValueType  Enum:UInt32  LogicalValues: 1 = Init;4 = interpret 
     }

}

Then, in .lang4 we could have definitions using the services and variable types defined above:

dummy Dummy
{

0x3911
    /dummyservice/dummyresource1[type==3]/value == 1

In .lang5 we could have something like:

ID=DUMMY_ID_5678,
    Persistierung=nein

Only .lang3 to .lang5 are actually generating files in our case, .lang1 and .lang2 just define types and values written in a custom language that can be used in the other .lang3-5 files.

The benefit of using grammars for .lang1 and .lang2 is that you cannot mistype stuff. You can assign wrong values, but those are very clearly defined. Thus, the mistake rate is greatly reduced.

and what do you store in the lists?

The definitions in .lang1 and .lang2. Basically, all the permitted values for each type of variable, or a list of all the services defined in .lang2,

shouldnt you be able to calculate that from anywhere

Not really as .lang1 and .lang2 are written in their own custom language.

maybe even store the result in the xtext index

That I am not familiar with. Could you please link to what you are reffering? Maybe an example

also the manual eclipse order seems wrong

I couldn't agree more. That's what I am also trying to fix, among other things.

[cdietrich]

can you give a concrete example how you write stuff to generators and read stuff from there?
if there are references to follow why cant the using generators then use this information

maybe based on a more hello worldy language,
and just use two languages

[cosminpolifronie]

Creating an example is way above my Xtext knowledge.

I am not sure what the misunderstanding is.
Lang5Generator::doGenerate uses static variables from class Lang1Generator which are populated when Lang1Generator::doGenerate is called. As such, I first need to parse the .lang1 file before being able to parse the .lang5 file.

It is not important what those variables store.

How would I make Maven respect this generation order? (.lang1 needs to be parsed before .lang5)

[cdietrich]

Usually you should be able to obtain the information you need in lang5 generator from lang5 generator without the need to store anything from lang1 or lang1 generator running before

this is why i asked you to be more explicit what you actually do store in lang 1 and read in lang 5

[cdietrich]

@cosminpolifronie any update here?

[cosminpolifronie]

Working on creating a minimal example to showcase the situation I'm in. It will take a bit, as I am not that familiar with Xtext.

What I have found out is that my proposed change would have not worked anyway, as the validation step does not call doGenerate. Even disabling the validation step and going directly for the generation of lang1 lang2 lang3 in the correct order doesn't currently work. I am not sure why, to be honest, but it doesn't matter.

I have reached the conclusion I need to make changes to the input files so that each is independently generable.
Maybe we could find a workaround after I finish the example, we will see.

I will get back to you.