Monday, March 14, 2011

Implement toString with Xtext's Serializer

Xtext uses EMF to generate the model API of the abstract syntax tree (or graph) of a DSL. For all implementation classes, the toString() method is generated. For a simple model element, this default implementation returns a string looking similar to this:

my.dsl.impl.SomeElement@67cee792 (attr1: SomeValue)

Well, this is not too bad. However this looks completely different to my DSL syntax, which may look like this:

SomeValue { 
    The content;
}

Especially for debugging and logging, I prefer that DSL like output. Since Xtext does not only generates a parser for reading such a text, but also a seralizer for creating the text from a model, I was wondering if that mechanism could be used for the toString method as well. (Actually, Henrik Lindberg pointed out the serializer class -- thank you, Henrik!)

In the following, I describe how to do that. Actually, this is a little bit tricky, and it will cover several aspects of Xtext and the generation process:
  • use the generated serializer for formatting a single model element
  • tweak the generation process in order to add a new method
  • define the body of the newly added method

We will do that by adding a post processor Xtend file, which adds a new operation to the DSL model elements. The body of the operation is then added using ecore annotations. But first, we will write a static helper class implementing the toString method using the serializer.

Use the serializer

Xtext provides a serializer class, which is usually used for writing a model to an Xtext resource. The Serializer class (in org.eclipse.xtext.parsetree.reconstr) provides a method serialize(EObject obj), which returns a String---this is exactly what we need. This class requires a parse tree constructor, a formatter and a concrete syntax validator. Thanks to google Guice, we do not have to bother about these things. Xtext generates everything required to create a nicley configured serializer for us. What we need is the Guice injector for creating the serializer:

Injector injector = Guice.createInjector(new  my.dsl.MyDslRuntimeModule());
Serializer serializer = injector.getInstance(Serializer.class);

Now we could simply call the serialize method for a model element (which is to be an element of the DSL):
String s = serializer.serialize(eobj);

Since this may throws an exception (when the eobj cannot be successfully serialized, e.g., due to missing values), we encapsulate this call in a try-catch block. Also, we create a helper class, providing a static method. We also use a static instance of the serializer.
Since this helper class is only to be used by the toString methods in our generated implementation, we put it into the same package.

package my.dsl.impl;

import org.eclipse.emf.ecore.EObject;
import org.eclipse.xtext.parsetree.reconstr.Serializer;
import com.google.inject.Guice;

public class ToString {
 private static Serializer SERIALIZER = null;

 private static Serializer getSerializer() {
  if (SERIALIZER == null) { // lazy creation
   SERIALIZER = Guice.createInjector(new my.dsl.MyDslRuntimeModule())
        .getInstance(Serializer.class);
  }
  return SERIALIZER;
 }

 public static String valueOf(EObject eobj) {
  if (eobj==null) {
   return "null";
  }
  try {
   return getSerializer().serialize(eobj);
  } catch (Exception ex) { // fall back:
   return eobj.getClass().getSimpleName()+'@'+eobj.hashCode();
  }
 }

}

Post processing

Now we have to implement the toString() method of our model classes accordingly. That is, instead of the default EMF toString method, we want to call our static helper method for producing the String.

A generic solution, which can not only be applied for adding the toString method but for all kind of operations, is to use a post processor extension (written in Xtend) to add new operations to the generated ecore model. The overall mechanism is described in the Xtext documentation. We have to write an Xtend extension matching a specific naming convention: <name of DSL>PostProcessor.ext. In our exampel that would be MyDslPostProcessor.

The easy thing is to add a new operation to each classifier:
import ecore;
import xtext;

process(GeneratedMetamodel this) :
 this.ePackage.eClassifiers.addToStringOperation();

create EOperation addToStringOperation(EClassifier c):
    ... define operation ... ->
 ((EClass)c).eOperations.add(this);

For defining the operation, we need:
  • the return type of the operation
  • the body of the operation

The return type is an EString (which will result in a simple Java String). In EMF, we have to set the type via EOperation.setEType(EClassifier). That is, we need the classifier of EString. With Java, this would be no problem: EcorePackage.eINSTANCE.getEString().
Unfortunately, we cannot directly access static fields from Xtend. At least, I do not know how that works. Fortunately, we can substitute EcorePackage.eINSTANCE with calling a static method of EcorePackageImpl. This static method can then be defined as a JAVA extension in Xtend:

EPackage ecorePackage(): 
 JAVA org.eclipse.emf.ecore.impl.EcorePackageImpl.init();

Note that we return an EPackage instead of the EcorePackage. I assume this is necesssary because we use the EMF metamodel contributor and EcorePackage is not available then. We can now set the EString classifier as return type of the operation: setEType(ecorePackage().getEClassifier("EString"))

Now, we need the body of the operation. Ecore does not directly support the definition of a body, that is there is no field in EOperation for setting the body. Fortunately, we can exploit annotations for defining the body. The default EMF generator templates look for annotations marked with the source value "http://www.eclipse.org/emf/2002/GenModel". The key of the annotation must be "body", and the value of the annotation is then used as the body of the operation. In the body, we simply call our static helper method for producing the DSL-like string representation.

The complete post processor extensions looks as follows:
import ecore;
import xtext;

process(GeneratedMetamodel this) :
 this.ePackage.eClassifiers.addToStringOperation();

EPackage ecorePackage(): 
 JAVA org.eclipse.emf.ecore.impl.EcorePackageImpl.init();


create EOperation addToStringOperation(EClassifier c):
 setName("toString") ->
 setEType(ecorePackage().getEClassifier("EString")) ->
 eAnnotations.add(addBodyAnnotation(
  'if (eIsProxy()) return super.toString(); return ToString.valueOf(this);')) ->
 ((EClass)c).eOperations.add(this);

create EAnnotation addBodyAnnotation(EOperation op, String strBody):
 setSource("http://www.eclipse.org/emf/2002/GenModel") ->
 createBody(strBody) ->
 op.eAnnotations.add(this);
 
create EStringToStringMapEntry createBody(EAnnotation annotation, String strBody): 
 setKey("body")->
 setValue(strBody) ->
 annotation.details.add(this);

If you (re-) run the GenerateMyDSL workflow, the EMF toString() implementations are replaced by our new version. You can test it in a simple stand alone application (do not forget to call doSetup in order to configure the injector):

public static void main(String[] args) {
 MyDslStandaloneSetup.doSetup();
 MyElement = MyDslFactory.eINSTANCE.createElement();
 e.setAttr1("Test");
 e.setAttr2("Type");
 System.out.println(e.toString());
}


Closing remarks


You probably do not want to really replace all toString methods with the serializer output, as this would create rather long output in case of container elements. In that case, you can add the new operation only to selected classifiers, or use the (generated) Switch-class to further customize the output.

Although the solutions looks straight forward, it took me some time to solve some hidden problems and get around others:
  1. How to create the serializer using the injector -- and how to create the injector in the first place
  2. How to access a static Java method from Xtend without too much overhead. Would be great if static fields could be accessed from Xtend directly.
  3. How to use the post processor with the JavaBeans metamodel contributor. If I switch to the JavaBeans metamodel, my extension didn't get called anymore.
  4. I'm still wondering where "EStringToStringMapEntry" is defined. I "copied" that piece of code from a snippet I wrote a couple of months ago, and I have forgotten how I found that solution in the first place.
  5. Sorry, but I have to say it: The Xtend version 1.0.1 editor is crap (e.g., error markers of solved problems do not always get removed). But I've heard there should be a better one available in version 2 ;-)

6 comments:

Florence said...

Thank you very much for this post !
I tried in the past to implement toString methods like you do, but I gave up because I didn't manage to specify the return type of the operation. Now it works, thank you again.

Oliver said...

Great! Exactly what I need.

But I've got one problem: Enums...

EvaluationException : Couldn't find property 'eOperations' for type ecore::EEnum
IPAPostProcessor.ext[478,11] on line 19 '(EClass)c.eOperations'
IPAPostProcessor.ext[66,49] on line 6 'this.ePackage.eClassifiers.addToStringOperation()'

How can I exclude Enums from the post processing?

Jens v.P. said...

@Oliver: I figure you will have to change the entry method of the post processor extension. I simply process all classifiers (to keep it simple), but you can add a filter there:

process(GeneratedMetamodel this) :
this.ePackage.eClassifiers.addToStringOperation();

E.g., a type filter:

this.ePackage.eClassifiers.typeSelect(EClass).addToStringOperation()

Maybe a nicer solution would be to create a new entry method passing the classifier, and then override this method accordingly. E.g.

process(GeneratedMetamodel this) :
this.ePackage.eClassifiers.doProcess();

Void doProcess(EClassifier c):
addToStringOperation();

Void doProcess(EEnum c):
;

Didn't test these methods, but it should work like that.

Cheers,
Jens

Anonymous said...

Hey,

thank you for the nice job! Works like a charm :)

Beside, I had to replace the type Serializer with ISerializer in the ToString singleton class.

Thanks!

kon

Jon said...

This helped me, thanks.

It wasn't clear to me where to put ToString.java, but on rereading "Since this helper class is only to be used by the toString methods in our generated implementation, we put it into the same package." I realised it should go in (for example) src/org.xtext.example.mydsl.myDSL.impl.

I subsequently found http://christiandietrich.wordpress.com/2011/07/22/customizing-xtext-metamodel-inference-using-xtend2/?blogsub=confirming#subscribe-blog and comment #3 (August 10, 2012 at 13:50) which yields this solution:


// src/org.xtext.example.mydsl/MyDslPostProcessor.ext

import ecore;
import xtext;

// delegate to an Xtend2 implementation:
process(GeneratedMetamodel this) : JAVA org.xtext.example.mydsl.MyDslXtext2EcorePostProcessor.augment(org.eclipse.xtext.GeneratedMetamodel);


// src/org.xtext.example.mydsl.MyDslXtext2EcorePostProcessor.java

package org.xtext.example.mydsl

import org.eclipse.xtext.xtext.ecoreInference.IXtext2EcorePostProcessor
import org.eclipse.xtext.GeneratedMetamodel
import org.eclipse.emf.ecore.EPackage
import org.eclipse.emf.ecore.EClassifier
import org.eclipse.emf.ecore.EClass
import org.eclipse.emf.ecore.EcoreFactory
import org.eclipse.emf.ecore.EcorePackage
import org.eclipse.emf.ecore.EcorePackage$Literals
import org.eclipse.emf.codegen.ecore.genmodel.GenModelPackage
import org.eclipse.emf.common.util.BasicEMap$Entry
import org.eclipse.emf.ecore.impl.EStringToStringMapEntryImpl
import org.eclipse.emf.ecore.EAnnotation
import org.eclipse.emf.ecore.EOperation

class MyDslXtext2EcorePostProcessor implements IXtext2EcorePostProcessor {

// called by org.xtext.example.mydsl.MyDslPostProcessor.ext
def static void augment(GeneratedMetamodel metamodel) {
new MyDslXtext2EcorePostProcessor().process(metamodel)
}

override void process(GeneratedMetamodel metamodel) {
metamodel.EPackage.process
}

def process(EPackage p) {
p.EClassifiers.filter(typeof(EClass)).forEach[addToStringOperation]
}

def addToStringOperation(EClass c) {
val op = EcoreFactory::eINSTANCE.createEOperation
op.name = 'toString'
op.EType = EcorePackage::eINSTANCE.EString
c.EOperations += op.addBodyAnnotation('if (eIsProxy()) return super.toString(); return ToString.valueOf(this);')
}

def addBodyAnnotation(EOperation op, String strBody) {
val body = EcoreFactory::eINSTANCE.createEAnnotation
body.source = GenModelPackage::eNS_URI
body.createBody(strBody)
op.EAnnotations += body
op
}

def createBody(EAnnotation annotation, String strBody) {
val map = EcoreFactory::eINSTANCE.create(EcorePackage::eINSTANCE.getEStringToStringMapEntry()) as BasicEMap$Entry
map.key = "body"
map.value = strBody
annotation.details.add(map)
}
}

Jens v.P. said...

@Jon: Thank you very much for the Xtend2 solution!