Library Bundles for your Xtext DSL

Xtext offers powerful cross-reference mechanisms to resolve elements inside the current resource or even from imported resources. External resources can either be imported explicitly or implicitly. My goal was to provide a library for my DSL containing a number of elements which should be referencable from “everywhere”. Consequently the “header” files from my library must be imported implicitly on a global level.

Creating a Library Bundle

To create a bundle for your library, use the New Project Wizard in Eclipse and choose Plug-In Development -> Plug-In Project. In this way as OSGi-based bundle will be created. Create a new folder for your resources to be imported globally (e.g. headers) and copy your header files (written in your DSL) to this folder.

Registering Implicit Imports in your DSL Bundle

Global implicit import behaviour is achievable using a custom ImportUriGlobalScopeProvider. Create your class in your DSL bundle in the package <your DSL package prefix>.dsl.scoping and extend org.eclipse.xtext.scoping.impl.ImportUriGlobalScopeProvider:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
import java.util.LinkedHashSet;
import org.eclipse.emf.common.util.URI;
import org.eclipse.emf.ecore.resource.Resource;
import org.eclipse.xtext.scoping.impl.ImportUriGlobalScopeProvider;
 
public class MyDSLImportURIGlobalScopeProvider extends ImportUriGlobalScopeProvider
{
    public static final URI HEADER_URI = URI.createURI("platform:/plugin/my.dsl.library/headers/myHeader.ext");
    
    @Override
    protected LinkedHashSet<URI> getImportedUris(Resource resource)
    {
        LinkedHashSet<URI> importedURIs = super.getImportedUris(resource);
        importedURIs.add(HEADER_URI);
        return importedURIs;
    }
 
}

The method getImportedUris() is overwritten and extends the set of imports retrieved from the super implementation. Note that I used a platform:/plugin URI here, which is actually resolved sucessfully in an Eclipse Runtime Workspace. In the itemis blog article about global implicit imports classpath-based URIs are used. A disadvantage of classpath-based library resolving is that it does not work out-of-the-box. In fact you have to create a plug-in project in the runtime workspace and add a dependency to your library bundle in MANIFEST.MF manually. I successfully avoided this problem by using platform:/plugin URIs which are resolved as soon as the library bundle is present in the run configuration of the runtime Eclipse instance.

Now your global scope provider has to be bound in the runtime module of your workspace. Open MyDSLRuntimeModule and add the following binding:

1
2
3
4
5
@Override
public Class<? extends IGlobalScopeProvider> bindIGlobalScopeProvider()
{
    return MyDSLImportURIGlobalScopeProvider.class;
}

If you start your runtime eclipse with the library bundle now, your global implicit imports should be resolved in the editor for your DSL.

 Resolving Implicit Global Imports in Standalone Mode

If you rely on the global implicit imports in standalone mode (e.g. in unit tests executed in your development Eclipse instance) the platform:/plugin URIs can not be resolved. But this can easily be fixed by using URI mappings in a ResourceSet. The following example shows how to create a standalone parser by injecting a ResourceSet and creating an URI mapping:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
public class StandaloneXtextParser
{    
    @Inject
    private XtextResourceSet resourceSet;
 
    private boolean initialized;
    
    public StandaloneXtextParser()
    {
        super();
    }
 
    public EObject parse(URI uri) throws IOException
    {
        initializeIfApplicable();
        Resource resource = resourceSet.getResource(uri, true);
        return resource.getContents().get(0);
    }
 
    private void initializeIfApplicable()
    {
        // Note: this is not the most elegant way, just for demonstration purposes
        // Just make sure that setupParser() is called once before you parse
	if(!initialized)
        {
            setupParser();
	    initialized = true;
        }
    }
 
    protected void setupParser()
    {
        resourceSet.addLoadOption(XtextResource.OPTION_RESOLVE_ALL, Boolean.TRUE);
        registerURIMappingsForImplicitImports(resourceSet);
    }
    
    private static void registerURIMappingsForImplicitImports(XtextResourceSet resourceSet)
    {
        final URIConverter uriConverter = resourceSet.getURIConverter();
        final Map<URI, URI> uriMap = uriConverter.getURIMap();
        registerPlatformToFileURIMapping(MyDSLImportURIGlobalScopeProvider.HEADER_URI, uriMap);
    }
 
    private static void registerPlatformToFileURIMapping(URI uri, Map<URI, URI> uriMap)
    {
        final URI fileURI = createFileURIForHeaderFile(uri);
        final File file = new File(fileURI.toFileString());
        Preconditions.checkArgument(file.exists());
        uriMap.put(uri, fileURI);
        
    }
 
    private static URI createFileURIForHeaderFile(URI uri)
    {
        return URI.createFileURI(deriveFilePathFromURI(uri));
    }
 
    private static String deriveFilePathFromURI(URI uri)
    {
        return ".." + uri.path().substring(7);
    }
}

The trick is to add an URI mapping which resolves the platform:/plugin URI to a relative file URI. Then the library resources are resolved by means of a relative path in the workspace and the global implicit imports can also be used in unit tests. For more information on standalone parsing please read this blog article.

Other useful resources:

 

Standalone Parsing with Xtext

Xtext is an awesome framework to create your own domain specific languages (DSLs). Providing a grammar, Xtext will create your data model, Lexer, Parser and even a powerful Editor integrated into an Eclipse IDE including syntax highlighting and auto completion 🙂

In this blog post I want to summarize some of my experiences with the behaviour of Xtext using different parsing approaches. These are useful if you want to parse input with Xtext in standalone applications or in the Eclipse context in order to get a model representation of your DSL code.

Approach 1: Injecting an IParser instance

The first approach uses the IParser interface. In a standalone application (that means if your code is not running in an Eclipse/Equinox environment), a parser instance can be retrieved using the injector returned by an instance of <MyDSL>StandaloneSetup:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
public class XtextParser {
 
    @Inject
    private IParser parser;
 
    public XtextParser() {
        setupParser();
    }
 
    private void setupParser() {
        Injector injector = new MyDSLStandaloneSetup().createInjectorAndDoEMFRegistration();
        injector.injectMembers(this);
    }
 
    /**
     * Parses data provided by an input reader using Xtext and returns the root node of the resulting object tree.
     * @param reader Input reader
     * @return root object node
     * @throws IOException when errors occur during the parsing process
     */
    public EObject parse(Reader reader) throws IOException
    {
        IParseResult result = parser.parse(reader);
        if(result.hasSyntaxErrors())
        {
            throw new ParseException("Provided input contains syntax errors.");
        }
        return result.getRootASTElement();
    }
}

Using this approach, your parse result can be retrieved with only very few lines of code. However, it only works in standalone applications. If you execute this code in the Eclipse context, the following errror is logged:

java.lang.IllegalStateException: Passed org.eclipse.xtext.builder.clustering.CurrentDescriptions not of type org.eclipse.xtext.resource.impl.ResourceSetBasedResourceDescriptions
    at org.eclipse.xtext.resource.containers.ResourceSetBasedAllContainersStateProvider.get(ResourceSetBasedAllContainersStateProvider.java:35)

To resolve this, the injector has to be created differently, using Guice.createInjector and the Module of your language:

1
Injector injector = Guice.createInjector(new MyDSLRuntimeModule());

Now the parser works fine, even in Eclipse. But if you use references to other resources or import mechanisms, you will find that the references to other resources can not be resolved. That’s why you need a resource to parse Xtext input properly.

Approach 2: Using an XtextResourceSet

To parse input using resources, you inject an XtextResourceSet and create a resource inside the ResourceSet. There are two ways to specify the input:

  1. an InputStream
  2. an URI specifying the location of a resource

In my implementation there are two methods for those two alternatives, respectively:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
public class XtextParser {
 
    @Inject
    private XtextResourceSet resourceSet;
 
    public XtextParser() {
        setupParser();
    }
 
    private void setupParser() {
        new org.eclipse.emf.mwe.utils.StandaloneSetup().setPlatformUri("../");
        Injector injector = Guice.createInjector(new MyDSLRuntimeModule());
        injector.injectMembers(this);
        resourceSet.addLoadOption(XtextResource.OPTION_RESOLVE_ALL, Boolean.TRUE);
    }
 
    /**
     * Parses an input stream and returns the resulting object tree root element.
     * @param in Input Stream
     * @return Root model object
     * @throws IOException When and I/O related parser error occurs
     */
    public EObject parse(InputStream in) throws IOException
    {
        Resource resource = resourceSet.createResource(URI.createURI("dummy:/inmemory.ext"));
        resource.load(in, resourceSet.getLoadOptions());
        return resource.getContents().get(0);
    }
 
    /**
     * Parses a resource specified by an URI and returns the resulting object tree root element.
     * @param uri URI of resource to be parsed
     * @return Root model object
     */
    public EObject parse(URI uri) {
        Resource resource = resourceSet.getResource(uri, true);
        return resource.getContents().get(0);
    }
 
}

In both cases, the resource set is injected using Guice. Also, the Eclipse platform path is initialized using new org.eclipse.emf.mwe.utils.StandaloneSetup().setPlatformUri(“../”). The load option RESOLVE_ALL is added to the resource set.

If an InputStream is provided, the underlying resource is a dummy resource created in the ResourceSet. Make sure that the file extension matches the one of your DSL.

In case of a given resource URI, your resource can be parsed directly using resourceSet.getResource(). Using this approach, all references (even to imported / other referenced resources) will be resolved.

The Dependency Injection Issue

Still there is another problem because we use Dependency Injection here in a way which is not exactly elegant. The point is that a class should not need to care about how its members are injected. In a well-designed DI-based application, there is only one injection call and all members are intantiated recursively from “outside the class”. To learn more about this issue, please read this excellent blog post by Jan Köhnlein.

I hope this post was useful to you. Please feel free to share your thoughts / ideas for improvements.

Creating your own SuperCollider-based Standalone Application

Let’s build our own standalone application based on the ingenious sound synthesis engine and programming language SuperCollider! These instructions were created with SuperCollider 3.5.5 and are suitable to create a standalone app for OS X.

Here is what we need to do:

  1. Download a source distribution of SuperCollider from here.
  2. Extract the source files to a directory and rename SuperCollider-Source to the name of your app (e.g. MyApp)
  3. In platform/mac, create a copy of the folder Standalone Resources and rename it to MyApp Resources (replace with your app name).
  4. Inside your MyApp Resources, delete the file SCClassLibrary/SCSA_Demo.sc.
  5. Copy any custom classes that your app requires inside the SCClassLibrary folder.
  6. Edit the modifyStartup.sc file to define what your app does when it starts. Here you can remove the post window, for example. Make sure to comment or delete the lines where SCSA_Demo is initialized.
  7. (optional) Change the icon (SCcube.icns). Download the application img2icns, which converts any images into the OS X .icns format. Overwrite the old SCcube.icns file with your icon.
  8. (optional) Edit the file English.lproj/MainMenu.nib. Here you can adjust the menu located at the top of the screen of your application. A better alternative to add new menu items for your SuperCollider application (on Mac OS X) is a custom class adding the following startup instructions:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
MyClass {
 
    *initClass {
        if(Platform.ideName == "scapp") {
            StartUp.add {
                this.createMenu;
            }
        }
    }
 
    *createMenu {
        var menuGroup;
        menuGroup = SCMenuGroup(nil, "My Menu", 13);
 
        SCMenuItem(menuGroup, "My Item")
            .action = { "Item selected!".postln };
    }
}

Ok, now it’s time to compile and package your standalone application. Open a Terminal window and execute the following commands:

1
2
3
cd /path/to/MyApp
mkdir build
cd build

This will create a directory build in your application’s source directory. The reason we do this is that the build artifacts are cleanly separated from the source files. Now we configure the build process using cmake:

1
cmake -DCMAKE_OSX_ARCHITECTURES='i386;x86_64' -D standalone="MyApp" ..

These commands make sure your app is built with support for 32bit and 64bit architectures (-DCMAKE_OSX_ARCHITECTURES=’i386;x86_64′) and that the resources for your custom app are used (-D standalone=”MyApp”). The two dots at the end indicate that the source files are located in the parent folder, not in the build folder we just created.

If you don’t need QT in your app, you can omit it, which will result in an application bundle consuming much less space on the hard disk:

1
-DSC_QT=OFF

If everything goes fine, cmake will check for some components and in the end appears a message similar to

-- Configuring done
-- Generating done
-- Build files have been written to: /Users/name/Development/SuperCollider/MyApp/build

Now the build configuration is saved, consequently we won’t have to run the cmake command again fur future builds. Now we’re ready to compile our app.

1
make install

After the build process, which takes some time when executed for the first time, your bundled App is waiting for you in build/install/MyApp/MyApp.app. That’s it 🙂

For future builds, you might want to program a litte build script, which updates the resource files and executes the build automatically. A very simple example could be:

1
2
3
mkdir ../platform/mac/MyApp\ Resources/SCClassLibrary/SomeFolder/
cp ~/Library/Application\ Support/SuperCollider/Extensions/SomeFolder/* ../platform/mac/MyApp\ Resources/SCClassLibrary/SomeFolder/
make install

This script creates a target folder and copies a bunch of classes from your Extensions directory to that folder. Save it as .sh file in your build directory, and simply execute this if you want a new version of your App. For security reasons, the file might not be executable from scratch. In this case, you have to modify its permissions:

1
chmod 755 build.sh

Now your script should be executable. And now: have fun with your app!