A model-driven solution to automate the configuration, evaluation and synthesis of recommender systems for modelling languages.

About

Droid is a framework that automates the configuration, evaluation and synthesis of recommender systems for modelling languages. The framework supports:

  • The Customisation of recommender systems for a particular modelling language
  • Assistance on the decision of which recommendation method works better for the recommendation task
  • Generation of customized recommendation services that can be integrated with external modelling tools

Droid is composed mainly of an Eclipse Textual Domain-specific language (DSL) that is the hearth of the Droid configurator. The configurator allows, via the DSL, the configuration of RSs for a particular modelling language, from preference settings to full performance evaluation. The selected recommendation method is deployed as a service; which then heterogeneous modelling clients can integrate.

Currently, we provide automated, out-of-the-box integration with the tree editor of the Eclipse Modeling Framework (EMF). But additionally, the generated recommenders can be integrated with other modelling technologies.

Get started

The goal of this tutorial is to present an introduction to Droid through a simple example of building a recommender for class diagram for UML models where attributes, operations and super classes are recommended to be added to classes.

This tutorial will guide you through the necessary steps to create a Droid project; define a set of pre-processing techniques and recommender systems to generate and evaluate. It will show you all the features that Droid provides for the automatic pre-processing, configuration, evaluation and synthesis of recommender systems for modelling languages.

This tutorial is designed for recommender systems developers and users of modelling tools that are interested in generating tailored recommender systems for arbitrary languages. Before proceeding with this tutorial you should have a good understanding of the Eclipse Modelling Framework (EMF). If you need more information regarding EMF you can find it here. A basic understanding of Eclipse IDE is also required because the examples have been compiled using the Eclipse IDE.

Creating a new Droid Project

To create a new Droid project first select File → New → Other… from the main menu. The dialogue will offer different project types. Select New Droid Project from the category Droid and continue via Next.



The wizard will guide you through the new project creation process. It will require you to introduce the project name and to select the format of the meta-model/models of the language for which the RS is being developed (Ecore/XMI, Ecore/Ecore, or Ecore/UML). Additionally, an optional checkbox offers a default set of recommendation setting options.

Here we set the name to Education recommender, selected the checkbox for default values for recommendation setting and chose Ecore/UML for the technology.



With this initial information, the wizard will require you to provide the necessary data for the training and testing of the recommender systems. If Ecore/XMI option was selected, the wizard will provide you with a meta-model loading page to select the Ecore file. Afterwards, another page will allow you to choose the folder containing the XMI model instances conforming to this particular meta-model. Conversely, if the Ecore/Ecore or Ecore/UML has been selected, the wizard will only requered you to select the folder containing the Ecore meta-models/UML models for the training as the meta-model for these cases is automatically registered by Droid. Here a folder containing 1,051 UML models in the education topic was chosen.



The Droid project is created and the editor is shown with the default setting configuration provided by the wizard of Droid. It describe the class subject to recommendations, called Target; the items to be recommended; the pre-processing techniques; and the recommendations configuration settings. This last one includes the splitting configuration, recommendation methods and evaluation protocol.

The following image presents a pre-configured Droid project. In the editor, you can define the target and items for the recommender systems, as well as the identifiers for each class. You can also modify the proposed setting or leave it with the provided default values. In addition, the Droid IDE provides code completion and code validation.



In this example, we have defined the class Class as target. For the items we defined ownAttribute with the alias "attribute", ownOperation with the alias "methods" and superClass with the alias "super classes". Additionally, we used as primary key and feature the name for the class Class and name and type for Property and Operations. The class Type is also defined and identified for the name. This class is necessary as type is not a primary type.



Defining the data pre-processing configuration

For the data pre-processing configuration each option is a list and Droid computes each combination. We specified special character removal as true, the edit Levenshtein distance merging number 2, 3 and 4, and for minimum ratings per items and per target the numbers 1, 2 and 3.



After we finished defining the pre-processing functionalities, we save our project and press the button Pre-Process. In the Droid Pre-Processing view, the result is shown in 4 different sections. In the first one, the data section, the statistics for the data provided is shown. Then, in the target/items section, the information regarding the target and items is presented. In the third one, the Settings section, each configuration combination is shown with the details. And finally, in the last section, pre-processing results section, the results of applying each particular definition is shown. The configuration options are ranked in order of relevance.



Defining a set of recommender systems to generate and evaluate

The recommendations section is divided into 3 parts. In the first one, Split, the configuration for data splitting is set. Here we defined a cross-validation type with a 10 nFolds following a per user. In the second, Methods, the different algorithms we want to try are defined as well as the neighbourhood sizes. Here we included each method supported by Droid and the neighbourhood sizes 10,20,25,50 and 100. Finally, the Evaluation part includes the metrics we want to specify to evaluate the recommenders, the cutoffs values, the maximum number of recommendations, and the relevance of thresholds. Here we specified each metric supported by Droid; cutoffs values of 5,10,15 and 20; maximum recommendations of 5; and relevance of threshold of 0.5.

After defining the necessary information and saving the Droid project a set of Java classes are generated. With this information generated, and the selected pre-processing configuration setting, we proceed to press the button Train.



Visualizing the results

Finally, The Droid Training Results view is automatically populated with the evaluation results in a drill-down table for each recommendation method and metric. The results are grouped by the recommendation methods categories: Collaborative Filtering, Content-Based and Hybrid. The table makes use of different colours to facilitate the comparison of the metric values based on the metric F1. The recommendation methods with results in the top 20% are shown in green; the ones under the median in red; and the rest are shown in orange. After double-clicking the prefered method a pop-up dialogue is shown asking if we want to deploy that particular recommender. After we press yes the selected method is deployed by Droid into a generic REST service called DroidREST.

Building recommender systems for modelling languages with Droid

Publications

  • Automating the Synthesis of Recommender Systems for Modelling Languages. 2021. Lissette Almonte, Sara Perez-Soler, Esther Guerra, Ivan Cantador, Juan de Lara. ACM SIGPLAN International Conference on Software Language Engineering (SLE)
  • Recommender systems in model-driven engineering. 2021. Lissette Almonte, Esther Guerra, Ivan Cantador, Juan de Lara. Software and Systems Modelling (SoSyM)
  • Towards automating the construction of recommender systems for low-code development platforms. 2020. Lissette Almonte, Ivan Cantador, Esther Guerra, Juan de Lara. LowCode@MoDELS