Implement Registry Extension (RXT) 2.0 + Associated UI support - Project Documentation

Project Idea

The initial project idea was to Implement RXT 2.0, a JSON based model to replace the existing XML based RXT configuration. But due to the complexity and the time limitation of the project, the scope was narrowed down to create a Jaggery Application to Add a new RXT to WSO2 Governance Registry.

Overview

WSO2 Governance Registry is a SOA based registry/repository to manage, govern and store resources. These resources can be any artifact varying from a plain text file to a WSDL, service, URI, API, Schema, policy etc. The artifacts can be associated with lifecycle management, configuration governance, development process governance, design and run-time governance, team collaboration and provided with access control mechanisms.

Configurable Governance Artifacts are one of the many ways of extending the functionality of WSO2 Governance Registry to support modeling any type of asset to suit the user requirements. The RXT model provides an extensible configuration language to create new types of artifacts.

The main objective of this project is to provide enhanced usability when configuring new types of artifacts and introduce a new intermediate JSON based RXT format which can be used when WSO2 Governance Registry is migrating to a JSON based RXT format.

Design

The basic design includes generating a JSON based RXT using the user input and send it to the Registry backend and convert it to the existing XML based RXT format and use it without breaking the current XML based RXT implementation. To achieve the task I came up with the following design. In the Jaggery Application ‘ArtifactBuilder’ user input is read and basic input validation is done. A JSON based RXT is generated from the user input. ManageGenericArtifact Admin Service is called via Jaggery and JSON is sent to the Registry backend. The JSON -> XML conversion is handled by a Util method and finally via the generated XML, RXT, the existing code is executed to add the new Artifact Type. Since the Newly generated JSON based RXT is sent to the Registry, it can be persisted in the registry at a later point of time as well. The diagram below shows a design of the proposed Application.

Implementation

The main Technologies used to create the Jaggery Application are, Bootstrap, Basic HTML/CSS, Javascript, Jquery, JSON and for the communication, Jaggery-JQuery. Bootstrap 2 is the main UI framework used in the Jaggey Application Implementation. To add the drag and drop functionality Bootsnipp Form builder was used [1] and to add/delete dynamic table rows the ‘Dynamic table row creation/deletion’ snippet was used [2]. Both of the snippets comes under the MIT license.

When user logs in to WSO2 Governance Registry and navigate to the List Applications, they can click on the ArtifactBuilder Jaggery Application which will navigate to the below UI, which makes it easier to model their Artifact with only a few clicks without the hazel of editing a XML. The below image show part of the Jaggery application.

User can enter the basic artifact details, drag and drop the content ui elements to build the UI, then add the UI and Relationship information and click ‘Save Artifact’ which will do the input validation and generate a JSON based RXT model to be sent to the backend.

All the input is read from Javascript via the Document Object Model (DOM) and transformed into the JSON format.

I have created a new method addJSONRXTResource in ManageGenericArtifactService and exposed it as a admin service method so that it can be called directly by passing the payload of a RXT JSON. A new Util class was added in order to generate the conventional RXT XML using the JSON string. Inside addJSONRXTResource after generating the xml, addRXTResource admin service method is called and the newly generated XML based artifact is added without breaking the existing XML model.

In the Jaggery application, once the JSON is generated, the addJSONRXTResource is called via a Jaggery WS-Request [3] by passing the JSON string as the payload.

Sample JSON RXT

{  
   "artifactType":{  
      "hasNamespace":"false",
      "iconset":"18",
      "lifecycle":"",
      "nameAttribute":"overview_id",
      "nameSpaceAttribute":"",
      "pluralLabel":"people",
      "shortName":"person",
      "singularLabel":"person",
      "storagePath":"/people/@{name}",
      "type":"application/vnd.wso2-person+xml"
   },
   "content":{  
      "Overview":[  
         {  
            "helptext":"help",
            "id":"id",
            "label":"ID",
            "placeholder":"id",
            "required":"checked",
            "type":"text"
         },
         {  
            "helptext":"help",
            "id":"name",
            "label":"Name",
            "placeholder":"name",
            "required":null,
            "type":"text"
         },
         {  
            "id":"gender",
            "label":"Gender",
            "options":"Male\nFemale",
            "type":"option"
         },
         {  
            "id":"address",
            "label":"Adress",
            "type":"textArea"
         }
      ]
   },
   "relationships":[  
      {  
         "relationship":"Association",
         "source/target":"",
         "type":""
      }
   ],
   "ui":[  
      {  
         "href":"@{stroragePath}",
         "type":"path",
         "value":"overviewa_id"
      },
      {  
         "href":"@{stroragePath}",
         "type":"path",
         "value":"overview_name"
      }
   ]
}

Jaggery App : https://github.com/ireshapm0/ArtifactBuilder/commits/master
Carbon-governance : https://github.com/ireshapm0/carbon-governance/commits/dev-rxt-json

The below screencast shows the basic end to end functionality of the Application when adding a new Artifact Type.

Future Work

• Multiple Table support for dynamic <content> section. 
• Minimize user input effort by adding auto-complete data 
• Generate UI from RXT - Edit RXT UI 
• Finalize content type fields and their usage

References

[1] http://bootsnipp.com/forms
[2] http://bootsnipp.com/snippets/featured/dynamic-table-row-creation-and-deletion
[3] http://jaggeryjs.org/documentation.jag?api=ws

Use Jaggery WS-Request to call Admin Services of WSO2 Products

During the project I had to call certain admin services's methods of WSO2 G-Reg 5.0.0. using jaggery application. Following are the steps I went through to discover available admin services exposed to outside in Greg. Since this procedure would be the same for any of WSO2 servers ( e.g. WSO2 ESB, GReg, IS, CEP, BPS, ES, AS, BAM, etc.) I will try to make this post as general as possible.

To find out the available Admin services:

1. Open “wso2greg-5.0.0-SNAPSHOT/repository/conf/carbon.xml” file

           For the general case it would be : <server_home>/repository/conf' directory

           and then open carbon.xml

2. Setting HideAdminServises property to 'false', will allow us to view the admin service's wsdl.

       (Not mandatory - If you are not willing to view the wsdl of the service in browser, you        can left out this step)

<HideAdminServiceWSDLs>false</HideAdminServiceWSDLs>

3. Start the server with the OSGIConsole.

./wso2server.sh -DosgiConsole

4.Once the server is started we can view Adin servoces :

                  Press enter & type 'listAdminServices'

osgi > listAdminServices

Part of the list of admin services of GReg 

5. To check the wsdl of the admin service in the browser:

                Add ?wsdl to the url at its end.

HTTP 403 error is thrown if we have not set the hideAdminServiceWSDLs to false.

wsdl of a sample admin service

To use Soap UI to test the services:

Soap creates a Soap Envelope and call the given endpoint. Basically it does the job of a Soap client.

1. Download and install SoapUI at http://www.soapui.org/

                       I.  Go to File > New SOAP Project.

                       II.Paste the wsdl url in the 'initial wsdl' box and click OK.

SoapUI 5.0.0- Add new SOAP Project

2. Click on the Admin Service method from which you want to call from SoapUI and try sending a request.

We have to set Authentication and security related settings to access the admin service.

                      I.   Click on 'Auth' below and click 'Add New Authorization'.

                      II.  Select type Basic and set the username and password. ( Default username                                                            and password for every WSO2 Product would be 'admin').

                      III.Insert the values of the parameters and click the submit request button.

SoapUI 5.0.0 Successful request to an endpoint

If everything went well, we will end up with successful response.

In the response envelope check the return tag to see if the response we are getting is correct.

To call an Admin Service using Jaggery WS-Request:

We don't have to find out available admin services and to use SoapUI to test services if Admin Service endpoint URL and the action that should be called. Basically we can call the same admin service method uisng Jaggery WSRequest.

Jaggery documentation can be found at http://jaggeryjs.org/documentation.jag?api=ws

Jaggery code to call the above Admin service.

<%
var rxt = "JSON RXT Config Should go here";
var log = new Log();
var ws = require('ws');

var addRxt = new ws.WSRequest();
var options = new Array();
options.useSOAP = 1.1;
options.action = "urn:addJSONRXTResource";
var payload = '<ser:addJSONRXTResource xmlns:ser="http://services.generic.governance.carbon.wso2.org"> <ser:jsonRxtConfig>'+rxt+'</ser:jsonRxtConfig> </ser:addJSONRXTResource>';
var result;

try {
    addRxt.open(options, "https://localhost:9443/services/ManageGenericArtifactService", false, "admin", "admin");
    addRxt.send(payload);
    result = addRxt.responseE4X;
} catch (e) {
    log.error(e.toString());
}
   print(result);
%>

Action:Method to be called from the Admin Service

Payload:Actual request payload to be sent to the endpoint.

Specify the name servers correctly when sending the payload. WSRequest's open() will open the connection with the endpoint with necessary authentication and then sends the payload. Result will get the response Soap Message in which it will print in the page itself.

Creating XML file from the General JSON

I have written a Java class with following methods to create XML from the JSON generated by the UI. Here I have used the GSON library, added it as a dependency to my project .

• parse(s)
• artifactDetailstoXML(artifactDetails)
• UItoXML(ui)
• contentTtoXML(content)
• relationshiptoXML(relationship)
• writeToFile(name.getAsString(),XML)

parse(s) method

Is used to create JSON model from JSON.

JsonParser jsonParser = new JsonParser();

JsonElement rxtJsonElement = jsonParser.parse(jsonContentAsString);

JsonObject rxtJsonArtifactTypeObject = rxtJsonElement.getAsJsonObject().getAsJsonObject();

Then passed the JSON file as String content(jsonContentAsString) to parse method.

JsonObject is returned from parse method and that object is used to retrieve the hierarchical information that is needed to create the xml file.

artifactDetailstoXML(artifactDetails) method

Assumption was made that all mandatory fields are filled and only checked if user has input data to namespace, namespaceAttribute and lifecycle fields.

UitoXML(ui) method

Checked if the there are more that one column data exists and if it does JsonArray was created accordingly.

Separate JsonObjects are created for artifactDetails, ui, content and relationship and passed to different methods to create XML content accordingly. Advantage in creating separate XML sections is that it gives flexibility to make changes easily in separate methods.

Then the xml contents returned from each of the methods are added together and created the whole XML content and passed to writeToFile() method to be written to a .rxt file.

JsonObjects, JsonElements, JsonArrays were used to create XML.

JSON generation from UI is halfway done. But I am in difficulty in generating JSON for dynamic UI part and now working on it.

Update after mid review

Last week I met with WSO2 Governance registry team and I demonstrated the current progress of the Jaggery Application I have developed. We, my mentor and some experienced people in the team then discussed on the project scope and my approach so far and beyond.At the end of the session I was able to clarify my doubts on the problems I faced so far and on future tasks.

As I mentioned in my last update several issues were recorded in the existing RXT , XML model, so there were difficulties in rendering the UI (in ES) only with the information it is providing. The necessity of keeping other meta data along with the xml model was quiet apparent. To overcome this drawback , it was decided to generate a JSON along with sufficient meta data, such that UI rendering can be done without much effort. And then XML which suits the existing model is generated from that JSON to be sent to the stub.

Also I was able to clarify some doubts regarding the RXT definition whether certain data in rxt. sxd, which were apparent to be unused should be considered in the design or not. i. e date,calendar, href,menu,column. Discussions further went on regarding the capability of implementing nested rest support where user can associate rxt to each other on the fly. So that rxt s can be associated automatically.

Ability to add multiple tables and multiple lists in content, delete preferred rows in Relationship and UI sections are discussed in the meeting too. As enhancements it was discussed to consider the ability to populate a browse button to find relationship source /target.

These days I am working on creating JSON from the UI and retrieving the XML from that to be sent to the server side. I am writing a Java class to create XML from JSON, and JSON will be generated from UI in JavaScript.

Herewith I have drawn the updated overall architecture.

Project: Implement Registry Extension (RXT) 2.0 + Associated UI support - Updates and Notes

This is an update of  my project so far in GSOC 2014. There was a slight change in the project scope since the JSON support for RXT was temporarily removed and the scope was narrowed down to creating a Jaggery app which could support creating a new artifact type with an intuitive drag and drop UI.

I had a meeting with my mentor and few others in WSO2 last week and below are the facts which were discussed.

• Finalized the requirement of a Jaggery app to create a new artifact type which could be installed to either Greg or ES via Management Console
• RXT <content> should be generated using drag and drop components and user shall be able to change the fields easily by dragging components here and there.
• The RXT XML should be generated in the client side using JS as well as some metadata should be kept with UI fields in order for ES to generate its ‘add metadata’ model.
• This model should also facilitate creating a JSON out of the UI, at some point of time.
• Once the XML is generated, a backend call will be made to Registry to add the RXT via Jaggery.

Current Progress

In the Process of finding a suitable plugin for drag and drop functionality, I looked upon Gridster.js [1] was used in UES and is very flexible in generating UIs. But it seems to be bit difficult to use it in this purpose, since it needed a lot of fine tuning to be able to cater this requirement. Then I found Bootsnipp Form Builder [2] which seems to be bit similar to the one which is required under MIT license and built on bootstrap. So decided to go on with it.

I was able to create a simple UI, create a Jaggery app and host it in Greg/ES and test the functionality. Below is a sample UI which has only the basic functionality to create a RXT.

                                           

I was told several issues the existing RXT, XML model has and the difficulties in rendering the UI (in ES) only with the information it provides and necessity of keeping other metadata along with the xml model. So I’ working on a way to find out the best possible way to bridge the gap between the RXT xml and the UI it is generating in ES side.

Next Steps

• Write a JS script to generate the RXT xml from the form data/HTML
• Keep other metadata which are not going in to the xml in a JSON model (tentative)
• Call backend Registry client/stub and send the RXT to registry.
• Try to load RXT xml data to the UI back when editing a RXT artifact.

Github Repo : https://github.com/ireshapm0/ArtifactBuilder

[1] http://gridster.net/

[2] http://bootsnipp.com/forms

JSON Parsing and Validation

Step I - JSON Parsing

To parse JSON data we can use  GSON library.

http://google-gson.googlecode.com/svn/trunk/gson/docs/javadocs/com/google/gson/JsonParser.html

When given the json file as a String to parse method above, it will return the parsed JsonObject as tree.

public static JsonObject parse(String jsonString) {

JsonParser jsonParser = new JsonParser();

JsonElement rxtJsonElement = jsonParser.parse(jsonString);

JsonObject rxtJsonArtifactTypeObject = rxtJsonElement.getAsJsonObject().getAsJsonObject("artifactType");

return rxtJsonArtifactTypeObject;

}

Step II - JSON data format Validation

Develop Shema

JSON Schema is a tool for validating the structure of JSON data. When we consider data format we need to know what does data elements mean and what the valid inputs for these elements are. To ensure we have obtained the required data we write a schema.

I.Manually write Schema for JSON:

There are certain keywords that are used to develop a Schema for any sort of json data.

They can be found at http://tools.ietf.org/html/draft-zyp-json-schema-03 .

Most frequently used key words are:

Ø  type

Ø  properties

Ø  items

Ø  required

Ø  maxItems

JSON is built on two structures:

Ø  A collection of name/value pairs commonly realized as “objects”.

Ø  An ordered list of values commonly realized as “ arrays”.

This defines the type keyword in a Schema.

properties are defined to validate the attributes in an object.

Within an array the list of items are defined by items keyword.

Whether a particular object array or property and item is a must for it to validate is defined by required keyword. 

MaxItems defines the maximum number of elements in array type of object type.

Similarly you can find how any of this keyword defines the validity of the JSON data against the Schema we develop.

Example:

JSON data to be validated:

{

"address":{

"streetAddress": "21 2nd Street",

"city":"New York"

},

"phoneNumber":

[

{

"type":"home",

"number":"212 555-1234"

}

]

}

Schema developed:

{

"type":"object",

"required":false,

"properties":{

"address": {

"type":"object",

"required":false,

"properties":{

"city": {

"type":"string",

"required":false

},

"streetAddress": {

"type":"string",

"required":false

}

}

},

"phoneNumber": {

"type":"array",

"required":false,

"items":[

{

"type":"object",

"required":false,

"properties":{

"number": {

"type":"string",

"required":false

},

"type": {

"type":"string",

"required":false

}

}

}

] 

}

}

}

  

Reference: http://davidwalsh.name/json-validation

II. Directly generates Schema for specific parsed JSON data format:

http://www.jsonschema.net/#

Step III - Validate functionality.

We can directly validate Schema functionality online.

http://json-schema-validator.herokuapp.com/

http://jsonschemalint.com/

Note:

To validate JSON content - http://jsonschemalint.com/

Update on 11.05.2014 -18.05.2014 week

As far as I understand the project by now, I can divide it into following tasks:

Task 1:

Currently XML is read and UI is rendered using UIGenerator class. I have to do it using

the same flow(with minimal affect to existing code) and read a JSON to render a similar UI.

So I am looking at the governance. generic and governance. generic.ui classes. Now I am getting much familiar with code and I would like to know whether these are the relevent code segments I am referring at the moment.

First I hope to create and test new Java classes to replace Util ,GenericArtifactsUtil and Manage GenericArtifactsService to do novel changes without breaking the current model.

As a milestone I will try to plug JSON based rxt to existing model and validate and parse it within 2 weeks time.

For now it is the main thing I am working on.

Task 2:

Drag and Drop UI

I have run the G-Reg product and did some experiments on how to create new artifacts.

In the existing system user has to create the XML file according to his requirements and upload it in order to configure new artifact. User has to be familiar with coding in order to make use of this system.

After looking at artifact configuration XML files I got an idea on a general artifact structure.

If drag and drop UI is implemented what is going to be drag drop elements and what is going to be required information.

I designed simple UI for that drag drop UI interface and it's elements and still I am searching for some kind of drag and drop UI framework with Apache version 2.0 license to cater this, where user doesn't need to be much familiar with coding.

http://aribaweb.org/Home.htm seems somewhat relevant to it but I still go through its documentation and demos and not having an exact idea on its usability for this project.

I hope it would be better if I get to know any of such drag drop UI component available in WSO2 carbon and some people working in those areas to get some guidance. I will carry on searching about this in between task 1.

Task 3:

Create the JSON using a UI template which we construct using drag and drop sort of UI.

For this task I have to first look upon task 2. So I am not going to search more about it since it mostly depend on how I going to enhance user experience through task 2.

Also I understand how to build the individual components and finally how the whole product is built in order to create wso2greg-4.6.0.zip. That is what we run as the product in the browser. I was new to ubuntu and now I am gradually getting familiar with it. To understand the code and it's functionality I have to get familiar with remote debugging.

Remote debugging:

To start Carbon server with remote debugging enabled type:

./wso2server.sh -debug 5005

in the terminal.

before enter Create debug pointer in code and configure a new debug window. Then enter the above command and start debugging.

I hope the knowledge and the understanding I got within last weeks about WSO2 Carbon, XML to JSON conversion and JSON parsing and validation would help me to fulfill tasks, with not much difficulty.