Evolution of Core Integration Framework

Evolution of Core Integration Framework

Preface

From very beginning our company developed software solutions for small and medium clients. Besides development, we support and grow our products based on a new customer requirements. At some point we realized, that all products we develop share common core functionality, that was rewritten over and over using similar libraries and methodologies.  This led us to develop Core Framework, which became basis for most of our new projects. Every product we developed required some sort of back-end and configuration UI. Shared components include:

  • User management
  • Role and permission management
  • Client management
  • Authentication and authorization
  • Running tasks on schedule
  • Logging
  • Reporting
  • Scripting

Having those components ready drastically reduces time to market for new products.

Technology Stack

At that moment we invested our effort into identifying technologies that would allow us to build a development platform. The requirements included UI/UX, back-end, database and interface between back and front ends. As for back-end we naturally chose Java as it was cross-platform product with very rich ecosystem of frameworks and libraries. Choosing database was a bit tricky. Many of our projects required different DB engines, such as MySQL, Oracle, Informix, PostgreSQL and many more. We choose Hibernate ORM as interface layer between application and database, as it uniformly support most SQL variants.

We evaluated many UI/UX frameworks. Since our platform targets mostly administrative interfaces, main requirement was richness and flexibility of UI components. Out of many options, such as Sencha, Dojo, JQuery UI and other we selected DHTMLX framework. It had very extensive documentation and robust examples. Ability to start using components without steep learning curve was one of the most important criteria. API was consistent and it was fairly easy to integrate different components into layouts.

We weren’t very excited in writing Web MVC only back-end. Besides browser based UI, some of of our projects front-end was written in different languages, such as Java, .Net and C++. We wanted to have universal back-end that can inherently support various front-end technologies. Since we were using Spring Framework as a basis of our back-end architecture, we decided to add Hessian to become a common interface between native front-end and our back-end, since it has bindings to most available languages. Unfortunately it was no JavaScript bindings for Hessian. We had to find something that will work without writing additional code. Direct Web Remoting (DWR) was the answer. Just adding annotations on methods and classes we were able to make our services accessible from any JavaScript engine.

Having a robust technology stack containing Java, Hibernate, Spring Framework, Hessian, DWR and DHTMLX we were able to combine all this together into Core Framework.

Pixel Core Integration Platform

Core Integration Platform

User Interface

We had to extend DHTMLX and add some wrappers around components to reduce boilerplate code and allow them easily integrate with our back-end via DWR calls. In the end we got very powerful and easy to extend UI framework that is easy enough to use for standard cases and extensible enough to build complex single page interfaces.

As user interface was evolving, we ended up using 2U layout a lot. Where on the left side we have a list of entities and details of selected entity on the right. Unlike regular web pages on websites, administrative interfaces are better fit in one screen and user should be able to view all necessary information at once. When there is not enough real estate for all the details to fit into the right pane without scrolling we use tabs which allow us logically divide selected object’s properties and display only relevant information.

DHTMLX grid component provide powerful support for infinite scrolling. Thanks to this functionality we can effectively present data in searchable and scrollable lists with billions of rows.

Thanks to robust form validation in DHTMLX user can clearly see what is missing or has invalid data.

Multi-lingual Support

Many of our clients are located in non-English speaking parts of the world. Core Framework was designed from ground up to support multi-lingual interfaces. Administrators can select default language and specify additional active languages.

Each user can select preferred language which will be used in the interface when user logs in.

Not only labels and menus are localized but data could be entered in multiple languages as well. This could be seen in Contact Types example below.

English locale

Chinese locale

Interface and data localization is a very tedious job, requiring matter experts in specific language. In some cases time to market is critical and translation cannot be done in time. For such scenarios Core Framework provides integration with Google Translation cloud services. When cloud translation is enabled, interface will automatically call this service to translate missing sentences to target language. Otherwise text in default language will be presented.

Unfortunately, DHTMLX framework does not support RTL languages, and hence we are unable to provide interface localization for Arabic, Azeri, Dhivehi/Maldivian, Hebrew, Kurdish (Sorani), Persian/Farsi and Urdu. The Central Semitic language alone is used by about 1.7 billion people most of whom are in the Arab world. This is huge market that we would like to enter. We strongly encourage DHTMLX developers to consider including RTL support into their products.

Scheduling

Scheduled tasks support allow products based on Core Framework to execute tasks at specific times.

Historical overview of scheduled tasks allow administrator to view tasks side by side. DHTMLX scheduler allow to displaying tasks in such grid. Users  are able to analyze and tune long running tasks, move execution time to limit impact of heavy tasks on other system’s functionality.

Scripting Support

Existing products are tend to evolve and as new requirements arise we need to add more functionality into the systems. Adding a new module usually require system restart.  Some of our products were deployed in an environments where restarts could be either undesirable if it runs 24/7 or maintenance windows are scheduled at very inconvenient times. After analyzing change requests across many products we realized, that besides bug fixes, most new functionality requests involving adding something to UI screen, change localization or develop a new scheduled task.

Since application is running in Tomcat servlet container, updating UI written in JSP/JavaScript could be done on the fly and no restart is required.

Changing localization messages was a bit more involved. Initially we had all our localized messages in text files embedded into WAR archive. We moved all the messages into database and created service that reloads messages on demand. By doing so we got rid of redeploys related to localization changes.

Initially scheduled tasks were written in Java and adding or modifying tasks was a somewhat painful process. We decided to allow clients to develop their own tasks in scripting language. Initially it was JavaScript only, but with introduction of GraalVM more scripting languages became available. Now we need to develop tasks and processes in Java only when we need to optimize specific tasks and/or scripting support is not enough for task functionality.

Robust scripting functionality of Core Framework allows advanced users to develop custom tasks in JavaScript, Python, R and Ruby. Such scripts have access to all Core internals.

Core scripts are not necessarily executable. Core Framework uses Velocity scripts to generate notification emails in HTML and in user specified language. Some complex SQL queries could be stored as scripts and modified on the fly.

Infrastructure

Initially, we had our products running on one machine. In many cases we provided an administrative UI for much larger systems. This UI was storing data into shared database. For most applications it was enough, as downtime due to system crash or reboot was not critical. It was ok for admin page to be unavailable for couple of minutes. But we were growing and our projects were becoming more and more complex and demanding. We began to deliver complete solutions to larger clients. And our clients were stressing to have high availability and scalability in the products we deliver.

Next blog post will describe clustering architecture of Core Framework and pitfalls associated with developing distributed systems.