Building a Large Chart Ecosystem with AnyChart and Native XML Databases September 22nd, 2011 by Timothy Loginov
Today we are glad to present you a detailed case study: Building a Large Chart Ecosystem with AnyChart and Native XML Databases by Dan McCreary.
Dan McCreary is an enterprise metadata strategy architect who has worked in the technology field for over 17 years. His career has taken him through the fields of solid state physics, computer chip design, supercomputing, working with Steve Jobs and NeXT Computer, and founding his own consulting firm — Integrity Solutions, which is now part of the WAM!NET Professional Services division. In the fall of 2000, Dan McCreary in collaboration with several other technology and marketing professionals formed Dan McCreary and Associates — a technology strategy development consulting firm specializing in helping firms align their business and technology strategies by effectively managing their metadata.
The AnyChart system, when used within a native XML database, is ideal for building and maintaining a very large number of charts that can be quickly managed by non-programmers. Central to the AnyChart product is the declarative nature of XML files and the focus on “what” should be displayed, but not “how” the charts are rendered. This allows the target platform to change from Flash to HTML5 (currently in Beta) without any changes by the chart authoring system. AnyChart’s domain specific language for chart specification can also be easily extended for business specific tasks. This approach has saved our customers tens of thousands of dollars building interactive dashboards as well as much higher quality and consistency of the final products.
This document shows how functions such as validation, search, quality, consistency, extensibility, and agility can be easily maintained using standardized, cross-platform XQuery libraries. This is just one example of how NoSQL (not only SQL) systems are ideal for managing semi-structured data.
There are many charting software packages that are designed to create a simple bar charts or line charts on the web. But most of these systems break down once you need to create multiple charts for multiple users in multiple business units and yet still retain consistency and quality across hundreds or even thousands of charts. Our experience is that AnyChart, when combined with native XML databases, can allow large complex charting systems to be easily maintained by non-programmers.
By using AnyChart system and native XML databases, we have created a large complex ecosystem where even very large numbers of charts can be quickly and easily managed. The secret is to combine the declarative nature of AnyChart XML files with a native XML database that can easily query all the charts in use based on a variety of search parameters.
This case study will show that when combined with a native XML database, AnyChart systems can be very easy to manage by non-programmers.
We now discuss several aspects of managing a large number of charting systems:
- Versioning and Differences
Our focus in each of these topics is to demonstrate how each feature saves organizations time and funding and makes the chart development process more flexible and agile.
There are two steps we use to check the format of our charts. We use XML editing tools such as the oXygen XML editor to enter chart data and the native XML databases to store our data.
Most XML database systems come with a single line function to validate an XML file against a set of rules specified in an XML Schema file. By creating internal standards for charts based around an XML Schema file, we can quickly validate the structure of all elements within a chart.
We have also used XQuery scripts to extract just subsections of all charts of specific types for validation. For example, we can extract all axes for time-series charts for consistency across many different chart authors and many lines of business.
Most native XML databases have integrated search functions. Some also make it easy to integrate fulltext keyword search for any text within an XML document. These features allow us to build highly specific search functions that have high quality search features that can be easily customized by non-programmers. Because XML description tags can be embedded directly into AnyChart files, this means that even full-text keyword searches can be used with AnyChart.
Central to the measurements of search quality is the improvement of both search precision (getting the documents you want) and relevancy (avoiding hits on the documents you are not interested in finding). By using customized search forms that allow us to do keyword search as well as adding element-level specification, we have found that our users can find the right sample chart usually in the first 10 search results. This means that they do not quickly give up and start creating a new chart from scratch that has already been created. This feature is central to our high-reuse strategy and lower cost per new chart development.
AnyChart provides a large amount of high-quality on-line documentation and example charts on its own web site that can be searched with search engines like Google. However, this documentation is not integrated into our local XML content management system, and the search tools frequently do not find the appropriate documents. By integrating documentation on requirements, testing, quality, and policy into a single search engine, all the related documentation can be found using a single search function.
There are many aspects to chart quality that should be reviewed by a human factors and user experience team. In the past many of these checks were manual and could only be done after charts were being used in a production setting. For example, many charts are used to find specific patterns in large data sets. Studies show that extra “chart ink” prevents this pattern recognition process. Disabling chart borders and grids should be automatically checked by the system. These checks can now be done automatically by the system as the chart is being developed.
Large organizations may have a large number of business rules to help users see consistent charts in consistent ways. For example, an organization may decide that all time-series charts should display dates across the horizontal axis using a 90 degree rotation for the date text.
To perform this check, a simple XQuery can be quickly created using a template XQuery application. The user only needs to have a sample consistency check template and have an XPath expression to extract the correct data.
Adding additional user-defined value columns is very complex in many relational database systems. But with AnyChart XML format we can add custom XML tags directly inside of AnyChart XML files without disrupting chart behavior. So, for example, we can add additional chart metadata that describes who requested the chart, who approved the design, when the chart went into production etc. directly in the Chart XML format. This allows us to store all of our chart data in a single document in our version control systems.
As we created advanced chart libraries, we found that many charts share common elements. These static elements can be placed in small XML files and then “assembled” using an XQuery based include process. In many cases the XML fragments also needed to be parameterized with elements such as custom titles and labels. XQuery functions allow the user to pass these parameters in using a simple inheritance process, much like object oriented systems. This allows each chart to be “contextualized” based on how it is being used by a project and supports high-variability charting systems as well as reuse of existing charts.
Many articles have been written about the need for business to quickly respond to changing business requirements. We have found that the combination of AnyChart and XQuery allows our team to quickly add new functionality or change functionally without re-building our core framework of tools. Our approach also makes it easier for new staff to quickly become productive without having to learn the details of how these systems work. Our business analyst’s focus much more on what the users want and leave the details of how the charts are rendered to the AnyChart application.
Many XML systems also integrate XForms as a key component of updating XML data stores. Using XForms and native XML systems allows us to quickly build forms for gathering new chart requests and updating chart development status. The combination of XForms, Rest interfaces, and XQuery back-end databases is called XRX. This simple clean architecture avoids many of the complexities of using objects and relational databases and is considered one of the key factors that allow a higher participation of non-programmers. We are still early in our use of XForms, but we feel that in the future many chart “specifications” will be generated directly from end-users using XForms applications.
Versioning and Differences
Because XML files can be “versioned” directly within XML databases, we can also easily store the AnyChart specification as well as the chart metadata directly in a single file and then compare the versions of these files using visual “difference” tools that are part of our XML development environment. We can also run detailed reports with XQuery that show the time span between a chart request and when this chart went into production to measure our responsiveness.
Many industries have strict reporting requirements about what events occurred in the software development lifecycle. For example, an auditor might want to know who required a specific feature, when the feature was first requested, when it was approved, who approved it, and what charts used these features. In the past these audits would take many weeks of research to complete and were very expensive. Our current system automates all these reports using XQuery reports on AnyChart files as well as joins to our software requirements management system that also is stored in the same native XML database. These features have already saved our customers many hours and thousands of dollars in audit expenses.
Using AnyChart with native XML database and XQuery saves organizations thousands of dollars and allows them to be more agile in development of new charting applications. Central to this is the process of enabling non-programmers to play a larger role in the management of chart creation.
Please contact Dan McCreary if you have any questions: http://www.danmccreary.com/
- ^ XML database: an XML database is a data persistence software system that allows data to be stored in XML format. This data can then be queried, exported and serialized into the desired format.
- ^ XQuery: XQuery is a query and functional programming language that is designed to query collections of XML data.
- ^ XPath: XPath, the XML Path Language, is a query language for selecting nodes from an XML document.
- ^ XML schema: An XML schema is a description of a type of XML document, typically expressed in terms of constraints on the structure and content of documents of that type, above and beyond the basic syntactical constraints imposed by XML itself.
- ^ <oXygen/>: XML Editor is a multi-platform XML editor, XSLT/XQuery debugger and profiler with Unicode support.
- ^ XForms: XForms is an XML format for the specification of a data processing model for XML data and user interface(s) for the XML data, such as web forms.
- ^ Representational state transfer: Representational state transfer (REST) is a style of software architecture for distributed hypermedia systems such as the World Wide Web.
- ^ XRX: In software development XRX is a web application architecture based on XForms, REST and XQuery. XRX applications store data on both the web client and on the web server in XML format and do not require a translation between data formats.