Relational network data model

The relational model has lost its exclusivity

The requirements for the functionality and structure of databases (DB), which are most fully implemented in relational systems, are now under the pressure of new requirements.

The first problem is low efficiency for big data. Sources of big data are social networks, video surveillance systems, spatial sensors, billing, etc. A relational database (RDB) works well if the data schema is precisely defined in advance, before starting the application. But big data is inherently difficult to structure at the database design stage. Only as information is collected, gradually, is its structure more apparent.

The second - search, calculation of queries in RDB with huge tables - this is a task of high algorithmic complexity. The use of indexing and hashing works well in more or less static BDBs, which are largely filled before the system is put into operation. And in conditions of the rapid arrival of new data arrays in real time, the advantages of these methods are leveled, since overhead costs increase sharply.

The third drawback of the RDB follows from the stringent requirements for data circuits within the framework of the canonical "normal forms". The need for a huge number of a wide variety of applications requires significant efforts to create data models, and the uneven skill level of programmers and tight deadlines lead to errors that require corrections and alterations. But any change that is already “living”, filled with DBD (“migration”) is an even more complex and time-consuming task, which in some cases has no other solution at all, like completely replacing the old database with a new one.

The “beauty” and rigor of the relational model implemented in SQL, for 3 decades, delighted programmers. The "old" models: network or hierarchical were almost forgotten. Yes, there are almost no such software products, with the possible exception of the “almost immortal” IDMS [1].

In the last decade, active work is underway to create alternative database management systems (DBMS), which are so simply called - NoSQL. Under this concept very different systems now fall that are very different from each other. It is interesting that the "old" network and hierarchical models are not included in the concept of NoSQL! Good reviews in this area can be found in [2,3,4].

The NoSQL category includes “graph” databases [5], which are abstractly close to the canonical network model CODASYL [6]. As the name implies, such systems are two unorganized sets - nodes (vertices) and edges (arcs). The main advantage of network databases is that navigation is “determined” not at the moment of processing the request , as in the RDB, but at the time of adding new data (for graphs - vertices and edges), it is completely true for graph systems. But the graph database is not structured before it is populated, unlike the CODASYL database.

Other most popular NoSQL database classes are “key-value” (example - Redis [7]) and “document storage” (example - MongoDB [8]). Since a detailed review of such systems is not the purpose of this article, it is important to note only the following.

NoSQL systems, as a rule, operate on the basis of distributed file systems providing scalability and reliability [9]. But the problem that is mathematically rigorously solved within the framework of the relational model is the integrity and consistency of the database (provided, of course, professionally competent design of the normalized scheme) is not posed at all in most NoSQL systems.

In total, today the situation is approximately the following: 75% of the databases are relational, NoSQL in its pure form is used in highly specialized systems, and combinations of various database models are used in highly loaded global network projects: Google, Facebook, Instagram, WhatsApp and the like.

Relational databases without SQL

In addition to the practical problems described above, the use of RDBs has recently seen other important trends.

In addition to the sometimes excessive “rigidity” of the relational model, its major practical (not theoretical) drawback is the complexity of data manipulation. The first option is to use the pipeline of operations on sets - unification, intersection, filtering, etc. in practice, it is almost never used, since it is associated with the expenditure of colossal resources and is justified only with "batch" processing of sets of requests of the same type. The second option - the SQL interpreter requires high professionalism, good knowledge of set theory, database theory, and considerable practical experience.

Object-oriented programming (OOP) in languages ​​has become the standard, but SQL is a procedural language and is very "friendly" with OOP. As a result, the solution to the problem of matching application program code with SQL queries based on a class library called ORM (Object-Relational Mapping - Object-Relational Mapping (Transformation) [9]) has gained popularity.

Using ORM classes allows a programmer to do without SQL when using RDBs. ORM automatically generates SQL queries to the RDBs to create tables and manipulate data. Most ORMs have interfaces with various popular DBMSs - SQLite, MySQL, PostgreSQL and others, which gives a choice without modifying the program code.

There are a lot of ORM implementations, even several for each programming language. They are all similar, therefore, for definiteness, in the future, by ORM we mean the corresponding library (package) of models of the Model class of the Django framework [10] in the Python language [11].

ORM is very “convenient” and programmers don’t really think that using this API they get not only the advantages of the relational model, but all its disadvantages. For example, in the code itself, you cannot override table models — add or remove a column, add a new table, etc. To make the database migration, you must first rewrite the code, then "rise one floor higher", and then restart the program. As a result, it is impossible to create an application that provides even the simplest changes to the data scheme in the process of the program without changing the program itself.

Data retrieval in ORM is implemented using chains of methods, for example, “objects.all ()”, “objects.get (...)”, “objects.filter (...)” in Django. Simple, beautiful and convenient, but what algorithmic complexity of executing SQL-queries generated by ORM, this will lead to, is not visible to the naked eye.

When a person writes an SQL query, it is assumed that he thinks and understands the cost of computing resources. ORM veils this task.

Hypertable as a new generation database

We have developed a new concept, methods and practical ways to combine the relational and network database models with the advantages of the ORM idea - the rejection of the use of special query languages, which allowed us to create a new database model and technology.

The key concept is a hypertable (GT) - this is a database as a set of relations (tables) in which are used:

1. “Relational” attributes (data domains), the values ​​of which, as in the RDB, are field fields with self-defined data for the corresponding table columns
2. "Network" attributes (link domains). We will call them ATS - attribute type "link"

The values ​​of the PBX fields in the rows of the table are explicit links to any rows in any tables included in the hypertable.

The concept of a hypertable introduced by us has nothing to do with the project [13], which was curtailed in 2016.

There is a working prototype - a set of tools in Python - the Hyper Table Management System (HTMS), which includes the following levels (from top to bottom):

• HTed hypertable editor (client) - a technological support service implemented as a website on the Django framework, which can connect to any server with a hypertable regardless of applications (it is functionally close to the PgAdmin utility for PostgeSQL to some extent);
• library of utilities and classes of the logical level - API for creating a database and manipulating data at the application programming level (replacing ORM);
• a library of utilities and classes of the physical level of working with the database, on which utilities and classes of the logical level are based (how the API can be used by experienced system programmers);
• the Cage class, which is designed to create on the client (application) side a “virtual” layer of cached remote access to database files;
• CageServer file server - software that works in multi-program and multi-threaded mode, implements functions for multi-user remote access to files on a server using TCP protocol.

In principle, it is possible to use the usual local file subsystem of the OS to manage files instead of Cage, and also use the Cage API and CageServer software as an independent HTMS tool for implementing remote distributed file access on any systems.

In future articles, it is planned to introduce readers to the HTMS system in more detail.