Skip to content
Snippets Groups Projects

Future of the Backend

    • Open Issue created by Henrik tom Wörden @henrik

    Current state

    • The backend has been stable for quite some time, but it suffers from performance problems.
    • There is a simple structure of tables (entities, double_data, reference_data, etc) that stores the information of all Entities
    • Records with their properties are basically stored as triplets (rec_id, prop_id, value)
    • So called "replacement" properties are used to represent "special" properties that cannot be put in the above structure by default: lists, subproperties, overrides...
    • This means that
      • Intensive calculations have to be done to create a Record from those tables (compared to reading a line from an sql table). This implies that it is not easy to retrieve data directly from SQL but the java Server is needed to make sense of the data. E.g. working with an SQL dump directly is hard/impossible.
      • Queries, Read and Write is complex and has high calculation costs
      • Many SQL performance perks cannot be exploited (e.g. because operations are done on temporary tables)

    Ways forward

    The minimal one

    Adjust the backend to remove those things that are perceived as the greatest impediments. For example: Remove replacement properties

    Who need structure?

    We replace the backend by a document database Backend such as MongoDB.

    Advantages

    • scalability: read and write is horizontally scalable since independent units are stored
    • flexibility is unlimited

    Disadvantages

    • The backend can no longer be used to assure consistency:
      • references might be broken
      • records may no conform with RTs (currently the case)
      • (eventual consistency)

    Back to the roots

    An SQL Backend that switches to another paradigm: Instead of creating flexibility by serializing complex objects to simple tables, we create the flexibility by modifying the table structure itself. Tables represent Records as it is typically done in SQL: Basically one table per RecordType and one row per Record.

    Advantages

    • Very fast Read, Write and search
    • SQL transparently describes stored objects
    • SQL queries are easier to understand

    Disadvantages

    • Wartung der Tabellenstruktur ist deutlich aufwändiger als bisher
    • Migration des Backends (Es ist unklar welche Tabellen im Kundensystem existieren).
    • (horizontale) Skalierbarkeit nicht so leicht wie bei noSQL
    • möglicherweise zu kleines Limit bei der Anzahl von Zeilen

    NoSQL

    The main NoSQL database types are document, key-value, wide-column, and graph. Key-Value databases are mostly used for caching, graph databases focus on the edges. While in LinkAhead edges are important, they are not the central role and we only support 1 relation among data (+ inheritance in the data model). Wide-column is effective in storing sparse data. This leaves us with document databases.

    Document Data bases

    • typically do not provide joins
    • related data is stored in single documents in a denormalized form

    Feature List

    Must-Haves

    • multiple inheritance
    • list values
    • RecordTypes can have Properties added and removed
    • Renaming of RTs, Ps, Rs
    • allow to have Properties with units
    • Versionierung
    • String-IDs
    • References among Records
    • References must not be broken/dangling
    • obligatory properties
    • Entity wise Permissions
    • Suchfähigkeiten
      • Konjunktion, Disjunktion, Negation
      • POV-Filter
      • Referenz-Filter (inkl. Backref)
      • File-spezifischer Filter
      • spezielle Behandlung von ID und Name
      • Filter nach Metadata (Created-by etc)
      • Filter ohne Name oder RT
      • Select Queries
      • Select inkl. related Properties

    Should-Haves

    • RecordType of a Record can be changed
    • Predictability of datatypes
    • I can change the data-type of a property
    • Possibility to model other relations (eg. part-hood)

    May-Haves

    • Role of an Entity can be changed
    • Subproperties
    • datatype overrides
    • Multiproperties (propertynames are no keys; maybe this should be classified "Should-not-Haves")

    Should-not-Haves

    • name overrides
    • description overrides

    Bewertungskriterien

    • Transparenz/Verständlichkeit
    • Skalierbarkeit
    • Vertrauen in das System (Bekanntheit der Lösung(sstrategie))
    • Funktionsvollständig
    • Performance
    • Reifegrad(Praxiserfahrung)
    • Theoretisches Fundament

    Reading

    Edited
    To upload designs, you'll need to enable LFS and have an admin enable hashed storage. More information

    Child items ...

    • Activity

    Please register or sign in to reply