• Targets of the rewrite
• Rewrite bottom up
• migration
• Current status
• plan

Morphium V5

Morphium originated in 2012 and has had 4 major releases since then. Now it's time to update some more and tweak some internals. This update is a major one, so parts of the internals have been completly rewritten while keeping Morphium itself sourcecode compatible.

Targets of the rewrite

The aim is to cut off old "braids" and update the code. It should remain as compatible as possible so that migration is easy.

Furthermore, the complexity should be reduced, there are some features in Morphium that are certainly rarely used or not at all, such as "optimistic locking" with the use of automatic versioning. These "features" are removed and the code is streamlined.

Furthermore, Morphium should remain compatible with the older versions. That means the changes to Morphium itself will be limited. The concepts will not change either, at most they will be tightened up a bit. So Morphium is still the linchpin of the API and every write access will continue to run via Morphium and every read access via Query (exception: aggregation with the help of the aggregator). In short: the use of Morphium essentially does not change.

JDK 11

The JDK used should also be updated during the refresh. Morphium V4 stays on JDK1.8, from V5 we will support JDK11 and later. Of course, the code should also use the new features of JDK11.

Morphium Object Mapping

In current tests, the Morphium Object Mapper is still one of the fastest and most feature-rich object mappers that we have found. Compared to 'Jackson', the higher performance and the features for handling lists and maps (handling 'Generics') have to be mentioned in particular - this makes the MorphiumObjectMapper really indispensable.

During the rewrite, however, some features that are also reflected in the object mapping were removed, which simplifies the code and makes it easier to maintain.

Morphium Cache and WriteBuffer

Morphium had declarative caching as part of the system from the start. This continues to be a key component. The cache implements the JCache API and could therefore be used for other purposes as well. The other way around, you can also use a JCache in Morphium for caching.

During the rewrite we will adapt the code to JDK11 and improve the structure of the code.

Morphium Messaging

Another key component is messaging. Particular attention will be paid to messaging to improve stability and reduce load. Specifically, by efficiently using the Driver API and WireProtocol. Another change will be the connection handling in messaging: currently messaging uses Morphium itself as a connector to the mongo. Since none of the features of Morphium are necessary for messaging or are sometimes even more of a hindrance, the messaging will communicate directly with the MongoDB in V5. In the current V5 implementation, the messaging uses a dedicated connection to the MongoDB for the push notifications (Watch) and a dedicated connection pool for the connection to the mongo. This means that the messaging is a little more detached from Morphium, generates less load and can be configured separately. This is implemented and the tests show, that the performance increases whereas the load on mongo is reduced.

MorphiumConfig

This "thing" has also grown a little over the years and is currently hardly usable with the 100s of getters and setters. The settings have to be split up a bit and used in different categories. In particular, because Morphium's own wire -protocol driver requires different settings than e.g. the InMemDriver.

Focus on Community Edition

However, we only implement this for the Community Edition. This means that the enterprise features are not considered in the rewrite. It can also cause the drivers not to work properly with the Enterprise MongoDB version. The same currently applies to the Atlas MongoDB implementation, which will certainly be the first to be submitted. Support for the Enterprise Edition features may also be implemented in one of the following versions if required.

Rewrite bottom up

The rewrite is done from "below", i.e. first the Mongo driver (see below) was implemented, which implements the Wire protocol (currently only usable for MongoDB from V5! For older MongoDB versions please use Morphium V4.x!).

The necessities and the structure of this driver is mainly determined by the MongoDB and propagates the architecture "up". On the way there, unused features are removed (e.g. auto-versioning), others are improved simply by adapting them to the new drivers.

In the end, a Morphium should come out that is almost 100% source compatible with the old version, but much more modern, slimmer, more beautiful!

Morphium driver

Morphium is more or less a replacement for the existing mongoDB driver with a lot more features and in some places significantly more security and / or performance.

Currently (i.e. in Morphium V4.x) there are 2 drivers: MongoDriver and InMemDriver. The MongoDriver in particular creates a bit of overhead because it is based on the official MongoDB driver, which unfortunately is now not just a driver, but also an object mapper. Many functions of the driver are not required at all, but some others are missing. So the aim is to introduce our own WireProtocol[^1:WireProtocol is the protocol that MongoDB uses for communication] driver, which is optimized for the needs of morphium. This can result in a significant performance advantage (currently between 10% and 20%).

2 drivers are implemented in V5:

• SingleConnectDriver- a driver that only keeps a single connection to the MongoDb, but monitors it and also implements a failover if necessary. This is used, for example, in the new messaging substructure (see below)
• PooledDriver - a full featured cluster connector with connection pooling and automatic failover.

InMemory Driver

Strictly speaking, there is a third driver, the InMemoryDriver, which makes it possible to run Morphium with an "in memory" mongo. This is particularly useful for tests. We use this as a TestDB in some projects and it is also comparable to other databases implemented for test purposes. The InMemory database should be (nearly) feature compatible to a real MongoDB installation. We're not 100% there yet, but we've reached a good 90%. Most features are already working, but some things take a little longer and might come later (such as "real" text indexes).

Why not the MongoDB driver

This question is asked more often and I would like to say a few words about it:

Features of the driver

The official MongoDB Java driver is not only a driver, but "embedded" in the driver is also an object mapper. This is probably super useful for many applications and completely sufficient.

Unfortunately not for us, because we want to configure some things rather than program them (declarative indices, etc.).

Two different ways of thinking collide. Which in the example of the MongoDB driver and Morphium has repeatedly led to... discrepancies. Some changes to the driver API have been a lot of work in Morphium.

The MorphiumDriver project has been around for a few years. It was brought to life because we had problems with the official MongoDB driver, especially in the area of ​​failover. Also, some features of the driver are at least a little questionable in my opinion.

These features, especially the object mapping, make access via Morphium a bit unwieldy. And some things happen unnecessarily twice. And that's exactly why we want to try to develop our own driver that does exactly what we need - and nothing more.

When accessing the Mongo in the stack trace, there are currently a good 20 calls within the MongoDB driver. With the new driver that's a maximum of 5!

new feature: InMemoryServer

Morphium in version 5 got its own server, which is also based on the InMemory database. This is naturally intended exclusively for test purposes and does not offer any security or anything similar. However, it is always helpful for testing a (simple) in memory mongodb (with the limitations of the InMemoryDriver). If you want to test it now:

The MorphiumServer understands the following command line options:

• -mt or --maxThreads: maximum number of threads in the server (= maximum number of connections)
• -mint or --minThreads: minimal number of threads (kept ready)
• -h or --host: which host does this server report back as (in the response to the hello or isMaster command)
• -p or --port: which port should be opened (all network IPs)

I would like to reiterate that this server is purely for testing purposes and is certainly not intended to be a replacement for any MongoDB installations.

migration

Migration should be relatively easy since Morphium's API itself hasn't changed. However, the configuration has changed:

• the class names were removed. It makes no sense to include your own query implementation. At least we were not aware that anyone had done that. The same applies to the defined factories etc.
• The drivers are no longer referenced via the ClassName, now the DriverName is specified. Valid drivers currently are: InMemoryDriver, SingleConnectionDriver, PooledDriver, MongoDriver. Unfortunately, the MongoDriver is currently not working and it is not 100% certain that it will remain part of Morphium
• direct access to the driver - in case someone did it - is still possible, but the driver API has changed significantly. That's why you have to reckon with major renovation work here.
• the optimistic locking feature has been removed. Unfortunately, everyone who has used this feature currently has to find their own implementation

Current status

Currently, at the end of August 2022, Morphium V5 is available as an 'ALPHA' version in Maven Central and can be used for your own tests. However, some features are not yet 100% implemented. This is how Morphium works only with MongoDB versions 5.0 and later. Also, the transaction feature is currently not working.

plan

in the coming weeks these things will be addressed in this priority:

1. Test stability / failover etc and adjust if necessary
2. Implement auto retry read/write everywhere
3. Finalise InMemoryDriver - add missing features, increase performance.
4. Rewrite MorphiumConfig and remove old settings, adapt new ones, possibly make them more modular.
5. Implement transaction handling (at driver level, morphium has supported this for a long time)
6. Complete or remove MongoDriver.