Which is the Best Way to Run Mautic for Agencies?

Every so often I get requests to deploy multi-tenant Mautic setups, those requests are usually from a Marketing agency that needs to run multiple Mautic instances for their customers or their internal usage, However, multi-tenancy is very often a misunderstood and misused concept.

Multi-tenancy is an architectural approach enabling a single instance of an application to be used by multiple organizations or users, the tenants. In multi-tenant apps, there is just one code base for everyone and the separation between clients is done in that single codebase and sustained by the database.

On the other hand, Multi-instance, what I call “Replication” provides each organization with its own codebase.

Let’s analyze in detail these 2 options and find out which is the best way to run Mautic for an Agency.

---

Multi-tenant = one unique codebase for everyone + more complex database setup.
Replication =  Each organization has its own codebase and independent database.

Of course this is oversimplified, there are several variants of each of those basic types, but for our analysis, the basic types will suffice.

Which is the best “multi-client” Mautic setup for Agencies?

What made multi-tenancy great in the first place?

These are the reasons why multi-tenancy was so popular a few years ago: 

• Multi-tenancy reduces costs by sharing resources and software across its clients. Software requires licenses and support, and the cost for this will increase if you need to buy them for each customer.
• Multi-tenant applications, only need to be monitored and provide administration for one single platform.
• A multi-tenant APP is easier to scale. When more computing power is required, the total capacity of the entire system increases and becomes more scalable when you bring in new hardware to the platform.
• Updating software versions or system resources is simpler.

Are those reasons still valid today?

Multitenancy uses one single codebase with the necessary user abstraction needed for the application to be used by many client companies, this was a good way to facilitate maintenance of the code base, however this strategy, in my opinion, is old and not well suited to a plugin enabled APP like Mautic, especially not if you want to keep the benefits of the open source code in a way that each of your clients is able to modify or extend their mautic code if ever needed.

If you were dealing with software licenses, then yes, maybe there could be a cost factor important enough to consider a mono-server, multi-tenant architecture, however, Mautic Being Open source and free of any license costs, you can use as many copies as you need for the same cost, zero. The same applies to all the software you need to run Mautic, from the Linux OS to the MySQL database, all the required components are FOSS (Free and Open Source Software). 

Some years after, the availability of unlimited resources is cheap and convenient, the difficulty of managing thousands of parallel codebases is now practically zero, and the newer tools allow for very quick ways to test, deploy and rollback new versions of software in a completely automated fashion.

Moving replicas of Mautic around from one server to another or even from one datacenter to another is a snap with modern technologies, making both scalability and portability an issue of the past, moving giganormous multitenant apps and their databases is complicated and troublesome.

If you also take into account the multiple advances on container technology, where version deployment and version rollbacks are a core part of the orchestrator, native to and embedded in the core of the system; Replication has become a much better way to deal with multiple clients for almost every single type of APP but so much more for Mautic as a FOSS software.

One big problem with replication of PHP apps like Mautic, was how these replication machines were built in the past, you had one database engine with multiple databases (or tables) for each of your clients and then an HTTP server, usually Apache serving the pages, each client would then have it’s own domain/sub-domain so Apache could serve the right content to each client. That model was a security nightmare! if you had 1000 clients, you had 1000 entry points to your system, anyone breaking the security of any single one of the 1000 sites was a menace to the system security, potentially affecting the entire server, very rapidly. Virtual machines and VPSs were a good alternative, but tremendously expensive and good enough only for the most expensive products. 

With containers becoming mainstream after the creation of Docker containers, the data leakage and security issues have been tremendously minimized if not totally dealt with, and the cost of running modern containers vs VMs or VPSs is about 1:100.

So it’s not about one or the other being better or worse, is that the reasons why multi-tenancy was started in the first place have now become irrelevant. Add to this the disadvantages of multi-tenancy plus the extra flexibility of replication in modern environments and choosing becomes a no brainer.

Disadvantages of Multi-tenancy:

• A multi-tenant app has less flexibility than a replicated app to set low-level configurations and personalizations and makes it from very difficult to just impossible to attend individual client needs when those deviate from the original design. 
• A multi-tenant app is much more complex than an equivalent single-tenant app. In the case of Mautic, it would require rewriting a lot of the APP.
• Protecting your clients from data leakage between tenants is a huge concern in multi-tenant architectures.
•  Since a multi-tenant application is backed by a single database, it has less moving pieces, however, failure of any of those pieces can prove a lot more disastrous and usually all tenants experience loss of service whenever there’s a service shortage for a multitenant app.

Modern replication takes care of all of those aspects, each tenant has its own database and its unique codebase that can be tweaked to satisfy any particular needs.

How to build a modern and highly available Mautic environment for Agencies where deployment of new servers is easy, secure and automated?

The answer is containers. Containers and container managers, also called orchestrators.

Containers are as secure as virtual Machines but thinner and lighter, with the amount of resources needed for one VM you can run as many as 100 containers.

Containers are portable and can easily be moved from one host to another in a matter of seconds, without service interruption.

The latest container technologies also take advantage of powerful file systems like ZFS, that allow for a minuscule blueprint of the containers on disk by only storing the differences between one container and the other instead of full copies of the data.

Let’s see what are our options for containers:

There are many ways to manage servers, VPSs and containers, I’m gonna concentrate on the 2 that are most popular and modern and in one older alternative that has been recently revamped by canonical, one that I personally think to be the best suited for Mautic and that I use for all these kinds of setups unless directly asked otherwise by my client.

Containers types:

– Docker: This tremendously versatile container is the joy of programmers and DevOps pros, it makes it easy to package single applications and to replicate them and allows for the best portability possible as Docker Containers can run on almost any operating system. They are heavily used as either a packaging tool for developers or to run microservices in production.

– LXC: This is the father of modern containers, most of today’s container technologies, including Docker, are built upon the foundation of LXC. As a very seasoned tool, it offers stronger security and because LXC containers have been around for many years they are also the most reliable.

There are many other container technologies out there, however, Docker is the most used and we’re also visiting LXC containers because they specifically fit the Mautic multi-user scenario beautifully.

What’s the difference between those 2 types of containers?

Docker is an “application container” ideally suited to host one single application, its capabilities are terrific for microservices. However in order to run, Mautic needs, besides itself, at least 2 more applications, a mysql database, an HTTP server. In a replicated environment for an agency, probably also a msg queue, proxy server, PHP interpreter, etc.
Can’t you package all those things into a Docker container? Technically yes, but it defies the very purpose of Docker Containers. Additionally, Docker Containers are not yet very good at persistent data. Docker is designed to be extremely portable, which means they are designed as volatile, disposable.

Stéphane Graber, LXD developer explains: LXD is designed for hosting virtual environments that “will typically be long-running and based on a clean distribution image,” whereas “Docker focuses on ephemeral, stateless, minimal containers that won’t typically get upgraded or re-configured but instead just be replaced entirely.”

LXC is a “system container” it’s the modern equivalent to the virtual machine or the VPS, it’s designed to be treated as a full operating system that can host multiple applications that are meant to be run permanently, based on that, an LXC container has its own storage and file system.

Because the current Mautic version was designed the old way, as a monolithic app and not as a series of microservices, LXC is the best match and together they work wonderfully well.

But containers just contain, the best part of using containers for replication is not the containers themselves but the container managers.

Container managers are the tools that allow all the magic you can do with containers.

– Portability, move containers around like you just don’t care 🙂

– Programmatically launch and manage the lifecycle of containers.

– Self-healing of applications, 

– High availability and live snapshots

– Clustering

Again there are so many tools to manage and orchestrate containers, the most prominent and second to none is Kubernetes.

Kubernetes, codename Borg.

Originally designed by Google and improved by a huge community with the greatest IT companies participating in its development, Kubernetes signifies the cutting edge of container management.

Kubernetes was, from inception, a container-agnostic manager, meaning that you should be able to manage any type of container with it, but the reality is that its development has very much been done around docker containers, alone, leaving other container technologies behind.
Canonical is doing a very good job bringing LXC containers to Kubernetes, however, they’ve also been doing a wonderful job at building their own container manager around LXC containers, it’s called LXD.

Kubernetes and LXD are not exactly interchangeable, they differ a lot architecturally, please pardon me if I oversimplify too much but this is already a very long post and I really want you to get to the end of it before it’s dark 🙂

LXD provides the most important tools to manage system containers, it doesn’t yet deal with things as self-healing as Kubernetes does and this is probably by design. If your containers are volatile, self-healing is a necessity, but system containers are not supposed to be volatile, so there’s no need for self-healing (tho it would make for a nice addition, of course)

LXD gives LXC containers an API, a simple(r) method of deployment, a solid and automated way to surface persistent storage to the container, clustering and the tools to move containers around seamlessly.

While Kubernetes and docker are all about moving containers around, LXD and LXC are about running virtual servers as efficiently as possible. The footprint of LXC containers is so small you barely use any resources, the speed at which LXC containers run is almost the same as the bare metal.

Kubernetes is very complex, the original codename of the project when it was started at Google was “Borg”, this says a lot! It was designed to run hundreds if not thousands of servers. LXD is much simpler, better suited for small and medium-sized clusters of a few servers.

Can you do everything Kubernetes can do but with LXD? Yes, you can, it’s just not there out of the box because it’s out of the scope of the project.

Conclusion:

• Multi-tenancy is not the optimal way to run Mautic for Agencies.
• Single-tenancy / Replication gives better security, scalability, portability, performance…
• Docker Containers are designed for the next generation of applications based on microservices, not the best fit for older apps like Mautic.
• LXC is a much older technology which also means it’s sturdier, more secure but most importantly better suited to run monolithic applications like Mautic.
• Docker is an application container, focusing on portability and microservices.
• LXC is a system container, focusing on performance, small footprint and providing the needed resources to monolithic apps.
• Kubernetes is the big dog of container managers, however it is quite complex, data persistence is still a work in progress and overall it’s just overkill for small to medium clusters.
• LXD is the recent and native container manager specifically designed for LXC containers, Just as LXC follows a zen approach to containers, LXD also follows this approach, together they conform the most efficient container + manager combo.

When I was first requested to build a multi-user system to run Mautic for an agency, I first followed the trail of breadcrumbs that inevitably brought me to Docker Containers and Kubernetes, probably just like everyone else. I then realized the shortcomings of this platform, specifically the shortcomings related to databases and surfacing storage to Docker Containers, but I kept looking around and discovered this wonderful combo that is LXC and LXD and how this older technology was better suited for Mautic.