• Create a ticket in the ticketing system if the issue needs attention but isn’t urgent.

• Immediately escalate the issue to a human if it’s urgent or delaying a response could cause harm.

• Create a general feedback note if no immediate action is necessary.

As a task that requires understanding of human language and simple decision making skills, this seems to be an obvious choice for AI!

What’s An AI Agent And What’s CrewAI?

An AI agent is a piece of software built around a large language model (LLM) that can act autonomously - make decisions, perform tasks, and use tools. Essentially, it's like a mini version of ChatGPT that you can integrate directly into your code, giving it access to a variety of tools. These tools can scrape the web, read from databases, parse Excel files, or perform virtually any function you can think of. By combining decision making capabilities with external tools, these agents can automate a wide range of tasks.

CrewAI is a python framework build on top of langchain that makes it very simple to define those agents, their tasks and tools and have them work together in a crew. A crew can consist of a single agent processing a single task or of multiple agents working together and even delegating work to each other.

Why CrewAI?

Today, OpenAI and others offer APIs to make requests to their LLM, have it return specific results, and even use tools. So why should we use crewAI, especially if we are only using a single agent to solve a single task?

CrewAI makes it easy to define agents and tools in a way that is reusable and very easy to understand. It provides tools for logging and monitoring, and - as we will see later - it is not tied to a specific LLM. Although crewAI uses OpenAI's GTP models by default, its agents can be driven by a large number of models, including local LLMs, so that sensitive data is kept in our system.

CrewAI is one of the most popular frameworks right now, but it’s worth mentioning that the same can be achieved with a number of other frameworks like haystack or ell.

Defining A CrewAI agent

Let’s get to it! You can check out the full code repository on github.

First, we define an agent. CrewAI makes this very straight-forward, we just have to provide a role, a goal and a short backstory. This will be used in the LLMs system prompt to let the language model know how it should act.

Agent(
    role="Customer Support Representative / Customer Service Agent",
    goal="Understand user feedback, decide what action to take and apply the tool to take the action.",
    backstory="You are an experienced customer support agent.",
    verbose=True,
)

Setting verbose=True will make the agent print it’s thoughts and decision making to the terminal, a very useful feature when it comes to debugging.

Creating The Tools

Next, we define some tools for the agents to use. Remember, we want the agent to decide how to handle a user message, whether to create a ticket in the ticketing system, escalate the issue to a human, or create a feedback note.

@tool("Save Task Tool")
def create_ticket_tool(ticket_description: str, user_id: Optional[str] = None):
    """Create a ticket in the ticket system that will be reviewed and solved later by a team member"""
    tasks_db.insert({"task": ticket_description, "user_id": user_id})
    return "Task saved"


@tool("Escalate issue tool")
def escalate_issue_tool(issue_description: str, user_id: Optional[str] = None):
    """Immediately escalate an urgent issue to a human"""
    escalations_db.insert({"issue": issue_description, "user_id": user_id})
    return "A human has been informed!"


@tool("Create Feedback Note Tool")
def create_feedback_note_tool(feedback_description: str, user_id: Optional[str] = None):
    """Create a general feedback note for the team to review"""
    feedback_db.insert({"feedback": feedback_description, "user_id": user_id})
    return "Feedback note saved"

Let’s break it down:

• The @tool() decorator marks our python function as a crewAI tool (and provides a tool name for the agent.

• The python docstring serves as a tool description and will also be handed to the agent.

• The agent will then receive feedback from the return statement.

• In this example, we simply use tinyDB to store these items.

You may be thinking: How can this work reliably? How will the agent know what parameters to provide? - And you wouldn't be wrong. For example, the agent gets a clue about the parameter type from its name: ticket_description suggests that the function expects a string. In the past, simply calling the ticket parameter would confuse the agent into trying to pass a dictionary with title and description attributes. The definitions above work for our simple case, but crewAI also provides a way to implement tools as classes and use Pydantic to thoroughly define their parameters and their types.

Defining A Task

We can now define a precise task for our agent to complete:

def create_message_processing_task(agent: Agent):
    return Task(
        description=(
            """
            Your task is:
            1. Read the user's feedback.
            2. Decide what action to take.
            3. Use the tools provided to take the action.
            
            These are the decisions you can make and their corresponding actions:
            1. If the user requires immediate help to prevent further damage, escalate the issue to a human.
            2. If the user is reporting on something that needs to be fixed or looked into at some point, create a task.
            3. If the user is providing general feedback, create a feedback note.
            4. If you think no action is required, you can ignore the feedback.
            
            You are done when you have taken the action.
            
            The user's user_id is: {user_id}
            
            This is the user's feedback:
            
            {feedback}
        """
        ),
        expected_output="Explain what action you took and why.",
        human_input=False,
        tools=[
            create_task_tool,
            create_feedback_note_tool,
            escalate_issue_tool
        ],
        agent=agent,
    )

Traditionally, a task can be defined much shorter and doesn't need to have tools attached. In our case, we don’t rely on task outputs directly, because we use tools to handle that. A task description could be as simple as: "Improve the wording in this text: {text}" , with the expected output being used directly from the task. Variables like {user_id} and {feedback} are provided and replaced dynamically once we get our crew to work.

Experience shows that there is a trade-off between task definition and performance and reliability. A broader task definition might give the agent more room to be "creative" and come up with a solution to more unforeseen problems. But in tests, it also significantly increased the likelihood that the agent would go around in circles or get stuck. A solution might be to be precise in the task definition, but allow the agent to delegate the work as a new task to another agent if complications arise.

Putting Together The Crew

It proved most effective to have a dedicated crew work on a specific task or set of related tasks with a corresponding agent or group of agents. Reusing crews, agents, or task instances often led to unwanted side effects in their behavior, so we now instantiate everything fresh for each incoming feedback item.

for message in feedback_messages:
        crew = Crew(
            agents=[agent],
            tasks=[task],
        )
        crew.kickoff(
            {"feedback": item["message"], "user_id": item["user_id"]},
        )

You can view the full source code on github.

Results

So far, the set-up agent works very reliably. Urgent issues where users cannot upload their assignments are immediately escalated, while the system formulates tickets for bug reports and forwards general feedback.

Using A Local LLM For Privacy And Data Security

Per default, crewAI will require you to have an OpenAI API key set and use the gpt4-o-mini model for agents (by the way, if you don’t have an OpenAPI API key, you should get one, their API is ridiculously cheap to use for development).
Im many scenarios, especially with EU clients, privacy and data security are big concerns, so businesses might be very reluctant to use remote models.

Using a local LLM in crewAI is as easy as installing Ollama, pulling the model and configuring the agent with something like llm=”ollama/llama3”.

Local LLMs have made incredible progress in the past: They can be powerful alternatives and they come in many shapes and sizes for many different purposes. Although a local LLM might produce comparable results, it will likely be a lot slower and changes in the agents behavior have to be expected. For example, an agents interpolation of tool parameter types from parameter names might be different.

Final Thoughts

Talking with my clients and colleagues, I found it surprisingly difficult to identify real-world scenarios where an actual team of AI agents could be used to make complex decisions and delegate tasks. These scenarios often touch on the core operations of a business - areas where clients generate the most value and prefer that humans make the key decisions. Some might argue that this is exactly the reason why these humans will be obsolete in the future. However, after working with agents for some time, I can confidently say that we’re not even close to a point where I’d trust current-generation language models to take on such responsibilities. That said, it's October 2024, and things could evolve faster than we expect.

However, we came across many examples like this one, where a single AI agent may be implemented to make easier decisions and derive actions. In these cases, some control flow might still remain in human hands or be managed through the predictability of traditional structures like foreach-, if-, and else-statements. Meanwhile, projects like crewAI offer new ways to define control flows where AI agents are actively integrated into the process

This can happen for a variety of reasons. Maybe you just underestimated the compounding complexity or maybe someone took good principles too far. A very common scenario is that growth expectations weren't met or that the company has not managed to hire enough new developers to match the number of systems.

An obvious approach would be to merge some systems back together. However, this is sometimes not useful or possible for various reasons, e.g. because different tech stacks were used.

For our startup, complex communication patterns that should ensure reliable communication between the services represent the greatest overhead. So we looked for compromises that would allow us to simplify this part of our architecture.

These solutions worked for us because as a startup, we can make certain compromises without risking major disadvantages. The kind of compromises your team can make may be different. In any case, it is important to understand the challenges we try to solve, the risks that arise from poor design decisions and the advantages and disadvantages of different compromises to be able to simplify a system. Nothing is for free.

I’ll start out discussing challenges and risks in inter-service communications, and how we usually tackle them in enterprise-level architectures. I will explain why these approaches may not suit your startup and what compromises can be made to move faster and build a system that is easier to understand and maintain.

Why should we replicate state?

There are multiple reasons to have well-isolated systems in enterprise architecture. One major advantage is that it enables multiple teams to develop and ship major parts of your system individually - at least in theory. Another good reason for a high degree of isolation is that your services can run independently. So if one service gets stuck, your clients will still be able to use the rest of your application - also at least in theory.

To reach that high degree of isolation, each system must own all the data it needs to operate properly. That’s because, if your user profiles service goes down, for your, say, friendslist service to still be operational, it needs its own copy of the necessary information, for example, a relation between user ids and user names.

That often means that your friendslist service has its own database and state updates have to reliably be propagated from the users service to the friendslist service. So, for example, if a user changes their name, you want the friendslist service to take notice and update its copy of the relevant information.

How we replicate state, the enterprise approach

There are multiple strategies out there to achieve this in the enterprise world. Oftentimes, architects choose some form of asynchronous event stream. So if a user updates their profile, a “profile-updated”-event containing the information is broadcasted and every service interested in this kind of information can consume these events and update the needed information in their own database.

This is often achieved using some kind of event feed. Meaning that all published events are stored sequentially so that they can be replayed at any time. This means that if you were to deploy a new service or, for some reason, wipe the database of an existing one, it would be able to replay all the “user-profile-updated”-events and end up with an up-to-date copy of the needed information. Two technologies often used for that kind of approach are HTTP Feeds and Apache Kafka.

Where it gets complicated

The problem with asynchronous, event-driven communication is that you have to ensure that state updates are actually propagated across systems. That means: Every state change usually involves two technologies, the database of your choice to persist the state and some kind of message dispatcher to publish the state update event for other systems to consume. The trick is to make sure that one action only happens if the other one is successful. We usually don't want to propagate a state change that wasn't persisted properly. But we also don't want to persist a change that wasn't announced to other systems.

This means we need to design transactional behavior that includes more than one technology.

Even if it’s ensured that your state update event will only be published if your database state was updated successfully (and vice versa), it’s still uncertain if that event successfully reached all supposed subscribing services and updated their states accordingly. What happens, if the transactional state update of the dependent service fails after it receives a state-update event?

As you can see, this stuff gets pretty nasty pretty quickly.

If something goes wrong, you might end up with contradictory states and no easy way to resolve them. You may not even notice the problem until chaotic behavior causes problems in far-away and seemingly unrelated places and processes.

Common approaches that try to resolve these problems include the Transactional Outbox Pattern or Event Sourcing. We won’t use any of them.

If communication goes wrong, the consequences could be dear

I remember that a few months ago, a big fashion retailer rolled out a new payment process that enabled customers to try on their items and then pay individually for the ones kept in the order overview. It was a pretty convenient feature, so I used it to pay for all my items and - as expected - all payments were listed as complete in my account.

Several weeks after placing my orders, my inbox was filled with notifications of overdue payments. Initially, I disregarded the emails, assuming it was just another delay in the system, as I had experienced this before. But then, I received another barrage of emails, now accompanied by threats of account suspension and even the possibility of legal action if I failed to make the payments promptly.

It seems like whatever system was responsible for sending out those reminders was not aware that my payments were completed.

Not only was that situation incredibly annoying for me as a customer, it also made me occupy their service hotlines for quite some time.

Imagine how messy something like this could be in the aftermath if it happens in your system. Not only do you have to find the bug. Your team may have to look for missing or existing payments and roll back whatever chain of events the overdue notice might have triggered. Some accounts might have already been unjustly suspended, cutting your revenue and, if it’s really bad, your legal team might already have taken things further against some of your loyal customers.

If your system misses a like on a post or doesn’t send out a push notification due to a swallowed event or an inconsistent state, it usually doesn’t really matter. Especially as a startup with a young app, you are on the frontline for other problems. But this example shows that, depending on what your app does, things can go bad really quickly if you lack a solid concept for inter-service communication.

Therefore, it is essential to ensure that, regardless of any compromises made to our architecture, these issues should never occur, or if they do, their impact should be neglectable.

Why small teams need a different approach

All these approaches come at higher, hidden costs. Maintaining copies of data in multiple sources will not only occupy a good part of your mental capacity for every future architecture and code change, but it also increases your cost and need for infrastructure maintenance.

As a software architect or engineer in a small team or a startup, your primary responsibility is usually to make progress. And that means you have to ship features and be able to quickly make changes to whatever you built. This requires making compromises, and the key challenge is to make these compromises in a manner that enables you to build upon them and sustain that progress.

If your team is small, there are really two ways to optimize your work. Firstly, you have to take some shortcuts to accelerate communication and implementation. Secondly, it's crucial to minimize the amount of future work that you create for yourselves. That means you have to minimize maintenance.

Your cost of maintenance rises with every layer of abstraction, with every test and with every deployed service. That doesn’t mean all those things are bad. But with seriously limited resources, we have to question which of those things are actually necessary and useful.

Let’s make some compromises

As we’ve seen, things are complicated. We don’t want to make any compromises that can cause state inconsistencies across our service landscape. But it seems there is no way around using complicated transaction and rollback patterns if we want to ensure consistency.

Since we would also like to avoid maintaining a large number of data sources, the solution here seems obvious: Dropping the idea that each system owns its own, independent state seems to resolve a lot of our problems. This would mean, you store your data in a single place, the single source of truth, namely the system that is responsible for it.

So, to tie it to the previous example: Only the payment system has information about which items are paid. Other systems won’t hold a copy of the payment information but will instead query the real state once they need it, for example, to figure out who will receive a payment reminder and who won’t

Back to a single-state solution

But wait, wasn’t there a good reason for each system to hold its own data?

For once, you pay with tighter coupling between the systems. Once the payments system is down, all other systems that need some information from it won’t be able to operate properly either. This is a problem for big enterprise systems with hundreds or thousands of concurrent users and customers but it's not a problem for our startup, at least not for a long while. Once it becomes a problem, the architecture can be improved.

Another problem is that you put a somewhat unpredictable load on your root system.

Let’s say one of your core services manages user profiles and is the single source of truth for user names. Because user names are needed in many other services, a single page load that queries several services can cause a lot of subsequent requests to the user profiles service. How many exactly is often hard to say as your architecture grows. But even if your app serves a few hundred concurrent users (and that’s quite optimistic in the beginning), these requests are easily manageable. Moreover, if both services run on the same machine or data center, HTTP requests are cheap and can be cached in the system holding the state of truth, if it becomes necessary.

The case for a message broker

Parts of your systems may still require a message broker to instantly publish events to other systems. Let’s say your app has a chatroom, a chat service that receives and stores messages and a push notification service that sends out any kind of push notifications to users' devices, not only the ones related to chat messages.

As soon as a user sends a new message, a push notification should instantly be sent to other users in the chatroom.

Implementing a periodic query for new messages from the chat service by the push notification service not only introduces a delay in triggering notifications but also creates an inverted service dependency. This requires the push notification service to know about the chat service and its API.

If the chat service were to make an HTTP post request to the push notification service, the direction of your service dependency is correct, but the chat service must be aware of all other services that might need to know about new chat messages.

A message broker solves all of these problems by decoupling services. The chat service publishes a “new message” event to a queue or topic, unaware of who might consume it. The push notification service consumes all messages with a certain routing prefix, unaware of their actual origin.

Because your message broker usually takes care of message routing and acknowledges received messages, this approach also scales well if you decide to deploy several instances of either service.

Does that mean we’re back to transactional updates? Not if we only use dispatched events to trigger actions, not to update data that belongs to another system. We would first store the new chat message in the chat service’s database and afterward publish a “new message” event to our message broker. In the unlikely case that that message gets lost, the worst that should happen is that the user doesn’t get a push notification.

It’s not perfect but it doesn’t cause problematic long-term consequences. As soon as the user reloads the chat history, the message will appear.

Now consider a third system that counts the number of notifications for the user. The overall number of notifications will be the sum of unread chat messages and other notifications. The effective way to make this work is to count and persist the number of “new message” events, add it to the count of other new notifications and publish the result. This is error-prone because a lost event can result in an incorrect count of unread messages indefinitely. To address this issue, the system should be designed to respond to the "new message" event by directly querying the chat service for the number of unread messages. This way, if any error occurs, it can be corrected with the next incoming event.

So far, these approaches and compromises resulted in an architecture that was easy to understand and resistant to state inconsistencies. It’s also relatively easy to maintain and works reliably in real-world app backends.

Some notes on our implementation

In our own app backend, we used the following approaches to implement the communication pattern.

Our synchronous inter-system communication was implemented using plain HTTP requests due to low dependency requirements and our familiarity with the technology.

Asynchronous real-time messages were distributed using rabbitMQ. The use of rabbitMQ was a suitable option, but there are other alternatives available.

We used a single MongoDB instance for data storage, with each service having its own separate database to ensure that no service directly queries another service's database.

To enhance consistency and ease of use, we implemented a common library for clients written in TypeScript. This library provides a typed interface for all internal HTTP request endpoints and message broker communications.

So far, none of the technology choices caused any problems. Would I have to implement this again, I would do it in the same way, using the same technologies. We chose the technologies above because we know them well. Different teams might be better off with different technologies.

What worked for us

These are the simplified principles that worked well for us:

• Don’t replicate state across different systems: It comes with too much overhead and complexity while not providing any benefits for small teams and processes

• Use HTTP requests to synchronously query data from other systems if it’s required

• Use a message broker to send asynchronous real-time information across systems.

• Use asynchronous events in fire-and-forget scenarios or to trigger a system to pull a state update from another system. Never use them to directly update another system’s state.

The compromise allowed us to maintain a compact codebase, minimize maintenance, and ensure consistency in states while ensuring real-time information flow across our systems. Although it may result in reduced performance and increased inter-system coupling, these trade-offs can be easily accepted for small teams with a limited user base in the early stages of product development.

Photo by Omar Flores on Unsplash

Deciding on a good backend architecture for our app, häpps, was one of the most challenging decisions we had to make. Even for experienced software architects, this decision is always a complicated one because it sets the stage for future development. The fact that we are a startup and therefore only two programmers with no extra capital complicates everything further. Also, making architecture decisions is an ongoing process, because young apps tend to evolve in unforeseen ways. It's essential to have a backend architecture that can quickly evolve and adapt to new requirements. Flexibility is a crucial requirement - it may even be the most important one. But what does it mean to build a flexible architecture for a startup?

Throughout my career, I have designed (or was involved in the design of) many backend architectures for both startups and enterprise systems. All of that experience went into the design and re-design of our own app backend. In this blog post, I'd like to share our discussions, arguments and insecurities, the resulting architecture, and the challenges and compromises we faced. I'll present the architecture that worked for us and what I would change if I had to do it all over again. However, please note that this article is not a definitive manual that is suitable for all apps. It is also quite opinionated, and the approaches presented may be debatable.

What does our app do?

Very simplified, the app allows the user to create so-called happenings. These are temporary groups organized not around a fixed set of people but around an occasion, something like a gathering or a party. Users can invite people to happenings and invited users can accept or decline invitations. Once a user accepts an invitation, they are part of that happening and can read and write in a chat and participate in further planning. Every happening also has a gif, video or image as its background media. On top of that, users have profiles and can befriend each other. A surprisingly complex part of the backend system is only concerned with push notifications. These notifications are sent on a great number of occasions: From friend requests to new chat messages, reactions, accepted invitations, time or place changes or as a reminder for open invitations and upcoming happenings.

The app as it is now has a fairly complex backend architecture spanning at least a dozen services and multiple means to share data and events between them. All of this was built mostly by one person and all of it runs smoothly. The overall level of complexity at the time of writing is much bigger than it was in the beginning. But mainly because of small features and improvements that required a lot of back-and-forth of information between services. The app's main features mentioned above are pretty much the ones we built from the very beginning.

Microservices usually don't work well for startups

When we started building häpps, the microservice hype was full on and deeply influenced our discussions. We quickly agreed that the term microservice is somewhat misleading because it seems to suggest cutting services as small as possible. Splitting your codebase into multiple independent services comes with a lot of disadvantages: Each service has to be maintained and deployed and increases the boilerplate needed for shared data structures, complicated communication protocols and infrastructure. By the time we released our app, the common library used by most services has been updated over a hundred times. And every service has to follow suit eventually.

And there is more: Most advantages of multi-service architectures for some enterprise-sized systems won't be relevant for a startup at all.

The most important one: The ability to deploy services independently doesn't matter for early startups. You don't have autonomous teams if you only have one or two teams. In this scenario, the individual deployment of multiple services is an impediment, not an advantage. The concern of the whole system crashing because one developer messed up on one end is also irrelevant as long as everyone works on all ends at once and you don't risk revenue losses because hundreds of users use your platform at the same time.

If you are lucky enough to reach a point where the horizontal scaling of a monolith isn't enough to keep your backend going, you will also be at a point where you have to rewrite big parts of your business logic and make performance improvements in data structures, database queries, caching and serialization. If you start with a relatively small user base, using up-to-date technology and frameworks, you should easily be able to scale your systems two or three orders of magnitude before you have to change your macro architecture.

So all of this sounds like a strong case against a multi-service architecture - and it is. But as you will see, we somewhat came around eventually. There are still reasons you might split up your backend. And there are better and worse ways to do it. If making services as small as possible in their distinct domain isn't the solution, then how do we determine reasonable boundaries?

Move fast and outsource things

There are a lot of components in your app backend that – although crucial – don’t relate to the core domain of your app. These are also the services that you shouldn’t build yourself:

• Auth services, obviously

• A push notification or cloud message distributor

• Media services that store uploaded images and take care of things like downsizing, cropping, scaling and caching

• A chat server (?)

One of my favorite decisions on this project was to use Google Firebase. We knew we didn’t want to use it for data storage or computation (and we’re glad we did not!). But we implemented Firebase Auth so we could use their simple kits to take care of everything auth-related, including social logins and password resets. These are the kind of things that you, as an app developer that has to deliver features, can’t afford to spend more time on than necessary to ensure safety.

We also used Google Firebase Cloud Messaging to have a single interface that delivers push notifications to both Android and IOS devices. This decision served us well for a while but Google's lack of flexibility to cover the ever-growing complexity of our push notifications made us doubt the decision. We might have been better off with a system like OneSignal.

We ended up using a self-developed service to take care of our uploaded media for the only reason that we had it sitting around from an old project with similar requirements. If you don’t already have one, I would recommend using a CDN or some kind of ready-deployable open-source container to do the job. Media management gets quite complicated quite fast when you try to keep your user's data usage within reasonable boundaries.

The chat server is somewhat of an unclear case. While building a chat is a rather complex problem, it is also one that has also been solved hundreds of times before. We had lengthy discussions about whether we should implement a ready chat server solution or build our own on top of a more low-level WebSocket framework. We eventually decided to do the latter and I am glad we did. All in all, it turned out that the actual chatting part of the chat server was easier to implement than we thought (that’s only true for the backend!). On the other hand, the chat turned out to be tightly coupled with many of the other app's core features, so it would have become more and more difficult to build on a generic product.

While we're at it, let's talk about reusable components for backend systems.

Consider building reusable things

To think about reusability on a level that goes beyond one app might seem a little over the top - but I promise you, it's not. You might be convinced that what you build now will be a hit, but the more likely scenario is that you fail, learn from it and end up building something again. But even if that's not the case - if the product you're building will be a success, building somewhat reusable things will increase your ability to adapt to whatever user feedback you get for your MVP.

By the time we started to build our second app, it was obvious that we wanted to move as fast as possible to be able to validate our ideas. We decided to use familiar technologies and a simpler architecture. We also looked at old projects to find reusable code. Our media service - a simple python app that handled image uploads and served static media - was one component that could be reused with minor modifications.

We knew that this time might not be the last time we decide to build a product - whether it would be a success or not. So we decided to try to build reusable services and components.

Not every service can or should be built with reusability in mind. One of the mistakes I witness the most is a tendency to build something generic for something that is very specific: Your domain logic. The result of this is an overly complicated, boilerplate code base that slows down your product development. Thinking about this will also push you in the right direction. Which part of the system does represent the core business logic of your product and which doesn't? The backbone of your chat server? It shouldn't have any special knowledge about your app. The user profile service? Doesn't care what kind of profiles it manages. The push notification gateway? It only manages device tokens and forwards notifications to Firebase. The async message hub? Its job is to register clients and deliver arbitrary JSON messages. All of these systems can be built in a reusable manner. They should be! That doesn't necessarily mean they all have to be standalone services connected by HTTP APIs. If you commit to a programming language and a web framework, you might as well build them as modules and keep them isolated well enough to be able to separate them from your monolith at any time in the future.

There should be no pressure to make everything as reusable as possible. After all, your goal is still to deliver quickly and in the end, it's not certain if you'll ever have another use case for these systems. But thinking about reusability also forces you to think about your whole backend system in terms of domain logic and isolation, something that will pay off later on.

Be pragmatic

Because not everything that works on paper is fun to implement in practice. We ended up splitting some systems and joining others based on compromises and pure pragmatism.

Our friendship service is one example of this. The app enables users to add others as friends. So in its essence, the corresponding service only manages relations between user ids. In theory, it could be used for all kinds of apps, so in the beginning, it was a separate service. But we soon realized that it was tightly coupled to the user profile service: Often, if a list of user profiles is returned to a client, each profile in the list should include a flag indicating if that user is a friend of the user requesting the list. On the other hand, when a list of friends is returned from the friendship service, it should often include profile information such as the users' names. We eventually ended up joining these services into one, but keep their logic in separate modules.

It's easy to lose yourself in thought loops about isolated systems. Consider an API that should return a list of friend suggestions. These suggestions might include friends of friends - information that's stored in the friendships service. But in a second iteration, you may also want to suggest people who attended the same happening as the user. This information comes from a completely different system and it would be easy to argue, the friendship suggestions should actually be a microservice of its own since it combines data from different domains. Then again, the minimal amount of business logic required to join lists of user profiles to a single list of suggestions doesn't justify the maintenance and deployment of a service of its own. In a situation like this, it might instead just be time to look the other way and implement some domain awareness between otherwise independent systems. It's not pretty but it works: Have one of your existing services return the user's friendship suggestions, even if it's not a perfect fit. In our case, we decided to let the happenings service do the job, to keep the friendships service clean of code that belongs to another domain. You might as well choose to do it the other way around or build a kind of miscellaneous service that works as a collection of smaller services.

Boundaries along sync/async communication patterns

Another reason to split your system into services relates to scalability - kind of. Consider a complex action that triggers a lot of complicated, asynchronous side effects.

Take for example a user who decides to join a happening. On the happening service level, a new relation between the user and the happening will be created before the backend sends an asynchronous update prompt to all registered clients, asking them to reload the updated happening from the server. This is a fast, synchronous request. But this action triggers a lot of other side-effects: A system message will be sent to the chat backend so that an info message is displayed in the chat - but only if that feature is activated. Personalized push notifications will be sent to an unknown number of friends and other relevant happening members. That in turn requires querying the database for other relations between happenings and users, existing friendships, user profiles and notification settings. The system will also have to update the notification badge count on the IOS devices of everyone who can see the happening. Determining the correct badge count, however, requires recounting or gathering the overall number of unread chat messages and several other pieces of information.

Performing multiple tasks synchronously can result in long request times, ultimately leading to failed requests. Additionally, the number of side effects generated by these tasks tends to increase at a faster rate than the number of core features. Even if you wrote every line of code in your project yourself, it can be challenging to determine how many other actions are triggered by complex actions.

In the bigger picture, decoupling these things is usually a smart decision. Splitting a system along these boundaries e.g. by using a message broker like rabbitMQ or Kafka can have advantages in terms of scalability later on. However, in the early stages of product development, it's often sufficient to just stick to one app and make use of internal event emitters.

How we did it and how we would do it next time

We ultimately divided our backend system into multiple services, based on all the considerations discussed above. Secondary services for authentication, push notifications, media management and monitoring were a no-brainer.

We also decided to separate systems along communication patterns and reusability considerations. However, we soon learned a lesson in pragmatism: Systems that talk a lot to each other - and only synchronously - should not be separated at first, even if their domains suggest it. Instead, strict modularization will take you a long way.

We are still incredibly pleased with the architecture we selected. It runs smoothly and scales effortlessly, particularly when we add new features. Even though it involves multiple domain services communicating with each other, we've encountered almost no issues related to inter-service communication. This is largely thanks to our comprehensive end-to-end test coverage and our straightforward yet robust communication patterns between services.

But this also required a lot of work: We spent a fair amount of time developing a common library that did not contain any domain logic but all the interfaces and data structures required for the communication between services. This again, was a compromise in favor of keeping things easy.

Developing robust inter-service communication protocols that are suitable for a startup takes time as well. All of this was good work that we would have to do sooner or later if our app would have been a smashing success - which it wasn't. And that's kind of the takeaway.

That's why, if we would have to do it all over again, I would choose a slightly different approach. In the spirit of moving faster towards a minimum viable product, there is still room to cut back on fancy architecture patterns and built a simpler backend architecture - you guessed it - our old friend, the monolith.

During the process of building and deploying a multi-service landscape, we learned a lot about software architecture. We gained valuable insights on how to cut, develop and maintain a multi-service landscape with minimum effort. We also learned a lot from the inter-service communication patterns we designed. However, if your objective is not to learn about the architecture of distributed systems but to efficiently ship a product, it makes sense to opt for a more monolithic approach. If we were to start over, I would choose to build a single system that consists of well-isolated modules with unassuming APIs between them. In place of a separate message broker like rabbitMQ, I would opt for a local event emitter, at least at first. This approach would have saved us a significant amount of time that we could have spent on new features - or for something that we have criminally neglected - marketing.

About flexibility

The most valuable lesson that we learned over the years is that you have to move fast. This applies especially to the early stages of product development. You will almost certainly be wrong about how exactly your product and its features will be received by users. Few ideas will actually end up being successful, at least at first. In that sense, having a high degree of flexibility also means: Whatever you build should be cheap enough to be thrown overboard in case it's not adapted by users, but solid enough so it can be easily improved and built upon in case it is a success. The goal is to spend as little time as possible working on unverified features so that you have as much time as possible to follow up on the ones that your users love.

Do not underestimate a well-designed app listing in Apple’s and Google’s app stores. Chances are, you will acquire a major part of your new users, maybe even most of them, because your app pops up as a store search result. If you get it right, you can create a valuable, constant stream of new users installing your app. Getting it wrong can - in the worst case - have very destructive and unintended side effects.

To get people to install our app, häpps, we were first thinking about inducing mouth-to-mouth propaganda (ridiculous, I know) and running ads in Google’s and Meta’s advertisement networks to reach people on Insta and co (have you ever seen an app ad on Instagram?).

That was until we learned that, especially for a young app, bringing people in using the app stores is an approach that pays off more and, most importantly, costs less than a well-done marketing campaign.

Then, we thought, the goal must be to get as many people as possible to find, install and try our app. After all, if someone doesn’t find what they’re looking for, they would just search for something else or uninstall it. We might even learn something. What could go wrong?

Applying ASO

You’ve probably heard of SEO or Search Engine Optimization. Consequently, there is also ASO, App Store Optimization. ASO is the wizardry associated with improving your app’s ranking in various app store listings. Like in SEO, app store optimization means you’re mostly flying blind since there’s no real insight into what factors affect your store listings, or exactly how.

Your app store appearance includes not only text but also images and, sometimes, videos. Moreover, your app will be listed in a certain category and has to compete with other apps in that category. There are learnings in all of these aspects.

Choosing a category

Take for example your app’s category. Having built a social app, the most obvious thing to do would be to place our app in that category. That however means, our app listing would have to compete with other social apps like Twitter and Instagram, whose associated companies may have budgets in the millions to optimize their app’s discoverability.

A smarter way about this might be to choose a category that is somewhat related to your app but includes weaker competitors and therefore leaves you with a greater probability to be found or even being listed as a trending app. Since our app was about the organization of happenings with your friends, we chose lifestyle. An interesting read about this topic is Choosing the Right App Category by Jonathan Fishman.

Content optimization

The next obvious thing to do was to hire an ASO expert to optimize our app store listing’s text. After an explanation and introduction to our app, Robert, the Fiverr freelancer of our poor choice, would get to work and improve our listing.

He would go about it by constructing a text that includes high-ranking buzzwords that aren’t used by too many other apps, like Beer, Party and meeting people. After all, these are all things you can do if you organize happenings with your friends.

As a result, we had a listing that was full of slightly unusual phrasings but still kind of managed to bring across what the app was all about. At least in the eyes of its inexperienced creators.

In hindsight, the first red flag was that Robert didn’t seem to fully understand our app’s purpose. We brushed it off by telling ourselves he only has to deliver ASO-optimized keywords, another poor assessment on our part that, looking back, should’ve been easier to avoid.

Unmet expectations lead to frustration

Our learning came shortly after we applied our ASO changes: They proved to be effective insofar that more people found our app through various search queries. But disaster was also imminent: We wave of one-start-ratings incoming. Getting rid of those is essentially impossible once they’re there, even if you fix the problem. It is also tremendously harder to motivate people to write positive reviews to make up for the bad ones.

As it turned out, people installed the app and created an account with vastly different expectations. Häpps helps people to organize events and happenings with their friends but it was found by people who expected an app where they can meet new people.

It’s a tough effort to get people to spend time on an app review. Disappointing people’s expectations seem to just do the job: The intelligible frustration drove many to leave us one-star ratings.

Although our app store appearances eventually turned out to be our most reliable source of new users, optimizing it proved to be much harder and more time-consuming than we thought.

For now, we ended up phrasing the text in a way so that it was easier to understand the app’s purpose. We also removed all the buzzwords that we considered to be misleading. These were small changes, but they worked: We didn't receive any more bad reviews after applying the fixes and ended up with a small but steady stream of new users.

Moreover, the unintended experiment showed us how powerful the influence of single words and phrases on people's behavior can be.

[Photo by 愚木混株 cdd20 on Unsplash]

What looked like a promising idea turned out to be the start of building multiple apps. Each made it to production, like häpps - Yay! 🎉 But guess what? None of them went viral and we didn't make a single penny with them.),

So, was it worth it? Totally! And here's why.

What it's worth

Even though we were already decent software developers when we started, I am sure we would never be in our current positions without building apps together. We both work as software architects, full-time developers, consultants, and product owners.

Bringing our ideas to life was much more complex than we would have thought. Building from scratch to product, and not just working on some unfinished side projects, was completely exceeding our usual skillset as software developers.

We did everything along this way on our own, from:

• frontend to backend

• wireframing to infrastructure

• product decisions to legal requirements

• user experience to software architecture

• distributing flyers to hiring freelancers

• creating marketing texts to interviewing people

• starting projects to stopping them before losing our sanity and dignity

• and so much more

When we started, we didn't have any idea of most of these things. And behind every door we opened, we encountered another world of complexity. It was and still is a constant journey of learning new things.

But after opening many of those doors, my view of my job and skills has changed. Having insight into all these topics helps me to keep an eye on the big picture, appreciate what it takes to make a product successful, and learn what I'm good at and where I should get help from others.

Actually, we did earn money with it

I now have a good full-time job where I can apply all the things I have learned on a daily basis. And talking to most of my colleagues, not many can look back on such a range of experiences that is not only focused on the tech side.

Although we didn't make any money directly from the apps, we earn money from this experience in our jobs now. Because we use it to help other people. And this effect is a long-lasting one, even if AIs might replace our jobs very soon. So although we have put hundreds of hours of work into the venture, we are sure it has actually paid off.

Welcome to Appventure Time!

Learning things by doing them yourself is invaluable. But we faced many challenges along the way where we would have been happy to take a shortcut, avoid dead ends and not find them out the hard way.

That’s why we've decided to share our experiences and the lessons we've learned on this journey. We want to share our decisions about technical issues, business issues, fuck-ups, personal learnings, and much more. For example:

• Why we did not choose a cloud-native infrastructure and what we did instead

• The real costs to develop and run our app

• App launch - expectations vs reality

• Approach and challenges of implementing a real-time chat with push notifications

• Coping with legal requirements & support

• and much more!

So join us as we explore the world of app building and all the fun and challenges that come with it. 😊
Subscribe to our newsletter or follow us on Hashnode to get notified when we publish new posts. Or just leave a comment on what topics you might be interested in.