My experience with a talking doll project coded in C highlighted the complexities of managing multiple inputs and outputs using raw threads. This experience underscored the absence of Go's streamlined concurrency features, such as channels and lightweight goroutines.

This article aims to bridge the gap left by many Go tutorials by delving into the fundamentals of concurrency, comparing Go's approach with the more manual methods in C.

What is concurrency?

You may need to know parallelism as well, this will make us easier to understand the concept!

Concurrency means structuring our program to handle multiple tasks at once, but not necessarily doing them at the same time. People like to use this kitchen analogy. Imagine you are cooking a dish. You need to do several things and switching between: start boiling water, meanwhile you can chopping onions, then get back to the boiling water. When you only have one CPU core, concurrency is an illusion of multitasking which basically it is not! It’s about managing lot of things and not doing all of them at once.

Parallelism means doing multiple things at the same time. Let’s get back to the kitchen analogy. Now you have a cook helper that can help you chopping onions while you are boiling water. Both of you doing multiple things at the exact same time simultaneously. You need to have multiple CPU core to do this. It’s about execution, not structuring.

When implementing concurrency, sometime we don’t care how the computer is executing things. Theses tasks could be run in single core or might be run in multiple core, the programming language is helping us to structure the process to make it look like run simultaneously.

Concurrency in Your Language

Many programming language have different way to implement concurrency. We are focusing on C and Go.

I think, Go’s concurrency it is easy to understand. However if we don’t understand how it is in C, we will not appreciate the luxury we have in Go. When I worked with C threads, every mistake was painful. Race conditions is first class concern, deadlock is your friend, anything you named it.

Go gives a safer tool to implement concurrency, but the core problems don’t disappear. Do you familiar with these: nil pointer panic, resource lock, deadlock, etc.

We still need to grasp the fundamentals, I think C is the best way to start.

How to re learn concurrency (may includes un learn)

1. Learn basic of concurrency in Go

Let’s stop pretending you have unlimited time. If you need to get code running—now—Go’s concurrency model is the shortest path between problem and solution. Here’s the bare minimum to get started:

package main

import (
    "fmt"
    "time"
)

func processTask(id int) int {
    fmt.Printf("Task %d starting\n", id)
    time.Sleep(1 * time.Second) // Simulate long running process
    fmt.Printf("Task %d done\n", id)
    return id * id
}

func main() {
    for i := 1; i <= 3; i++ {
        go processTask(i)
    }
    time.Sleep(2 * time.Second) // Wait for tasks to finish (not production-safe, but fine for demo)
    fmt.Println("All tasks launched.")
}

See it by yourself, no explanation needed at this point.

2. Recognize several concurrency patterns.

Let’s start listing popular concurrency pattern in Go, here are the patterns every junior should know without excuse. These patterns introduce correct and maintainable concurrent code, which save your life in the long run.

pattern 1: Fan out / fan in

When to use: You have a bunch of work and want to split it among workers, then collect the results.

You are processing multiple things at once. Try experimenting processing things only using single loop, do you spot the different. Is it faster?

package main

import (
    "fmt"
    "time"
)

func processTask(id int) int {
    fmt.Printf("Task %d starting\n", id)
    time.Sleep(1 * time.Second) // Simulate long running process
    fmt.Printf("Task %d done\n", id)
    return id * id
}

func worker(jobs <-chan int, results chan<- int) {
    for job := range job
        results <- processTask(job)
    }
}

func main() {
    numJobs := 10
    numWorkers := 3
    jobs := make(chan int, numJobs)
    results := make(chan int, numJobs)

    // Send jobs
    for j := 1; j <= numJobs; j++ {
        fmt.Printf("Sending job %d\n", j)
        jobs <- j
    }
    close(jobs) // Close the jobs channel to signal no more jobs will be sent

    // Fan-out: launch multiple workers
    for w := 1; w <= numWorkers; w++ {
        go worker(jobs, results)
    }

    // Fan-in: collect results
    for a := 1; a <= numJobs; a++ {
        res := <-results
        fmt.Printf("Result received: %d\n", res)
    }
}

pattern 2: Pipeline

When to use: Data needs to be processed in stages (think: factory line).

You can actually spawn multiple identical or different worker per stage, while they still meeting to the same stage. I have never use this on my usecase however.

package main

import (
    "fmt"
    "time"
)

func processTask(id int) int {
    fmt.Printf("Task %d starting\n", id)
    time.Sleep(1 * time.Second) // Simulate long running process
    fmt.Printf("Task %d done\n", id)
    return id * id
}

func processTaskAdvanced(id int) int {
    fmt.Printf("Advanced Task %d starting\n", id)
    time.Sleep(1 * time.Second) // Simulate longer process
    fmt.Printf("Advanced Task %d done\n", id)
    return id * id * 2
}

func workerStage1(jobs <-chan int, results chan<- int) {
    for job := range jobs {
        results <- processTask(job)
    }
    close(results)
    fmt.Println("Stage 1 processing complete")
}

func workerStage2(jobs <-chan int, results chan<- int) {
    for job := range jobs {
        results <- processTaskAdvanced(job)
    }
    close(results)
    fmt.Println("Stage 2 processing complete")
}

func main() {
    jobs := make(chan int, 10)
    // Send jobs
    for i := 1; i <= 10; i++ {
        jobs <- i
    }
    close(jobs) // Close the jobs channel to signal no more jobs will be sent

    // Single worker for stage 1
    stage1 := make(chan int)
    go workerStage1(jobs, stage1)

    // Single worker for stage 2
    stage2 := make(chan int)
    go workerStage2(stage1, stage2)

    // Collect results from stage 2
    for result := range stage2 {
        fmt.Println("Result:", result)
    }
}

pattern 3: Worker Pool

When to use: You need to control concurrency—don’t spawn a million goroutines and pray.

When you encounter random slowdowns, you can still process the other by spawning multiple worker.

package main

import (
    "fmt"
    "time"
)

func processTask(id int) int {
    fmt.Printf("Task %d starting\n", id)
    time.Sleep(1 * time.Second) // Simulate long running process
    fmt.Printf("Task %d done\n", id)
    return id * id
}

func worker(id int, jobs <-chan int, results chan<- int) {
    for j := range jobs {
        fmt.Printf("worker %d started job %d\n", id, j)
        results <- processTask(j)
        fmt.Printf("worker %d finished job %d\n", id, j)
    }
}

func main() {
    jobs := make(chan int, 100)
    results := make(chan int, 100)

    // Start 3 workers
    for w := 1; w <= 3; w++ {
        go worker(w, jobs, results)
    }

    // Send 5 jobs
    for j := 1; j <= 5; j++ {
        jobs <- j
    }
    close(jobs)

    // Collect results
    for a := 1; a <= 5; a++ {
        res := <-results
        fmt.Println("result:", res)
    }
}

pattern 4: Select Statement

When to use: You need to multiplex—wait for multiple things at once.

Try running the program. Observe that the program is receiving from both channels without blocking one another.

package main

import (
    "fmt"
    "time"
)

func main() {
    ch1 := make(chan string)
    ch2 := make(chan string)

    // Goroutine 1
    go func() {
        time.Sleep(1 * time.Second)
        ch1 <- "message from ch1"
    }()

    // Goroutine 2
    go func() {
        time.Sleep(2 * time.Second)
        ch2 <- "message from ch2"
    }()

    // Using select to handle multiple channels
    for i := 0; i < 2; i++ { // 2 since we expect two messages from both channels
        select {
        case msg1 := <-ch1:
            fmt.Println("Received:", msg1)
        case msg2 := <-ch2:
            fmt.Println("Received:", msg2)
        }
    }
}

pattern 5: WaitGroup

When to use: You need to wait for a bunch of goroutines to finish before moving on.

Observe the goutput order may vary due to goroutine scheduling.

package main

import (
    "fmt"
    "sync"
    "time"
)

func main() {
    var wg sync.WaitGroup

    for i := 0; i < 5; i++ {
        wg.Add(1)
        go func(n int) {
            defer wg.Done()
            time.Sleep(time.Second) // Simulate work
            fmt.Println("Hello from goroutine", n)
        }(i)
    }
    wg.Wait() // Block until all goroutines call Done()
}

pattern 6: Mutex

When to use: You need to protect shared state—because goroutines WILL stomp on each other if you don’t.

Observe the output of the program to see how the counter is incremented by different goroutines.

package main

import (
    "fmt"
    "sync"
)

func main() {
    var wg sync.WaitGroup

    var mu sync.Mutex
    counter := 0

    for i := 0; i < 10; i++ {
        wg.Add(1)
        go func(n int) {
            defer wg.Done()

            // Simulate some work
            mu.Lock()
            counter++
            fmt.Printf("Counter at goroutine %d: %d\n", n, counter)
            mu.Unlock()

        }(i)
    }

    wg.Wait()
}

// Output:
// % go run main.go
// Counter at goroutine 9: 1
// Counter at goroutine 4: 2
// Counter at goroutine 0: 3
// Counter at goroutine 1: 4
// Counter at goroutine 2: 5
// Counter at goroutine 3: 6
// Counter at goroutine 7: 7
// Counter at goroutine 8: 8
// Counter at goroutine 5: 9
// Counter at goroutine 6: 10

3. Understand what goroutine and channel is

Before you copy-paste another “concurrent Go example,” stop and ask yourself these two questions:

question 1: How many tasks do you actually need?

If you just want to run something in parallel, use a goroutine.

go doSomething()

That’s it. No threads, no memory management, no boilerplate.

#include <pthread.h>

void* doSomething(void* arg) { /* ... */ return NULL; }
pthread_t tid;
pthread_create(&tid, NULL, doSomething, NULL);
pthread_join(tid, NULL);

In C, you have to create a thread, pass a function pointer, manage the result, then join. Every single time. If you forget to join, you leak resources. If you pass the wrong pointer, you get undefined behavior. Go’s goroutine is a single keyword.

question 2: How do you pass data between tasks?

If your goroutines need to talk to each other, you need a channel.

ch := make(chan int)
go func() {
    ch <- 42 // send data
}()
fmt.Println(<-ch) // receive data

Direct, safe, and built-in. No locks, no structs, no memory headaches.

#include <pthread.h>
#include <stdio.h>
#include <stdlib.h>

typedef struct {
    int value;
    pthread_mutex_t lock;
} Data;

void* worker(void* arg) {
    Data* d = (Data*)arg;
    pthread_mutex_lock(&d->lock);
    d->value = 42;
    pthread_mutex_unlock(&d->lock);
    return NULL;
}

int main() {
    Data d;
    d.value = 0;
    pthread_mutex_init(&d.lock, NULL);

    pthread_t tid;
    pthread_create(&tid, NULL, worker, &d);
    pthread_join(tid, NULL);

    pthread_mutex_lock(&d.lock);
    printf("%d\n", d.value);
    pthread_mutex_unlock(&d.lock);

    pthread_mutex_destroy(&d.lock);
    return 0;
}

That’s so many lines of setup just to pass one integer. Miss a lock, your value is garbage. Miss a destroy call, you leak memory. Channel exists so you don’t have to write this anymore.

Why does this matter?

In Go, you focus on the problem that matter to you. It help you to ignore problem in concurrency, make it possible for you to ship fast.

Remember, the problem is still there

You think just because you’re using Go, all your concurrency problems vanish? Wrong. ere’s a classic mistake: leaking goroutines and memory.

package main

import (
    "fmt"
    "time"
)

func leakyWorker(done chan bool) {
    for {
        select {
        case <-done:
            fmt.Println("Worker stopped")
            return
        default:
            // Simulate work, but never actually receive "done"
            time.Sleep(100 * time.Millisecond)
        }
    }
}

func main() {
    done := make(chan bool)

    // Start 5 workers, but never signal them to stop
    for i := 0; i < 5; i++ {
        go leakyWorker(done)
    }

    fmt.Println("Main finished, but workers are still running and eating memory...")
    time.Sleep(5 * time.Second)
}

What’s wrong here?

• Go routine is started, but never stopped.

• done channel is never closed or signaled.

• Worker is looping forever, eating RAM and CPU.

Imagine doing inside HTTP handlers or message consumer. You will kill your server eventually.

Go is safe, but it’s not idiot-proof. It’s just tiring to see this issue in production.

4. Let’s normalize “it depends” statement

If you are looking for a silver bullet, you are wasting your time. How many time do I witness overusing concurrency without meaningful goal, just making the code hard to read and maintain!

“It depends”, is not a lazy answer. It’s the only honest answer in real life.

Don’t just follow a blog post that uses channels everywhere. They are just a blog post after all.

Analyze the context, what’s the problem you are trying to solve? Ask your favorite AI to deep dive into the problem, start comparing multiple solutions.

Now that you see different solutions, validate both the problem solving and the implementation. Is it satisfying the requirement? Is it safe to run in production?

Conclusion

In this era, people easily forget the fundamentals. AI spits out code, people just straight to “tab tab tab and push”. When you don’t know a thing, you are just a monkey pushing buttons.

Real engineer review fundamentals. Want to design a solid architecture? Start learning system design. Want to write a solid concurrent Go code? See underneath it, how concurrency really works in C. If you never fought with threads, locks, and memory leaks in C, you will not appreciate the power Go gives you.

Stop being a code monkey. Before you hit accept, read carefully. Test it, break it, fix it. You are senior reviewing your junior AI robot’s work. If you don’t catch the bugs, nobody will.

Respect the fundamentals. Use the tools, don’t let them use you. If you are really a senior, act like it.

Having a Goal in Mind

While it's possible to start analyzing metrics immediately, it can be beneficial to begin with a personal goal. This goal can help you stay focused and motivated. Ask yourself what you aim to achieve - is it faster delivery, fewer bugs, or improved estimation?

Start with Popular Metrics

Get familiar with established metrics like something as follow:

• DORA Metrics (Deployment Frequency, Lead Time for Changes, Change Failure Rate, Time to Restore Service): These offer insights into your team's and potentially your individual impact on delivery and stability.

• Cycle Time: How long it takes for your code contributions to go from start to deployment.

• Throughput: The number of tasks or stories you complete within a specific timeframe.

• Code Quality Metrics (e.g., bug count in your code): While harder to isolate individually, tracking bugs related to your changes can be insightful.

Getting Started

Tracking key metrics can provide valuable insights to enhance your decision-making and optimize your workflow. Here's how to get started in two simple steps:

• Automate Data Capture: Manual tracking is inefficient. Explore using local tools or scripts to automate the process. If your company permits, consider implementing data capture on company servers, ensuring secure access and the potential for sharing anonymized data with your team.

• Create a Visualization: Once you've collected data, visualize it to uncover trends and patterns. Even a basic spreadsheet with charts can effectively reveal insights that align with your objectives.

How do you actually start?

Don't be discouraged by the lack of advanced tools. Begin with the data you can access. I started by analyzing my JIRA ticket history to measure my average cycle time and throughput. You could also track incidents directly caused by your code changes and measure your average time to recovery.

Take the initiative and act instead of waiting for others to do it for you.

Lesson Learned

While my manager doesn't require detailed productivity metrics, tracking my own has been incredibly beneficial. It helps me objectively evaluate my performance, pinpoint areas for improvement, and focus on professional development. This data-driven approach provides concrete evidence to support my self-assessments and career growth.

Tracking your productivity isn't about micromanagement; it's about self-awareness and empowerment. In the rapidly changing landscape of the Fourth Industrial Revolution, understanding your capabilities through data will make you a more disciplined and effective professional. It's an investment in your personal and professional growth.

I'd love to hear about your experiences with tracking your own productivity. What metrics have you found most valuable?

What does “engineer” actually mean?

Writing code is just a small essence of software engineering, what it truly lies is in applying a systematic and disciplined approach to solve problems and create appropriate solutions.

1. Problem identification
   Start by identifying the goal of the project. Usually, it comes as a problem statement and brief background.

2. Research the background
   Proactively research the problem given to you and possible solutions from similar problems.

3. Requirement specification
   Prioritize and list the most important point for the project, one project with another will be different even if it is similar.

4. Brainstorm the solution
   Think of all the alternatives. Don’t just focus on one solution, a risk overlooking a better solution.

5. Develop the solution
   Communicate the solution properly to the executor. Have proper control to be able to meet the plan.

6. Test and revise
   Be ready to revise your solution once it has been developed. In some cases, you don’t have the privilege anymore to restart everything from point zero.

7. Result communicated
   Set up a demo, presentation, report, or display to others. Pretty sure, as a project owner, you are working on this project on behalf of someone else.

My Kitchen Renovation Project: an unexpected engineering masterclass

As the project owner of my kitchen renovation, where I report directly to the stakeholder (my family: wife, kids), I found myself relying heavily on the very same principles I use daily as a software engineer:

1. Problem identification
   My family and I meticulously thought about our needs for the new kitchen space. We also discussed some of the characteristics like the flow of usage, aesthetic, budget, and deadline.

2. Research the background
   I personally spent hours on kitchen layout and design trends on pinterest, materials on local depository, kitchen appliances on shopee.

3. Requirement specification
   Since I could not copy paste what I have seen on pinterest, I began listing down some requirements that need to be finished given the problem we have set.

4. Brainstorm the solution
   Almost for every item, I come up with some alternatives. The layout for example, deciding on L shape or C shape comes with pros and cons. As well as choosing ceramic over granite, and others.

5. Develop the solution
   I hired one senior construction worker and a junior helping him. I communicate my plan in a language we all understand. They also gave feedback when my plan needed more effort or was uncommon, so we all can still achieve the project within the budget and deadline.

6. Test and revise
   No matter how perfect your design is, the execution could go wrong unexpectedly. That’s when you need to revise your design and come up with another solution.

7. Result communicated
   For my case, I invite my family over about weekly or biweekly to check the progress. I presented the progress we have reached so far.

The Universal Discipline of Engineering

My experience working on the kitchen extension project showed me that being an “engineer” is like having a special way to solve problems, any problems. It does not matter if you are writing code or building a kitchen ceiling: you still need to know the problem is, plan all the way, do the work, and finally check if it is right.

Shaping Your Career and Personal Growth

Understanding this universal definition can be incredibly empowering you personally and most importantly your career!

1. Enhanced Problem Solving
   You will have a structured thinking on identifying the problem, proposing the solution, up to work on it effectively.

2. Improved Communication
   The planning session will help you to articulate your thought process and invite a healthy discussion with all stakeholders. You also get a chance to collaborate with individuals from different backgrounds and different intentions, focusing on the shared language.

3. Increased Adaptability
   You will come to realize that a perfect plan will break eventually, by applying your engineering mindset for a dynamic situation makes you more versatile.

4. Personal Empowerment
   By recognizing your inherent problem solving, it boosts your confidence to tackle challenges beyond coding.

Conclusion

The last kitchen extension project really showed something for me: I'm an engineer, not just a software engineer. Seeing how my problem-solving skills translated so well, and with my family happy with the result (and me too!), it inspired us for other future projects like building a kitchen cabinet, carport, and even handling electricity and AC installation.

Of course, as a professional Software Engineer, I need to carefully balance the time I dedicate to these personal projects with the crucial effort of honing my core software engineering skills. :(

Anyway! Have you ever found your “engineering brain” kicking in outside of your domain realm? Share your experiences in the comment below! What unexpected projects or situations have you approached?

Later, I learned that one-on-one meetings are not an annual event to discuss my employees' performance only. As a Tech Lead, I schedule biweekly 1:1 meetings with all my team members. Of course, I also regularly have 1:1 meetings with my manager.

I have come to appreciate just how valuable this meeting can be for everyone, including the company. In this article, I will tell you What is 1:1 meeting? What could you expect? How do you benefit from it?

What is 1:1 (one on one) meeting?

I will take the definition from “The Effective Manager” by Mark Horstman

A regularly scheduled meeting between a manager and each of their direct reports to discuss anything the direct wants to discuss.

Let’s take a key takeaway from it:

1. Regularity is crucial, consistent is the key

2. Focus individually personally, that’s why there is only you and your manager

3. Owned by a direct report, not the manager. We own the agenda, not our manager.

Another definition also comes up from my favorite book “The Manager’s Path” by Camile Fournier

An essential feature of good working relationship

She explains the reason of why 1:1 exists:

1. To create a human connection between us and our manager

2. To create a regular opportunity for us to speak privately with our manager about whatever needs discussing.

What you should expect

Typical 1:1 sessions may vary, depending on your needs and the culture of organization. Commonly, people use it for status updates, or to dive deep into ongoing or recent projects, although I prefer to avoid this scenario if we already have daily update meetings.

Worst case scenario, you never had 1:1 session with your manager except twice a year talking about the regular performance review cycle. I highly avoid this kind of situation, since you don’t prefer your feedback to be given only on performance review, it's too late.

There are some agenda that you may consider bringing up in your next 1:1 with your manager:

1. Feedback

In this scenario, you can bring up the recent project. Directly, ask him about things you could improve, then he will gladly give you the feedback or deliver the feedback collected from others. Focus on the correct way of doing it, although not every theory is practical.

2. Guidance

Quite similar to when you receive feedback, here you focus on the guidance of how you should perform and behave in the company. Remember, not every theory is practical, that’s when you need practical guidance, “this is how we do it in this company”

3. Career Development

Another portion to talk about is making sure you still perform in your current role, at the same time you still strive for mastery to excel. This is very important, even if you are not ambitious to participate in the next promotion, life is quite boring when you haven’t had a challenge. Ask for a project to stretch, so you can taste the unknown unknown to uncover your potential.

4. Personal matter

The small portion of the agenda you could bring up is to talk about personal matters and concern related to work. Sometimes, people bring up annual bonuses or changes in company policy. Do not expect your manager to know all the details, they probably as clueless as you, but rest assured that he will bring it up to higher-up management. If you want a job rotation to another product team, this is also where you can talk about it.

How to prepare

1. Come up with your agenda

Remember that this is your agenda. Although, it is 100% ok for your manager if they come with his agenda. Just be sure you also had one.

Depending on the company culture, your manager is the one to schedule the meeting regularly. He always comes up with his agenda. This is usually happen when you are still junior, since you probably had no idea what to talk about.

Regardless of the situation, try to make a bullet point on what do you want to bring up. Refer to the some agenda I had mentioned earlier.

2. Be honest and open

Be vulnerable, there is no point in defending your statement and behavior in this session. There is just no point. You need to receive as much as you can.

I had a friend who always had trouble with his manager. My friend had their own opinion on how things should work, which was not aligned with his manager. In this situation, the best way to come up is not to force the manager. Remember, “this is how we do it in this company” is all you need.

The thing is, you may need to deliver feedback to your manager. This is totally fine. Although you can not force your opinion on everyone, as a professional who sees flaws in the company, you need to raise that right away and come up with a possible solution.

3. Take notes and actionable items

Walk the talk. Don’t let your valuable to go waste. Take a note and come up with an action item.

Too lazy on taking a note, invite AI note taker that you can integrate into Google Meeting. There are plenty options available. Even if you had the session offline in the office.

You probably do not to do active note-taking. During the meeting, you may need to focus on the talk and take one or two keywords for you to develop later on. Remember, you still need to create a human connection with your manager.

Typical 1:1 Session Run Down

For you who had not had a 1:1 meeting session before, I will give you a reference on what the session look like.

• 00:00 - 00:05 → small chit chat
  Get to know personally. Bring up a light topic like “how did you get home yesterday?”, “Had you catch a rain on the way home?”, “The meeting with the product yesterday was quite awful, we could do better, bla bla bla.”

• 00:06 - 00:35 → your main agenda, about 1 or 2 things
  Bring a recent event to the table, ask your manager on how you could be better with it. Ask for feedback or guidance. (refer to previous section)

• 00:36 - 00:45 → additional agenda
  Spare some time for your manager’s agenda, you never want to miss what they want to talk to you.

• 00:46 - 01:00 → buffer time
  Use it if you have to

Conclusion

Just like everybody else, I had mistaken what 1:1 for a regular performance review cycle and status update. I missed the opportunity to create connection and growth. The misconception is common, and that’s why not everybody agrees with this concept.

Even now, I still face difficulty on maintaining my 1:1, with my team members and my manager. While it is easy for me to control the agenda for my manager, I have consciously tended to update the latest and ongoing projects. Toward my direct reports, while it is easy for me to leave the agenda to them, I tend to preach for a solid 1 hour without ever actively listening to their story and concerns.

Afterall, for sure I had the best professional connection in my career once I had my regular 1:1 session.

What is your experience with 1:1 meetings? Do you have them regularly, if not what stopping you? Let’s learn from each other.

References:

• Fournier, C. (2017). The manager’s path: A Guide for Tech Leaders Navigating Growth and Change.

Yet, many still default to them. This is a dangerous game. So trying out an idea with a simple MVP project should not be in your hand. “Premature Optimization Is the Root of All Evil”, indeed, but underestimating the problem is a guaranteed path to disaster.

If you are reading this, it is because you understand this fundamental truth: complex problems require well considered architectures. You are here to find the right one, and I commend you for it 👍

Some Architectural Patterns

Choosing the right architecture needs a deep understanding of your project requirements. Once you have a clear understanding, you can explore suitable architectural patterns. Here are some common architectures.

Layered Architecture

Organize application into horizontal layers, each with specific technical responsibility (e.g. presentation, business, data access).

Suitable for simpler application with clear separation of concerns. Easy to understand and implement, but can become difficult to scale for complex systems.

Service Oriented Architecture

A collection of services that communicate with each other with a central service bus. Note that it is not a microservice.

Pipeline Architecture

Process data through stages or filters, each performs specific transportation on the data. It consists of filters and transformers.

Suitable for applications that process data in sequential manner.

Microkernel Architecture (Plug-in Architecture)

Consist of core system and optional microkernel (plug in module). This allows for extensibility and customization without modifying the core system. Think of web browser extensions.

Suitable for systems with high extensibility and customization.

Event-Driven Architecture

Focus on applying a particular process for a particular emitted event. When implementing, it can be challenging to debug and trace the flow.

Suitable for application with real time data processing, asynchronous communication, and complex event processing.

Microservices Architecture

Decomposes the application into small and independent services that communicate with each other over the network. Each service has its own clear business justification.

Suitable for complex applications with high scalability, independent deployment, and diverse tech stacks.

Orchestration-Driven Architecture

It involves a central orchestrator that coordinates the interaction between services. Not only do you need to maintain the process flow, but the running infrastructure as well.

Suitable for complex business processes that need to be coordinated all together through central command and monitoring.

Practical Tips

Effectively navigating complex architectural challenges requires a structured approach. Here are some actionable tips to guide you:

1. Deeply Understand the Problem

Don't rush into solutions before truly grasping the problem's nuances. Dedicate focused time to gather information. I find a focused session (like a Pomodoro) extremely valuable for this.

During this time, gather all available facts, evidence, and documentation. Critically verify incoming requirements and analyze the problem domain. Key aspects to consider include:

• Background: What is the context of this problem? What are the business drivers behind it?

• Scope: What are the boundaries of the problem? What is included and excluded?

• Explicit and Implicit Requirements: What are the stated requirements, and what are the unspoken assumptions or implied needs?

• User Flows: How will users interact with the system? What are the typical user journeys?

• Data Schema: What data will the system handle? What are the relationships between different data elements?

2. Strategically Review Architectural Patterns

Familiarize yourself with common architectural patterns and their trade-offs. As mentioned earlier, common patterns offer different strengths and weaknesses. The goal is not to force-fit a pattern but to select the one that best aligns with the specific requirements of your problem. Remember: there's no silver bullet. Consider these factors:

• Alignment with Requirements: Does the pattern address the key scalability, performance, security, and maintainability needs?

• Company Standards: Does the pattern align with existing company norms, standards, and infrastructure?

• Team Expertise: Does the team have the necessary skills and experience to implement and maintain the chosen pattern?

3. Validate with a Targeted Prototype

Before committing to a full-scale implementation, build a small, focused prototype to validate key assumptions and identify potential roadblocks early. This allows you to experiment with different approaches and refine your design before investing significant development effort.

While AI assistants can be helpful for quick explorations and code generation, remember that a well-designed prototype should still be carefully planned and executed to provide meaningful insights. Focus on these aspects:

• Key Functionality: Focus on implementing the core features that are most critical to validate the architectural choices.

• Performance Testing (if applicable): If performance is a critical concern, include basic performance testing in the prototype.

• Feedback and Iteration: Use the prototype to gather feedback from stakeholders and iterate on the design as needed.

4. Conduct Regular Architectural Reviews

Architectural reviews are essential throughout the project lifecycle, not just at the beginning. Regular reviews foster shared understanding within the team, identify potential issues early, and ensure that the architecture remains aligned with evolving requirements. These reviews should be:

• Regular and Scheduled: Establish a regular cadence for reviews (e.g., bi-weekly or monthly).

• Well-Prepared: Come prepared with specific topics to discuss and any relevant documentation.

• Documented: Use tools like Architecture Decision Records (ADRs) to document architectural decisions, their rationale, and any alternatives considered. This creates a valuable historical record and facilitates future discussions.

Conclusion

The practical tips outlined above—understanding the problem, reviewing architectural patterns, prototyping, and conducting reviews—are valuable in various scenarios: starting a new project, integrating with existing systems, adding complex features, or pursuing overall system improvements. However, simply knowing these steps isn't enough. The effectiveness of each step is crucial. This begs some important questions:

• How do you truly understand a complex problem? What techniques do you use to gather requirements and analyze the problem domain?

• How do you efficiently identify relevant architectural patterns? Do you have a systematic approach to evaluating trade-offs and selecting the best fit?

• How do you create a compelling and informative prototype that effectively communicates your architectural vision?

• How do you clearly and persuasively communicate your architectural decisions to stakeholders, ensuring buy-in and shared understanding?

If you're still with me, it's because you recognize that mastering these skills is an ongoing journey. Our discipline has developed countless frameworks, methodologies, and tools to enhance our effectiveness at each of these stages. From domain-driven design and use case analysis to architectural decision records (ADRs) and various prototyping techniques, there's a wealth of knowledge available. The danger lies in "wandering around," reinventing the wheel, and not leveraging these existing resources.

This moment is an opportunity for reflection: to acknowledge the vast landscape of knowledge that you may not yet have fully explored—the "unknown unknowns." Recognizing these gaps is the first step towards bridging them and becoming a more effective and impactful software architect.

Now it is your turn. Reflect on a complex problem you have faced and how you approved the architectural challenges. Above of all, what were the biggest lessons you learned? Share your experiences, insights, and even your mistakes in the comments below. Let’s learn from each other and collectively improve our architectural problem-solving skills.

Ford, N., & Richards, M. (2020). Fundamentals of software Architecture: An Engineering Approach. O’Reilly Media.

Khononov, V. (2021). Learning Domain-Driven design: Aligning Software Architecture and Business Strategy. O’Reilly Media.

The Guide to the Software Engineering Body of Knowledge (SWEBOK Guide)

Everyone wants simplicity. Customers, for instance, love services like "Buy Now, Pay Later"—a simple way to shop without worrying about payment. What’s hidden behind this simplicity, though, is a complex system that manages loans and installment financing.

Simplicity is tempting, but opting for an overly simplistic system design for complex problems can lead to bigger issues down the road. While simple architectures work well for MVPs, what happens when your business needs to solve more than just simple problems?

Definition of Simple Architecture

What is Layered Architecture?

In many software projects, layered architecture is a simple, commonly used approach. It divides the system into distinct layers, each with a specific responsibility. These layers typically include:

• Presentation Layer:
  This layer handles external interactions. In frontend (FE) projects, it deals with the UI and user input. In backend (BE) projects, it typically handles API requests and responses (controllers).

• Business Logic Layer:
  The core of the system, where business rules and logic are implemented. This layer focuses on fulfilling stakeholder requirements, abstracting away technical details.

• Persistence Layer:
  Responsible for interacting with storage systems (like databases). It usually contains functions or methods for reading from and writing to databases.

• Database Layer:
  This is where your data resides. Whether it's a relational database, NoSQL, or in-memory database, this layer manages data storage, schema design, indexing, and keys (such as in Redis).

How It Works

Layered architecture follows the separation of concerns principle, meaning that each layer is responsible for its specific tasks. The layers are technically partitioned—each group of components solves a specific technical role within the system.

One of the key rules in layered architecture is how components interact across layers. Notice in diagram below, where Services component is interacting to Entity, then Entity is interacting to Value Object.

Typically, components in a higher layer (e.g., Presentation) can access components in the layers below (e.g., Business Logic or Persistence), but lower layers cannot directly call higher layers.

However, in practice, challenges arise. For example, you may run into issues where components from different layers need to communicate, or you might be tempted to skip a layer to make development more convenient (e.g., bypassing the Business Logic layer to directly query the database).

While this architecture is intentionally simplistic, it works well when a project is starting out, as the clear separation between layers makes it easy to organize and manage the application.

When to Use (and When Not To) — Practical Tips

1. It is good for starters.
   Layered architecture is an excellent choice for beginners due to its simplicity and clear structure. It provides a solid foundation for understanding basic architectural principles without overwhelming complexity.

2. Use it on small to medium-sized applications.
   This approach works well for applications that don’t require extensive scalability or performance optimization. For small to medium projects, it ensures maintainability and straightforward implementation.

3. Clear, well-defined separation of concerns.
   The architecture promotes a clean separation of concerns, with each layer focusing on a specific responsibility. This makes the codebase easier to understand, test, and maintain.

4. Avoid it for complex systems with evolving requirements.
   Layered architecture struggles to adapt to complex systems with dynamic and growing requirements. Its rigid structure can lead to bottlenecks and difficulty scaling as the project evolves.

5. Not for high-performance apps.
   Applications requiring high performance may outgrow the layered approach, as the additional abstraction layers introduce latency and can become a limiting factor under heavy load.

6. Prevent using it for systems with dynamic needs.
   Systems with rapidly changing or unpredictable requirements demand flexibility, which layered architecture lacks. It is best suited for stable environments where requirements are well-defined.

My Thought on Layered Architecture

After the microservices hype, simple architectures gained attention. At first people use microservice as a silver bullet, making a lean engineering team to manage dozens of services that could be handled with a more straightforward layered architecture. This again created a misunderstanding, where layered architecture was seen as a one-sizes-fits-all solution, without considering how projects could evolve into more complex systems over time.

In some cases, teams are tempted to apply a simplistic design to an already complex problem, which can backfire as the project grows. Understanding the limits of simplicity is crucial, as it might lead to difficulties down the line.

My advice: understand when simple architecture is appropriate and, just as importantly, when it isn’t. Knowing the right time and context for applying this approach is key to managing complexity in evolving projects.

Let’s continue the discussion in the comment section, on why layered architecture is not suited for complex problem 🙌

Conclusion

While a simple layered architecture can work well for smaller applications or well-defined problems, it is not a one-size-fits-all solution. As software problems grow in complexity, the need for a more flexible and scalable architecture becomes apparent. If you're working on a complex system, consider adopting an architecture that supports flexibility and scalability from the beginning.

Are you working on a system that could benefit from a more complex architecture? Share your experience in the comments or reach out to discuss how you can optimize your software architecture for the challenges ahead.

But hey, we also asked to raise our opinion. They say, because discussion matters. Eventually, we will make an agreement on how things will be done. It requires everybody to commit to the result of the discussion.

For these past few months, I have just realized what the word "commit" truly means. You can find more the details here, I believe the one that we all mean is this "to carry into action deliberately".

But, what if our heart doesn't go with the discussion? We feel that this discussion is a waste of time because they started it all wrong. Committing to something requires our heart and passion, what if we do not agree with that? Burn out, huh? Need some healing?

For me, I was told to be bold about something I believe until someone gives a reasonable argument, then we change our beliefs little by little. A bit frustrating to be honest, even for myself. But what if the argument is completely unreasonable for us but everybody agrees with them? Am I wrong here?

That's when I first heard about the idea of agree to disagree.

As a newbie, I might be wrong about what it truly is. The Wikipedia's explanation about this is when all parties tolerate but do not accept the opposing positions. In my opinion, this is what we can do when we happen to be in this position in a professional working situation.

The discussion must come to an end, because we have a limit to perform the discussion. The business keeps going on during our debate, but it needs our discussion to be resolved fast. Not everyone agrees but put simply the best bet and asks everybody to commit, at least for now.

There always be another time. I mean, if you could not prove your argument now, there will always be another time to convince others. In a discussion, everyone's voices are not treated equally. Frankly, it might happen when you sit close enough with your leader, making your leader listen to you more. Or, your microphone is not performing well when discussing over an online meeting.

Act professional. Speaking your idea is an act of professionalism, opposite to silence. As a white collar, you are paid to express your idea. When you do not speak, you might consider changing your career to a blue-collar role. But hey, it is not that bad to express an opposite idea to others. Most of the time, I feel relieved when listening to the opposite idea.

So guys, next time we run to a discussion, do we agree to disagree?

I learned all of those from learning a new language and then finding the unit test framework people typically use. Various frameworks teach us different ways of making unit test, they even teach us the philosophy of unit tests.

It might not be a complete reference of various unit test strategies existing in this world, so if I missed some important thing, please let me know then I can make an update for the next persons coming.

Problem With Testing

Before moving to the point, I would like to brief you this simple problem of how to create a unit test for the very smallest part of your code.

Sometimes, we didn't have much choice when the existing code has been equipped with a unit test. As a n-th person working toward the code, your job is to make that unit test all green withstanding your very current changes. Some changes on the unit test may be required, or you may need to add new cases to an existing one.

When there is no existing unit test or you make radical changes to the code, you will mostly need to think about how to write the unit test. You want it to be simple enough to write, yet convincing enough to withstand future unlimited alteration.

Behaviour Driven

This one is by far the easiest to understand because it is so intuitive to think about. To put it simply, our flow must behave somewhat according to what we desire. To work with this strategy, you may need to write down some cases with preconditions, actions, and a set of expectations.

It is easier for other developers to read. Most unit testing frameworks that go with this strategy require us to write a simple and meaningful comment.

Most of the frameworks using this strategy also uses Gherkin notation, it is even very convenient if you are serious about assuring your code through the unit test.

Test Coverage Driven

The unit test may be not on your agenda so far. At then end, you still need to convince everyone that you have a set of unit tests to withstand future changes. So far, I believe this strategy is the easiest one to encourage people to write it because it has a clear quantitative measure.

I don't prefer this strategy of writing unit test. It encourages you to leave something when you may haven't. Since you have written a unit test that went through these lines, there is no point to add another going through these.

Let's be honest, we prefer to use this strategy because we can show this to our manager right?

Test Table

Test table comes in very handy, sometime you don't need any testing framework to write one. For newbies, coming up with the idea of writing a test table is surprising. I even hadn't seen that until someone wrote a test table on our code base.

Other than behaviour-driven, the test table is also my favourite when you have a very simple unit to test. For me, it could be a tiny function that branched out to improve the readability of the main flow. We can then bombard this small function with tons of cases.

What's next?

I also wondered up to this point, what would you need to know after knowing all of this? For me, I would like to know a real example of unit test, some parts of me wants to know the step by step of planning to write a unit test.

Let me know if you want to know something.

You have a service that has a dynamic load of incoming traffic to perform heavy-weight operations. Luckily, the client doesn't demand an immediate response from our service.

If possible, we still need to deliver the result fast enough, because the client doesn't want to wait any longer.

Solution

Let's say we have a service named P, which always receives operation requests from clients.

Service P will forward any requests to a queue named Q. Every time service P successfully forward the request, the client needs to wait for the result of the operation.

On the other side, there is n number of service C. Every service C[i] will consume any possible request from queue Q.

Normal Case

Under the hood of queue Q and every n of service C, you will notice for every request will only be processed by one of any possible service C. This is a very important remark that becomes what a message queue is.

After a service has begun processing a request, the next request is available for processing for any available service C. But that request is not effectively gone from the queue Q.

Service C shall give an indication, of whether it needs to be ACK (success and ok to delete) or NACK (error and no for delete). When that request is given an ACK command, we won't have a further issue.

Handling NACK

NACK is used as an indication that the operation of the request is failed. Retry would do, and we can give it a try. But we need to set some limits on that.

1. Simply just re-queue

2. Re-queue and track

3. Send to dead letter queue

Simply just re-queue a request. This is typicaly of what service C will do. Hoping the next service C that will process this request will do just fine. If there are n number of service C and there are n-1 needed to be requeued, there is at least one service C that will process the remaining request. It means the queue still moves on.

Re-queue and track how long it has been re-queued. We may need to give special treatment after some retrying.

Send to the dead letter queue would be the best option to send this problematic request to the dedicated queue of problematic requests. We might need to take a look at this queue later manually.

References:

• https://www.rabbitmq.com/tutorials/tutorial-two-python.html

Software Development Life-Cycle (SDLC) comes in various styles and practices, but formally we have these processes:

1. Analysis

2. Design

3. Development

4. Testing

5. Deployment

6. Documentation

7. Maintenance

For me, the "Analysis" is the hardest part. In this article, I would like to tell you why upfront requirement analysis is so important and how to deal with the way high pace startup works.

What is Requirement Analysis

Its goal is for us to have an understanding of the business requirement. To be able to do that, formally we can set this three outputs:

1. Scope of Work

2. Options evaluation

3. Cost-benefit

Most of the time, we often only produce "Scope of Work", because intuitively it can save our ass when we were missing some features.

Evaluating possible options could give us the grand picture of what would it look like. It is also a way to have mutual knowledge with the product owner. Remember, the word "options" is plural, then we need to produce some.

As a professional Software Engineer, we must communicate the possible cost and demonstrate the benefit of each option. The product owner will find it very helpful. It also prevents underestimate and overestimate feeling from stakeholders toward us.

Miss Conception of Requirement Analysis

Waterfall methodology as in SLDC is something Every computer science graduate should know, and also everyone hated. Got their first job working in Waterfall, and believe me, they will be begging for their life to move on.

For some reason, I often got confused when performing requirement analysis means we are going back to the way Waterfall does: doing it up first, continuing with the upcoming process, and never going back.

Working in a high pace startup with a very ambitious and tight deadline, for example, it is normal that we tend to skip taught-to-be non-essential formal processes. I agree that we need working software as our top priority, but how can we do that when not understanding the requirement?

Even in a near-perfect environment, performing requirement analysis also tend to be forgotten. Try to imagine, we are given very detailed business requirements, access to the business team, communicative product owner. The very first thing we gonna do is jumping into coding, what a shame.

I am not suggesting holding back product delivery with a mouthful of discussion. What we assume to be a mouthful, actually is something fruitful later process in our SDLC.

When Working in a High Pace Startup

Those who are very fans of the startup movement will find a time-consuming analysis as something wasteful. I agree with them though. Spending too much time thinking will not get us anywhere, but leads us to useless overthinking.

When I was reading Lean Startup, it just clicks into my mind, as a Software Engineer we tend to overdo analysis, design, and even coding!

Lean Startup tells us, to handle the uncertain reality of the business we need a fast and frequent feedback loop: build, measure, learn

Lean Startup is talking about business and product development, but I find resonance with what we are doing in software engineering. Since the business and product team is working under an uncertain business reality, so do we when facing product requirements. By applying this fast and frequent feedback, we are hoping to see high-quality software day by day.

We expect ourselves to analyze requirements differently by now:

1. Tactical win vs strategic movement;

2. Just do it, look it for ourselves;

3. Be friends with data usage: log, metrics, data, etc.

First Step toward Perfection

Learning by doing is the best learning method for beginners. So, there are some ways for us to practically learn how to perform requirement analysis better in a high pace startup environment.

Expose yourself to the discussion. When your product owner or tech lead presents the very basic background and product requirements, don't just stare at them blankly. Write a note, make some questions, and draw a diagram to make you understand. Direct communication is the most effective communication for thousands of years now, don't let this timing getaway.

The discussion is still running on, even after everyone has agreed. Leave a comment on the requirement, ask some questions, and ask for clarification.

Write something, talk is cheap and it will spoil in no time. Writing your requirement analysis will make your argument comes stronger than anyone. My experience proves it, we do not need a sexy leadership role to influence the technical decision.

Remember the formula, it will help you on performing the analysis. Requirement analysis produces the scope of work, choices evaluation, cost-benefit.

It is always important to showcase our achievement. When we were learning backend for example, then you need to tell your future employer that you are worth to hire. The only way that came up to me was portfolio, certification was not really a thing 4 years ago at least on my circle.

Working on portfolio gives validation that you gain something from your learning journey. It is a milestone that you have already passed certain level.

Building a portfolio was not suited to my learning characteristic. I got intrigued to learn everything until I became confused on what to learn. For example, choosing programming language to pick was not an easy stuff. After I had decided the language, some said that we need to master specific framework popular enough.

The thing is, building a portfolio wasn't really great.

I don't quite asked on my portfolio when conducting technical interview. During my first job hunting, it was take home project that they were interested to. They did ask much a glimpse of my portfolios.

Later when I was looking my second employer, they asked me on my working experience. Something that I did for my past employer.

If I had known working experience matters most, I would not have been working on portfolio, I would have got working experience like internship or part time. A true and real working experience that I can tell to recruiter, rather than something imaginary of made up usecase (portfolio).

I am not saying portfolio is a bad thing and everyone should avoid working on it. Here, I am giving you an option that might need more attention: working experience.

You may have problem on finding your first job because you don't have any experience at all. That's when portfolio matters the most. But you need to be aware, your first job is a part of your learning journey, hence you do not need to work all out on the portfolio.

As long as you learn something, able to create a simple demonstrative portfolio to validate your knowledge, you do not need to go any further to just get your first job.

I wish we have a good luck on our software engineer career journey.

This is something we had always done in coding tutorial. Since we met it in the very first phase of our learning, we often underestimate it.

We are now talking about complex data structure on our module, it is something that difficult to draw in ER diagram. Next, how are we going to create an object of that. How do we even update that? Think of a set of rule that will limit our CRUD operation.

Don't underestimate things shall we? 🧐

And now, you are living in 2022 when they pay you millions but you are dragging a critical mission about months. Shame on you, you get it over and over in the tutorial but you still could not get fast? 😳

Well, I do not blame you. We always drag others. Not because we are bad, but the problem does require a significant amount of attention on us. Be happy, you are still worth it to pay that much 😆

Again, you are living in 2022, or you have passed already. Stack Overflow is our best friend, we also have Reddit if SO doesn't give us any good. Even, there are bunches of people who have been writing an interesting project which exactly solves your problem.

By the way, we are still talking about CRUD. Did you know there is a long live open source project that can give you a rich feature back office admin dashboard called Directus?

Directus can give you a boost where you only need to think about the data structure, where it can help you for the CRUD. I use it for my latest personal project!

For you who greed enough, there even open source project that can give you both front end and back end. I have been seen it quite sometime, Budibase, it even seems as shiny as directus does. Can not wait to test it on production.

I did not get a chance to really invest on boilerplate, which I believe there are tons of it already in you favorite languages.

But, let's be honest. We often prefer write it from scratch than adopt existing one. The true reason that I feel mostly is adopting new one requires time and effort to learn. Just spit it, we are too board to learn.

At least, that was my case. I hope you are not, because again you are too expensive just to work on these repetitive works. Better to allocate your brain on scalability issue that has been around for sometime in your company.