Soulful Socio-Technical Architecture
- December 6, 2021
When I read the data about employee engagement worldwide in the State of the Global Workplace 2021 report by Gallup, I was shocked to find out that, on average, the engagement level of people is about 20%. That means only 2 people out of 10 are actively satisfied and thriving in their job. Only the USA and Canada surpass 30%. Most other countries are way behind, with Western Europe scoring an ashaming 11%. This is a general statistic, regarding any type of job, but it properly reflects the unhappiness I found in developers inside many organizations, especially the largest ones. I also observed how the most expert of them seem to be those suffering the most.
I wanted to understand more, so at Alcor Academy, we started our research as well.
We have been interviewing several IT industry leaders and managers asking what are, in their opinion, the most relevant issues they face in their environment.
We found out what they perceive to be their top problems:
- The development teams are not aligned with business value
- Teams are entangled with dependencies driving low productivity
- An unclear, top-down decision making jeopardize trust in the management team
- Attracting and retaining expert talents given the high turnover
- Most of the people remaining are disengaged and disconnected
I agree with Patrick Lencioni when he says that “a healthy organization has fewer politics and confusion, higher morale and productivity, lower unwanted turnover, and lower recruiting costs than an unhealthy one”.
From this point of view, both the Gallup stats and our findings are showing all the signs of an unhealthy industry. So, what can we do to change this disastrous trend? How can the industry heal?
I believe a collective mindset shift has to occur.
It’s the collective mindset of an organisation that determines an organisation’s overall effectiveness, productivity and degree of success. – Bob Marshall
That’s the reason why at Alcor Academy we focus on soulful socio-technical architectures that help promote healthy organizational cultures, where everybody wins: stakeholders, with higher productivity and reduced cost; customers, delighted by great products; and developers engaged in their work.
The term “socio-technical” was introduced after a study by Eric Trist & Ken Bamforth published in 1951. Their article analyzed the detrimental consequences and broken promises of the British nationalization of the coal mining industry that happened in the previous decades. The research was titled “Some Social and Psychological Consequences of the Long-Wall Method of Coal Getting: An Examination of the Psychological Situation and Defences of a Work Group in Relation to the Social Structure and Technological Content of the Work System”.
The “Long-wall method of coal getting” was a new way to extract coal, based on either the introduction of new technology (like underground railroads and mechanical underground diggers) and a social review of tasks and hierarchies involved in the process.
In the previous “Hand-got” method people worked in little, self-organized groups to mine coal (2-20 people). Their tasks were multiple, and the choice of teammates was critical for high productivity. Salary and bonuses were negotiated directly by each group, which had internal coordination and supervision on the final execution of the job. They were capable of responsible autonomy and able to vary the work pace in correspondence with changing conditions. This social structure proved to be ideal for adapting to the risky and uncertain underground situation.
The introduction of the new mechanized “Long-wall” also brought an entirely different social structure. The new method split the activities over three shifts instead of only one, and each shift did different things. People were spread out more spatially and temporally. Furthermore, they were segregated and paid by skill. So the social groups were defined by occupational roles, rather than cohesive personal relationships.
Even the three shifts were segregated, with no handover. Mistakes in one shift carried over to the next. Finally, also the role of management changed and became directly involved in the coordination and control of workers. As a result, workers felt they were driven by management, establishing a norm of low productivity.
The reform was meant to get the most out of the new technology, but the outcome was far from expectations. There were continuous problems with management. Absenteeism and turnover were at an all-time high, while strikes became the norm.
Faced with low productivity despite improved equipment, and with drifts from the pits despite both higher wages and better amenities, those in authority have increased their interest in organizational innovations. […] A qualitative change will have to be effected in the general character of the method so that a social, as well as a technological whole, can come into existence.
If attention is paid to the social as well as the technical system, then a choice can be made that leads to optimal performance of the system as a whole, even if that requires a less than an optimum state for the separate dimensions.
What Trist claims, is a direct consequence of the Theory of Constraints. In a complex environment, which is a system of systems, increasing the performances of only one internal system in isolation has not the effect of improving the whole. It often has the opposite effect, as shown by the dreadful mining reform.
In the Cambridge Dictionary, we find two interesting definitions for Architecture, if we drop those referring to the most common one related to the design of buildings.
- “The design and structure of a computer system, which controls what equipment can be connected to it and what software can operate on it.”
- “The structure of an organization, which influences the relationship between its parts and employees.”
If we think of Conway’s law, however, we can see how the two definitions are related to each other: the structure of the organization constraints the design of its digital systems. This clearly proves why the most common anti-pattern I’ve found in organizations is the “Monolithic, Big Ball of Mud”. Most of them are fixed, highly centralized hierarchies with poor communication structures entangled with entropy, after all. What would you expect?
Russell Ackoff wrote that “your understanding increases the larger the system you comprehend. Knowledge increases the smaller element you comprehend.” So, let’s journey from knowledge to understanding. Let’s start from the smallest element of our architecture and go inside-out this time (for a change).
The smallest element in our architecture is our code. At its best, it looks similar to the picture.
There are several variations and names over this idea, developed and expanded by several people. So the literature is vast and it doesn’t need more. (onion, hexagonal, port and adapters, clean…). The consensus here is that good architecture is modular and loosely coupled.
But our code neither runs by itself, nor it is developed for its own sake. Our code is always meant to support another activity and has to interact with other people and/or other activities. These interactions between activities and people, driven by the very basic flow of value, are not at all valued in hierarchical environments, because they cross departments orthogonally to the organizational chart.
Yet, “the key in making great and growable systems is much more to design how its modules communicate rather than what their internal properties and behaviors should be.” (Alan Kay)
That’s where, in my opinion, Eric Evan’s idea of Bounded Contexts and Context Mapping comes into play. The modular architecture has to live inside its bounded context of Communication. And this context represents a modular sub-system of the organization. Autonomous in how to perform its activity and how to implement things internally, but not in the way it interacts with other subsystems. That’s the critical aspect that needs to be well designed and defined, but in my experience is often overlooked.
To manage a system effectively, you might focus on the interactions of the parts, rather than their behavior taken separately.
So how do Bounded Contexts interact? Technically, the most relevant decision is between synchronous or asynchronous communication (this topic is widely covered in the literature as well). But before thinking about that, there must be an architecture of bounded contexts in place. How do we split them? In my experience that seems to be the point where organizations struggle the most.
The key here is to consider the organization’s value stream and analyze how the value is added to the end product or service. We can simply think of it as a temporal sequence of activities necessary to design, produce and deliver the value proposition. And, ideally, most of these activities would benefit from the support of an information system.
If this holistic view of the organization’s activities is missing, what chance is there of creating sub-systems that communicate in harmony with each other, smoothly mapping the flow of value with the flow of information?
If we look at software development from a value stream perspective, rather than simply looking at these activities as a process, we can emphasize the creation of value. As we have seen before, it’s all about improving the whole process, not just the parts.
The activities of the flow are independent but interconnected; this is one of the properties of a Complex System. Each activity performs different tasks in autonomy but feeds information back to other parts of the System. They are needed for the chain of value to progress to the next stage.
For example, in an e-commerce company selling physical items online, the value stream activities include:
- Acquiring (physically or remotely) the items to be sold
- Describing their physical details (color, qualities, pictures, and other information useful for their placement on the market)
- Assessing and interacting to handle the quantity in stock
- Creating and managing the mechanism for customers to browse the catalog, place an order and pay, dispatch the items
In parallel, there’s a marketing system to place advertisements, and after the sale is done, there will be some customer care functionality. This and all of the above are activities of the value stream.
They can happen in parallel, and the stream might be somewhat complicated, but they can be carried on independently of one another. However, they have dependencies; for example, the system in charge of letting customers place an order obviously needs to know the stock of the available items – they are interconnected.
To keep the overall entropy of the information system under control, the business must correctly separate and group these activities into their contexts. The crucial point is finding the perfect balance between independence and interconnections. (Incredibly similar to balancing coupling and cohesion in Object-Oriented programming, after all.)
But our contexts aren’t just for digital systems. In my view, the activity itself is part of the context, as well as the business people performing it. This is the point where the communication between technical people and business folks should be frequent, clear, and open. And that’s where the siloed departments divided by skill fail miserably. Exactly like in the coal mining reform!
To understand more about principles and strategies for evolving the teams in that direction, I suggest reading “Team Topologies” by Matthew Skelton and Manuel Pais. In their great work, they explain that there are many different ways teams can interact (collaboration, x-as-a-service, facilitation), with several different purposes (stream-aligned, enabling, complicated-subdomain, platform). Their size matters as well, because there is a limit in the cognitive load of individuals. Furthermore, the bigger the team, the weaker the relationships among its members.
These considerations are the most important when we do architecture design, in my opinion. Yet in most environments, there is not a defined strategy for how teams collaborate and what is their purpose in relation to the flow of value. This means that the design of these key aspects is often accidental.
Instead, imagine a network of small, autonomous, self-organizing, multi-skilled teams, united by purpose and values: how far could you go?
Unfortunately, the reality is often very different in too many environments. A rigid hierarchy of control is in charge to perpetuate itself, united only by command-and-control. A Ponzi scheme of power gets in the way of any innovative idea and systemic change.
When everyone feels just like a cog in the mechanism, it is not a surprise to see low levels of engagement. That’s what happens when people feel unheard, unsupported, and have no influence in their environment for creating a better future. And the higher the knowledge of the professionals, the deeper the feeling that their knowledge is being scorned. I sadly observed how the morale of most talented developers in these environments is often gloomy.
A healthy organizational culture is deeply important: when there is no joy, there is also very little creativity. If you leave the house and realize that your car has been stolen, will you be in the mood of creating something amazing?
Joy, feeling one’s own value, being appreciated and loved by others, feeling useful and capable of production are all factors of enormous value for the human soul. – Maria Montessori
That’s why I believe good architecture also has to consider the soul of the individuals: we need Soulful Socio-technical Architectures!
Soulful Socio-Technical Architecture
Let’s go back to the miners for a moment. Did they find a way to improve the situation following the suggestions of Trist and Bamforth? Some experiments have been done with a new method called “Composite long-wall”. Composite because it consisted of the merging of the new technological innovation of the “long-wall” method with the social structure of the old “hand-got” method.
Segregation between shifts was removed and a proper handover was adopted. Autonomous, multi-skilled workgroups were re-introduced, together with a group bonus payment system. Supervision was returned internally to the groups by leaders elected by the workers themselves.
The outcome of the experiment proved that Trist was correct. Absenteeism was cut in half, incidents and sickness decreased sharply, the turnover stopped, the overall costs were reduced. While productivity had a 25% boost.
The research team found what conventional wisdom had held to be impossible: the working of the conventional, semi-mechanized, three-shift cycle by a set of autonomous workgroups.
They had to push the controlling management out of the pits for the miners to regain the engagement again! So what can we learn from this? I believe the main problem of the IT industry is very similar to the miners’ one with the introduction of the “long-wall”. We are witnessing a lack of healthy organizational cultures. Hierarchies and authority are slowly killing the soul of teams in many organizations.
I agree with John Seddon when he claims that “the problem with command–and–control is ‘control’ rather than ‘command’.”
If you really trust somebody, would you control them? But “trust lies at the heart of a functioning, cohesive team. Without it, teamwork is almost impossible.” Without trust, the soul of a team is doomed!
That’s why to let the soul of your teams bloom is essential to move away from rigid command-and-control and bureaucracy, giving decision power, budget, and autonomy to teams.
Accelerate book shows: “Approval by a manager or CAB simply doesn’t work to increase the stability of production systems. However, it certainly slows things down. It is, in fact, worse than having no change approval process at all.” It’s way worse because this process has the psychological effect of scorning the competence of developers while adding entropy in the system for no pragmatic reason. Of course, they end up feeling like a cog!
The effectiveness of authority decreases as the educational level of subordinates increases. When the educational level of subordinates is higher than that of their bosses it becomes negative. – Russel Ackoff
Another thing we can learn from the story is that individual pay, completely detached from the performance of the organization, is not ideal. Keeping payment and performance reviews at an individual level foster internal competition instead of collaboration and promote the development of individual agendas. Exactly the opposite of shared purpose and values.
If “90-95% of the worker’s performance is governed by the system and only 10-5% by the individual” as Edwards Deming claimed, what sense does it make to have 100% individual pay and performance reviews?
Finally, to create a magic environment where the soul of our teams can thrive, we need to create the conditions for strong relationships to develop and flourish.
People are not our greatest asset. It’s the relationships between people that are our greatest asset. Bob Marshall
In my experience, one of the most effective ways to foster authentic relationships rapidly is learning together. The possible activities to promote are many: learning time and budget, a community of practice, extensive top-quality training, conference allowance, innovation time, meetup organization and speaking, internal company events (open space, hackathons, code retreats..)
If ignoring the opinion of knowledgeable people has the effect of scorning their knowledge, these activities have exactly the opposite effect! They prove with facts that sharing and expanding collective knowledge is taken very seriously and considered a priority!
The social consequences of such an environment are amazing: people begin to enjoy the process of sharing immaterial things, like knowledge, understanding, and awareness. The environment is overwhelmed with authentic gratitude and generosity for each other, while the levels of mastery and creativity keep growing by the day. That’s when the soul of the team thrives!
Socio-technical systems design provides a new worldview of what constitutes quality of working life and humanism at work. It facilitates organizational innovation by recommending the removal of many elite groups and substituting flatter hierarchies, multi-skilling, and group decision-making. It wants to replace tight controls, bureaucracy, and stress with an organization and technology that enhances human freedom, democracy, and creativity. – Enid Mumford
And when one socio-technical system has a thriving soul, magic will happen…
Have you ever seen some pictures of the Wieliczka salt mine near Krakow in Poland?
The mine was included on the World Heritage List in 1978 because it “provides outstanding testimony about the socio-technical system involved in the underground mining of rock salt.”
A large number of sculptures and other decorative features carved out of rock salt by the miners in the ages survive inside the excavations. The subterranean chapels are particularly notable for their artistic value, their ornate decorations, and fittings. Among the artifacts there is even a reproduction of Leonardo Da Vinci’s “last supper”!
What inspired these miners to undertake such amazing and creative projects? Do you think it was the management to order them to do it?
It was the social fabric that bound the miners together – mostly expressions of faith and unity that helped them endure the harsh environment. One could imagine how the impetus to create these works of art snowballed – some miners took the initiative to overcome the dreariness of hard labor. As the mine was successful and the miners dug deeper, they experimented and tried new things. With each new chamber came an opportunity to try something new. And the cycle continued for centuries.
Trist & Bamforth described how the mechanization of the longwall method changed (some may say destroyed) the social fabric of the organization. They included examples of how the workers bonded inside and outside the mine, such that their families would support each other in times of need. The reform, with the excuse of the new technology, broke those ties and changed the relationships, leading to the loss of morale.
Perhaps Wieliczka offers another perspective on social fabric: freedom for members to express themselves and transcend the drudgery of the environment.
With the global knowledge of firms made possible by information technology, it is easier now than ever to judge organizations based on many tangible factors. But what about an organization’s soul?
Should it matter to the consumers how firms, businesses, or governments treat their own members? How closely connected are its members? If one were to visit Wieliczka when it was still open and ventured deep down into the chapels, what would one think?
“My, how beautiful”? Or “My, how many excess workers are down here with all this time on their hands?”
I would propose this – the soul of the organization is the degree to which it provides the will and abilities to promote the good and the wisdom in all its members and relationships.
Information, knowledge and understanding enable us to do things right, to be efficient, but wisdom enables us to do the right things, to be effective. Science pursues data, information, knowledge and understanding: what is truth; but the humanities pursue wisdom: what is right. – Russell Ackoff
Frenz Herzberg once wrote: “if you want someone to do a good job, give them a good job to do.” I am not sure if the miners would agree. I rephrase it:
If you want someone to do a good job, build a Soulful Socio-Technical Architecture.