Carl had been employed as a quality assurance developer at Codecs-R-Us for the last several years. Leaving his previous job was a decision that he hadn’t take lightly as it had required a pay-cut and a relocation. However, joining Codecs was his way of reorienting a skillset that would have otherwise been squandered programming the benign business logic of some insignificant ad widget or progressively optimizing the performance of some destructively addictive social media app.
Working at Codecs wasn’t considered glamorous by outsiders but, to those that understood its significance, it was meaningful work. Carl was now involved with developing software that influenced fundamental aspects of human perception and action. He appreciated that the company culture was much less rooted in the “move fast and break things” way of thinking and more in the way of the “measure twice, cut once” approach.
Carl happened to be part of the Techno-Wisdom department.
In the innermost social area of the office stood a table-like surface featuring little more than an inviting, elegantly designed button. Upon actuation, a precision-machined titanium slot would open and release a single ticket. As he approached the dispenser, Carl reflected on his job transition and the reasons behind it. He could see how the care and thoughtfulness put into the experience of dispensing a ticket reflected his reasons for choosing this company. As he gently pressed the button, he consciously savored the mechanical, tactile feedback it produced. When his next assignment floated up through the ticket slot, his appreciation for the design of this analog experience was reaffirmed. The tickets themselves were made of an ever-so-slightly cream colored, heavier weight card stock with chamfered edges. The necessary information was printed on the ticket using a handsome font and well structured composition – an attention to detail attributed to their CEO.
The CEO believed that if society was going to promote working in the office, then we owed it to ourselves to design our physical spaces and experiences with as much care and detail as our digital ones. The company successfully accomplished with how it dispenses development tickets without sacrificing the merits of a digital solution. Each ticket was imbued with a specially coded chip that could communicate only with the assigned employee’s workstation and the embedded facial recognition systems in the dispenser matched his receipt of the ticket to the assignment on the systems back end. This served as a seamless two-factor authentication for delivery and execution of a development assignment – perfectly blending a natural, analog experience with digital efficiency. The ticket served as a physical embodiment of the assignment – its design and delivery are part of a curated experience that signaled the importance of every single assignment, regardless of its apparent size and complexity or who it was assigned to.
Carl briefly glanced down at the ticket as he walked towards the beverage station near the rear of the social area. It read:
Department: Techno-Wisdom Codec
Issue: Decompression error; Reported and confirmed by multiple users in the electric utility sector; Guidance from tech-sector causing arbitrary centralization of data and data systems leading to systemic malappropriation of resources and lost opportunity.
Suggested Direction: Consult with documented user stories and proceed with caution when adding to the codec.
Additional References: See previously produced report on appearance statistics of this bug.
Carl knew right away that his plan to stop by for a warm, caffeinated beverage before returning to his workstation was no longer an afternoon treat but rather a necessity. This next assignment was going to challenge him and a jumpstart would be crucial to making meaningful progress with the remaining hours of the day.
Carl had a favorite beverage for such occasions: black tea with milk, like the British drink it. Carl wasn’t British. And though he adopted this habit from his wife, neither was she. Much like the ticket dispenser, the beverage station was crafted and stocked meticulously. It wasn’t merely tailored for the typical coffee drinker but also accommodated the preferences of people like Carl and others. The station boasted an array of tea leaves, not just tea bags, along with various high-quality milk options. Additionally, it featured fresh ginger root and spices for those who wished to transform their drink. Given the task at hand, Carl believed the extra effort would be well worth it.
While waiting for his specially brewed, aromatic concoction, Carl discussed his apprehensions about the latest ticket with a colleague. This colleague worked in the Social Traditions department and, as a result, had really matured as a developer over the last decade. The colleague not only confirmed this was a serious issue but also shared that he had heard rumors of similar issues reported from other departments as well. Coming from such an experienced developer, this revelation implied that Carl had his work cut out for him.
Carl walked briskly yet steadily back to his work station, ensuring safe transport of the hot beverage. As he sat down, he retrieved the ticket from his back pocket and rested it on his desk where it was promptly recognized by his integrated workstation. His screen awakened, presenting him with a digital copy of the ticket, an acknowledgement of its addition to his backlog, and also the relevant documents referenced in the ticket.
His terminal noted that he was reading from one of the user issue reports and prompted him with a modal option to print. Up again, Carl made his way to the printer to fetch the requested documents, understanding that he could absorb the material more deeply in this fashion.
On the top of the printed stack was a user issue report from a senior engineering manager in a large electric transmission utility. Carl began to read…
I’ve been working on what I’ve been referring to as a “data integration strategy” for my 9-5 for the past year. I know, it may sound like a frustratingly meaningless phrase but I do believe this is the right term and level of analysis. One of several major drivers for designing this strategy is to counter a foolish, top-down corporate initiative that commits a modern data sin—the arbitrary centralization of data and data systems. Truth be told, a similar danger exists everywhere and is committed not only by amateurs but also by technology professionals. The trouble stems from the suggestion that all data and business problems can be solved by creating “one place for all your data,” making for an intoxicating sales pitch. It seems easily comprehensible on the surface, giving everyone “good vibes“.
My encounter with this initiative is not my first experience with such misguided concepts. This issue arose when dealing with multiple modalities of physics-based time-series sensor data from the grid. Data created from sampling rates above the 1-5Hz range, as it turns out, requires a complete rearchitecting of the entire technology stack for managing the data. While that issue has been addressed (a story for another day), the nature of our work has necessitated an inclusion of more data modalities than just time-series into our world. As a result, I’m now compelled to articulate and promote a superior strategy but am concerned about how to address the cultural headwinds.
Carl began to palpably sense the issue but desired more context. He flipped through the documents to locate the referenced report and continued reading. Carl’s confidence grew from his review of the document. He continued sipping on his tea and reflected on the situation, identifying four primary issues:
- The use of techno-wisdom to sell a utopic vision into an organization.
- The concerns surrounding the downstream incentives of those that would be tasked with executing such a vision.
- The ease with which the techno-wisdom can be used to express very real shortcomings of existing data and data systems.
- The unaddressed impracticalities of migrating from a legacy state to the acclaimed ideal state of a fully centralized data system.
As such, Carl acknowledged that to address this bug properly, the initial modification he needed to make to the Techno-Wisdom Codec was to explain each of these four issues. The codec was going to require additional information for its decompression function to transfer wisdom into correct human action.
The Appeal of the Utopic Vision
Carl turned inward towards his terminal and began to add the additional data needed by the codec.
A commonly expressed techno-wisdom advocates creating “one place for all your data”, promoting easy accessibility, superior quality, thorough documentation, and employee training thereby enabling a self-service paradigm where all employees can effortlessly use company data to inform every business decision.
Two elements make this wisdom seductively appealing to both executives and engineers alike. However, it’s precisely this seductive allure that should trigger skepticism. The root logic behind this skepticism aligns with the classic “its too good to be true” wisdom.
The first element that contributes to its appeal and low-friction buy-in is the simplicity in which it’s expressed and imagined. Consider the alternative – the cognitive burden of envisioning the breadth and diversity of all of the company data and data systems is so high that it is virtually impossible for a single individual to comprehend fully, let alone know how to use it. However, when presented with an alternative vision where all data is centrally collected and collated, the mind relishes the ease at which it can seemingly comprehend it, intoxicating the customer with the utopic promise of a perfect order.
The second appealing element is the notion of a self-service paradigm – a magical promise for both individuals and organizations. For the executive, it envisions a workforce suddenly capable of multiplicative productivity as a result of seemingly bringing order to data and addressing an apparent skill gap. However, it neglects the real, underlying issue of limited human capital and the challenge of creating meaningful data products atop existing, full-time commitments. For the individual, it portrays a world where, with minimal effort, they gain access to an infinite realm of magical superpowers that aid them in their jobs and ideally make their work effortless – not to mention the presumption of career advancement for opting in to this world view.
Taking the notion of “one place for all our data” literally (e.g. single database, data warehouse, data lake, etc.) leads down a dangerous path, tyrannically imposing one bureaucracy and one technology on the organization’s diversity and complexity. Such a utopic vision is neither meaningfully achievable nor (to the degree it is achieved) guaranteed to be useful.
Carl submitted the contribution for commit and validation against the main branch. It returned affirmatively but with the following prompt:
>> Codec decompression compiled satisfactorily but has raised the following issue: While we should be generally skeptical of this “too-good-to-be-true” strategy, there must be a reason for why it exists. Assuming this is true, what should we consider as the scenarios where this is validly applied?
Carl hesitated before responding but soon realized that the agent was correct. The negative outcomes of applying wisdom of any kind often stem from literal interpretation and broad application. However, there may be scenarios where it is indeed the correct approach. Carl returned to his terminal to key in his guidance.
- When there is a high degree of uniformity among a collection of data, it is logical to allow and even encourage centralization. Under this scenario, there are limited unintended consequences because the presumptive spectrum of use cases would be the same for the entirety of the data. Put simply, if the data is intrinsically homogenous enough, we can integrate it without much a priori knowledge of its future use.
- When data of any modality or size is required for a single calculation or analytic, centralization becomes necessary. It must be aggregated into the analytic workspace and ultimately onto the same (often virtualized) machine, though perhaps not simultaneously. This is analogous to, though not equivalent to, materializing data from a database before a calculation. Under this scenario, the imperative is clear: data should be consolidated.However, this conclusions rests on the assumption of a well-defined objective. <BREADCRUMB> Return to this node to address balancing this idea between experimentation with well-defined data products </BREADCRUMB>
- Building on the previous two points, a generalization emerges regarding the appropriateness of centralizing data.Centralization is deemed acceptable when the precise desired usage of the data is known a priori. As the uncertainty of the future usage of the data increases, other dimensions should drive architectural decisions, policies, and optimizations.
- In summary, bringing maximal order to data and data systems proves efficient when everything about the data and its future use is known. However, this transforms into tyranny when the underlying complexities are left unaddressed. <AGENT_REF> Reference Ashby’s Law of Requisite Variety and Conway’s Law for complementary, reinforcing concepts </AGENT_REF> This tyranny manifests as imposed uniformity and constraints that hinder adaptability and responsiveness to evolving data needs, stifle innovation, and limit the overall ability to handle unforeseen challenges. Striking a balance between conformity and flexibility is essential for navigating the intricate landscape of data management and ensuring the resilience of an organization’s information architecture.
Carl submitted this contribution to the codec agent and was provided with a positive but sparse response:
>> Accepted.
The Downstream Incentives of the Workers
Carl was then on to the next issue.
The issues with the techno-wisdom of “one place for all your data” extend beyond its seductive simplicity and involve those that stand to benefit from its successful seduction of engineers and executives alike.
Numerous organizations and individuals stand to gain from the literal use or overuse/abuse of this techno-wisdom.These can be explained through three mechanisms. First, the forced centralization of data and data systems can be employed as a tool to cultivate and/or preserve power, authority, and control. Alongside the direct benefits, there are secondary advantages including ensuring job security.
This overlaps with the second mechanism: crudely put, arbitrary centralization of data and data systems is the modern equivalent of “shifting numbers from column A to column B.” Effort is expended, but no substantive work is accomplished, resulting in a lack of real value creation. However, that does not mean that nobody benefits from this effort – it provides individuals an opportunity to bill many hours at their hourly rates. They can do this with confidence because they know that even if the “work” were to ever dry up, it would happen on such a time scale that, by then, there would be another metaphorical column that arises to shift the data to next.
Lastly, laborers in the tech-market are highly brand-aware or brand-sensitive as it pertains to the technologies that adorn their resume or CV. Supporting a lift-and-shift operation to the current technology or data architecture du-jour is personally advantageous because it establishes relevance in the marketplace and ensures maximum job mobility. However, these incentives are often in opposition to the needs of the organization and its true data objectives.
In electric utility companies, service organizations like IT operate as cost-centers, disincentivizing excellence and incentivizing a focus on survival. Consolidating power around a central data system ensures business dependency on the service organization. Creating billable hours for low-complexity, low-risk work guarantees continuity of the service organization. And adorning employees’ resumes with the latest, sexiest technologies delivers personal security even if their jobs in the organization were to be sacrificed.
Once again, Carl submitted his latest contribution to the codec agent for processing and consideration. The compilation appeared to be taking longer than usual and Carl noticed his tea had begin to cool. He decided to stretch his legs and warm his tea while he waited. Upon his return to the workstation, his screen acknowledged his presence and awakened. He found his terminal reading:
>> Codec decompression compiled with the following warnings: Simulator generated a collective inner monologue from a simulated, archetypal utility IT organization with the following sentiment: “All Your Database Are Belong to Us”.
Yes, the agent was hallucinating from that last contribution but Carl acknowledged that it had a point. He therefore reasserted his previous contribution with the following addendum:
There is a scenario where the business might default to a literal interpretation of techno-wisdom and relinquish control and authority to the IT organization. If the business lacks the means to act independently with its data, then the “something is better than nothing” approach is valid.
Moreover, it’s typically unwise to engage in a confrontation with a bureaucracy larger than your own. Perhaps a more prudent strategy for a business capable of acting independently is to operate locally in its best interest without directly opposing potentially competing initiatives that adhere to sub-optimal philosophies. In essence, focus on solving your own problems and refrain from entering into unnecessary conflicts.
The compile time for this contribution was even slower than the previous one. Carl could sense that he was truly pushing for a rewiring of the codec. It was as if the bug had infiltrated and feasted on the decompression mechanism like an army of termites attacking the foundation of a home. Carl’s contributions were lifting the home with hydraulic jacks so the foundation could be slid out and replaced. For him, this served as evidence that he was on the right track. His tea was now gone and any more caffeine would be counterproductive. It was time for something fizzy!
To be continued…
Sign Up
Subscribe to receive email notifications whenever a new blog post is published.
You can unsubscribe at any time.
[…] the decompression bug in the Techno-Wisdom Codec. He marked through the first two topics he had previously addressed leaving two more vying for his […]