Principles In Practice – It Will Go Wrong

work safe

It’s Murphy’s Law. If it can go wrong, it will. What’s your application’s opinion on errors? If we take a stance on errors, these principles will tell us exactly how the application will behave. Shared understanding means less miscommunication. Here are some of the things I’m talking about.

Operators have the accurate information – your data is already wrong

If you’ve ever heard the following conversation, you know data entry will always be out of sync with the real world. “I know the database is telling me that bus 7 has transmission 8 installed, but I’m looking at transmission 3. Don’t tell me I can’t uninstall transmission 3, because I just did.”

What does that mean for implementation? Favor “soft validations,” warning that errors are present, but allow overrides. If I have a misallocated part in the database, I need to uninstall it from where the database “thinks” it is and mark that uninstall as something needing follow up by people in the field.

It’s all fun and games until someone dies. – The data must have integrity

Particularly in the medical field, the configuration of a medical device should be exactly what you intend with no mistakes or corruptions. Corrupt data can kill. Also, data must meet certain validations to be used, but can fail during editing.

Never write over the data file. Always write to a shadow file, test the integrity and in one atomic action, move the new data file over the old one. If there is ANY failure of integrity, the old data file is a safe fallback position.

In addition, the internal data structures might be “persistent” allowing undoing to a known good state if any operation fails an integrity check. Some transient operations might lead to bad values, like changing a medical dosing schedule might temporarily exceed safe dosages within a time frame, but you should not exit the schedule edit until the schedule validates.

If it crashes, I can use the browser, right? – The application must stay running

This is useful for kiosk or embedded applications. However some long running applications also need to handle failure.

Use multiple executables. The less an application does, the less likely it is to crash. In addition, a supervisor application can restart parts of the application that fail. Operations with risky components like using third party drivers or that can enter bad states because of bad data can run in their own sandbox and die if they ever enter a corrupt state.

There is real money involved

Real money requires an audit log. In addition, duplicate submissions are a killer and should be identified ASAP to avoid duplicate charges.

Operations, like purchases and fulfillments, are seen and recorded as first class data. The engineering team may lean on CQRS for these operations. Operations may be given an identifier early in the process. For example your cart my always have a “next order Id” so commands may be made idempotent preventing duplicate submissions.

Take a bit and think about how your application can fail and what can go wrong. Pick your battles and identify the ones that are the most likely and the ones that cause the most havoc. How are you going to handle them?

No Comments

Principles of Architecture

work safe

There is no correct software architecture. But there are an unlimited number of ways to incorrectly architect your software. The best we can do is state some principles and use those to help narrow the solution space of all possible architectures. This will help us identify a more correct architecture.

Architecture – a quick definition

Building Architecture is a combination of two things, making the building suitable for purpose and making the overall structure pleasing to the eye.

Making a building suitable for purpose is mostly dividing the space in to boundaries. Like rooms are often grouped near one another. Some structural considerations may help this grouping, such as bathrooms being above the kitchen to limit the number of wet walls. Some rooms may have internal subdivisions like nooks and archways that don’t limit the flow or utility of a space but visually define these regions. This defines the utility of a building.

Aesthetic considerations help make the building feel as it belongs. This means it fits in the surrounding landscape as well as feels balanced and consistent with itself. The balance and feel contribute to the habitability of a space.

Utility and Habitability in Software

Software architecture also controls the utility and habitability of a program.

Software Architectural utility is not the UX, but the software’s ability to perform its tasks quickly enough, reliably enough, and a host of other so-called –ilities. This is done by determining if you need extra caching for speed, separation of the active data set from the reporting data set, splitting the application in to multiple programs to better handle failure, and interfacing with other systems. And that’s just a start!

Software Architectural habitability is about the ease of development. Simple changes should be simple. Changes that depend on multiple business concepts interacting should only be as complex as those interactions. This is done by being deliberate about what code interacts with other code, by drawing boundaries between software concepts that reinforce boundaries between business concepts.

These forces can be in tension. Adding robustness to an application can make changes harder. Making it more flexible can degrade performance. Focusing solely on making easily changeable software if fine for small scale software. Wringing every last ounce of throughput is perfect for less complex software. Finding the balance is the art of Software Architecture.

Principles for Improving Communication

If you need someone called “Architect” on your project, the odds are you have a team larger than the two-pizza rule. That means that several decisions a day are going to need the architect to be there, and some times she can’t be present in person. Be stating not only the design goals but what purpose they serve, the architect empowers the team to make correct choices when they find themselves on their own.

It isn’t sufficient to say the software must be robust. But if we say minimize the impact of failure on non-related operations and enable a manual retry of failed transactions we are clearly stating what we mean by robust. When dealing with a design decision I can now ask myself a few questions to guide my implementation.

  1. Does failure impact other users?
  2. Do I have enough information to recreate this request if it fails to complete?
  3. When I fail, can I undo the partial operation?
  4. Are my operations Idempotent?

By stating the outcomes desired instead of the means we use to get there, the architect prevents the same problem being addressed by nothing more than a bunch of null checks and try/catch blocks.

It’s Only Business

Theses principles should come from the business case for the software. How many users are you supporting at once? Are they _occasionally connected_ or always connected? What is the impact of failure on a sale? The impact of failure on trust in the system? How about the impact of failure in the lives of thousands of children? Will someone die if your software falls down or corrupts data?

What are the reasons for your architecture choices? Have your design choices helped or hindered those goals? Is there tension between the design impacts of your business goals? Do you have some design changes to go deal with?

No Comments

It’s not been abstracted away yet, has it?

work safe

There is a bug in my code that I finally squashed last week. I had been visiting it and kicking it down the road for a couple of weeks as the feature is still being written. When I move to a specific view in our application, I needed a TextBox to gain focus. When navigating away from this view, some button or other will grab focus so returning will not show the TextBox as having focus. The view knows when it is being activated, so why can’t I just tell it to focus on the TextBox then?

A Timer? Really?

Elsewhere in the inherited code base, I saw them do essentially that. When the view is activated, tell the TextBox to focus. Only they used a timer with a very short duration. This is obviously a code smell, so I did it very differently. Here’s some sanitized WPF code to describe what I saw and what I did.

Their Code

A DispatchTimer runs on the UI thread. So I made an assumption the only reason this was being done was to make sure UI code was happening on the UI thread.

public void FocusTextBox(object sender, EventArgs e){
  var time = new DispatcherTimer{ Interval = TimeSpan.FromSeconds(0.01) };
  timer.Tick += delegate {
    timer.Stop();
    if(FocusedTextBox != null){
      FocusedTextBox.Focus();
      FocusedTextBox.SelectionStart = FocusedTextBox.Text.Length;
    }
  };
  timer.Start;
}

This works. It makes my skin crawl because it uses a timer, seemingly only for thread hopping.

My Initial Version

I just told the Dispatcher to run some code for me. The simplest way to get code running on the main thread.

public void FocusTextBox(){
  Action makeFocused = delegate {
    if(FocusedTextBox != null){
      FocusedTextBox.Focus();
      FocusedTextBox.SelectionStart = FocusedTextBox.Text.Length;
    }
  };
  Dispatcher.Invoke( DispatcherPriority.ApplicationIdle, makeFocused );
}

But, as the documents say,

Events and Sequence Diagrams

GUI applications run via an Event Loop. We all but ignore this by hiding it behind APIs like Application.Run that handle the loop for us. Our base classes wire in to the event loop allowing buttons to register their touches. All this abstraction gives us an illusion of concurrency. But it isn’t really concurrent.

If you debug a .NET application, you will notice that a line of code that triggers an event will immediately stop the execution of the current method and start executing the event handler.

I was using the Dispatcher to ensure I was working on the UI Thread, but what if I started on the UI Thread?

I’m not 100% positive, but here’s the theory for what’s happening:


* A touch down event changes tab

* This causes the tab selection index to change triggering an event handler

* The event handler sets the focus on the text box

* and returns control to the tab control

* The tab control finishes

* and returns control to the event loop

* which processes a touch up event changing focus

joy.

My Final Version

The fix was simple. I need to add an event to the queue without interrupting the current pass through the even queue. Fortunately, the Dispatcher allows us to enqueue a bunch of operations and even manipulate them in flight so to speak via a DispatcherOperation. Let’s try calling BeginInvoke on the dispatcher instead!

public void FocusTextBox(){
  Action makeFocused = delegate {
    if(FocusedTextBox != null){
      FocusedTextBox.Focus();
      FocusedTextBox.SelectionStart = FocusedTextBox.Text.Length;
    }
  };
  Dispatcher.BeginInvoke( DispatcherPriority.ApplicationIdle, makeFocused );
}

And it works. The touches are fully processed before I try to set focus. And I don’t use a timer for flow control.

Takeaway

The main takeaway is the event loop still exists, even if our abstractions try to hide it. Event based programming is amazing. But we must remember that it only creates an illusion of concurrent programming. All of those actions are still happening sequentially. It does pay to really understand the arcane secrets programming frameworks use to enable their abstractions.

No Comments

Music as a Metaphor

work safe

Music is a highly structured art form… usually. This structure makes use of repetition and variation of form. Music can be monophonic or polyphonic, one voice or many voices interacting with each other. That’s what makes it interesting as a way of exploring code structure.

Symphonies

A Symphony is a collection of songs, called movements, connected in some manner. They share a collection of themes and other musical ideas. But each song operates in its own reality.   Each song has its own musical key, tempo, and time signature giving each movement a very different feel. Each song still serves to bolster the Symphony as a whole.

Some applications have such different responsibilities we express each responsibility as a different executable program. A time sheet software may have a time entry client, a time entry server, time code management, a reporting system, and a payroll system. The themes that string everything together are the Ubiquitous Language and shared data interfaces. Each program has a shared understanding, but can act on that understanding in the manner that best fits its unique responsibilities to the application as a whole. In addition, some programs/services may serve to alter the data from one format to another, such as waiting until generating invoices to transform from a format used by most of the application to the format needed by our third party software.

Verse-Chorus Form

Your basic pop song has an internal structure called Verse-Chorus form.  The verses have similarities to their internal structure but vary differently in the content. The variance is usually the words of the verse. The chorus is the same, used to break up the verses.

We can see Verse-Chorus form in an app with several data screens, like the time entry client.  Each screen is a Verse stitched together with both a data layer and navigation framework, the Chorus.

On occasion, a song has a radical change called the Bridge. This can alter the musical key as well as rhythmic underpinnings and has little similarity to the rest of the Verse-Chorus form but serves to add interest before returning to the original structure. As our time client grows, we find reports and off-line syncing that are important to our application, but are structurally very different from itself.

Canon

A Canon is a way of manipulating a melody. The simplest example would be Row Row Row Your Boat, where the melody is time shifted by one third. When this happens twice, for a total of three instances, we get a special class of canon called a Round and can repeat the layered melody Ad infinitum.

In our time application, we can think of the canon as the series of transformations we need to apply to our employees’ time data this billing period.  We collect, group, transform, and collect again.  In C# this can be a rather long list of Linq operations.  In SQL, a very complex query.  But a lot of the data accumulation and transformation can be carefully done in parallel.  Much like a canon.

Why Care?

This is an incomplete metaphor. I wouldn’t use it as a means of laying out your code base. Instead, I can see it used to seed discussion about similarities and structure. By understanding that parts of your application have similarities and differences, the hope is we can group our code in a way that helps separate responsibilities, give clarity, and let developers quickly find the code they need.

No Comments

Programming is Math?

work safe

This sculpture is known by some students as the Cecil T. Lobo Whatzit. This sculpture was installed on the Rose-Hulman campus by the Civil Engineering Department as an example of 17 different the steel joints. It is an amazing technical challenge to find harmony and balance with in the constraints of 17 steel girder joints, all “Golden-Gate Red”.

Quite frankly it looks like junk to my untrained eye.

I had a conversation one morning where we were talking about art for the sake of art. We talked about collaborations and other art projects that are interesting to those steeped in the art form, but not to the uninitiated.

One phrase I keep hearing stated as a universal truth is “Programming is Math.” I’ve tried to ask people what they mean by this, but I have yet to find someone who doesn’t think I’m trolling them. The problem is EVERYTHING is Math. Economics is Math. Engineering is Math. Architecture is Math. Even Painting is Math.

But everything is also Poetry. Ask a Poet.

I was trained as an Engineer. Capital E. I took classes on fluid dynamics, forces on static structures, and circuit design. I see the world as a list of Knowns specific to a problem, a bunch of Given universal truths and equations, and a solution I’m attempting to Find. Elegance never entered in to that much. But then I began seeing how solutions work for real people, how they impact their lives. Defining the problem became much more interesting. Math was only something I used to get there. In programming that was usually Algebra, Boolean Logic, and Trigonometry.

Okay, lets get back to software. There are a bunch of techniques we use when architecting software. There are books and books of patterns on the subject. We try to find a pattern to match the entire solution, but we will fail. That’s not to say we shouldn’t understand these techniques and explore them, just that we shouldn’t confuse a study with art just because it is well done.

The problem I have with this is first and foremost, it is seen as an obvious truth that needs to be stated, but once stated is self evident. When making such a statement, you should be offering help on the road to enlightenment, not merely shouting from the mountain top that you have achieved satori. It annoys me that I’ve yet to find a way in to this secret understanding of Math having always been horrible at wrote computation we teach kids in American Math classes. But it also feels counter to what I have come to understand about software in my 13 years as a professional practitioner.

“The Second Law of Consulting: No matter how it looks at first, it’s always a people problem.” – Gerald Weinberg

The things that make my life difficult aren’t a lack of technical choices, but understanding the implications of how those choices fit together. Especially if I wasn’t there to participate in the evolution of thought and circumstance.

The thinking of Programming as Math feels like we are suborning the technique to the expression. The art is getting lost in the technical aspects of the art form. I want to say communication and clarity to my fellow developers and collaborators is far more important than technical details of my solution. There are techniques that help me do that, but only if they serve the communication.

And I still can’t see where programming like this is Math as I understand it.

No Comments

Architecture Vs. Structural Engineering

work safe

We like to use a metaphor comparing software design to building architecture. Some people accept that this is flawed. Others cling to this metaphor too tightly, seeing developers as little more than tradesmen installing carpet in a new skyscraper.

A Building Architecture project includes a Structural Engineer. Her job is to ensure the building stands up. Her job is to determine what materials are used in the wall interiors. Her job is to make sure the windows will withstand the forces of straight line prairie winds against the 30th floor.

She is active in the design. The architect defines taste, purpose, form, function, and values. The Architect’s role is to define the spaces where she works. This, in my mind, is a very different role in software. I posit that most conflict between Architects and development teams stem from misunderstanding the role of the Structural Engineer.

Lets look at a large system. I have a web application that is used to track employee time spent on projects and turn that in to purchase orders. We have concerns about using this tool on client computers exposing sensitive data. We need to manage project membership. We would also like to use that to build employee and company “résumés” used in explaining our expertise to our clients. I’m not picking this real example because I think our current incarnation of this tool is wrong, but because this tool is an example of a large system that has many responsibilities.

What should I as the Architect spend my time doing? Should I start mandating the use of certain threading patterns? Building the ERD for the highly normalized database? Or start determining the bounded contexts, core contexts, use cases, external constraints, target response times, and security requirements?

I would get my Tech Leads in to a room and discuss how each context needs to talk with the other contexts. I would focus on how the contexts are realized and coordinate with each other. I would let my Tech Leads figure out the internals.

How do I make sure I understand the design tensions between the contexts? I would need to work on several teams, pairing with developers who work deep in the code to find where our abstractions are breaking down. An building architect spends time on site during construction for just this reason.

But I would make sure the teams are still talking, and still hitting those request times. They are my system engineers. While I may or may not know better than them as a developer, my job is to teach them to make the right choices, not make decisions for them.

No Comments

Why I Blog

work safe

I told a friend of a plan. It was one of those crazy master plans that will probably change to world. He seemed nonplused. I said, “How about a writers workshop for blogging?” He said, “I don’t like writing. So why should I come?”

I won’t convince a veteran of the “Blog Battles” of ’aught 11 and ’aught 12 like him that writing is fun if he has already decided it isn’t. But maybe I can convince him, and maybe you, that blogging is important. And I’ll try by sharing with everyone why I blog.

To think things through

When approaching an idea that I may not quite completely understand, I don’t always consider it from all angles without some sort of framework. Blogging is one of the frameworks I use. By trying to think through an explanatory narrative, I come to understand what was a vague notion before.

To find later

I have come across the occasional hard fraught nugget of knowledge. Unfortunately, I have mind that is quirky about what actually sticks. Fortunately, I have referenced my blog myself.

As a shorthand for explaining to others

Nothing makes me happier when having a chat with my colleagues and being able to say, “Have you tried #{thing}?” Bonus points if I can point them to a blog post where I actually have something constructive to say.

To participate with the community

I look forward to the SEP blog battles. Not only is it fun to think about how I’m going to approach the topic, but I love reading the posts my coworkers write. It is wonderful to see a new perspective from the thoughts I’ve been having.

Because I have time

I used to spend evenings for a month at a time prepping presentations. I wrote code while watching TV with my Wife. I’m now a Dad. I would rather spend time with my Wife and Daughter than a laptop. But I still read. And since I have my phone at almost all times, I can write when my baby Daughter sleeps on my shoulder. Like now.

To improve my communications

The most important thing I do all day is communicate. Usually, it is written communication. I write for the compiler. I write to my friends in chat. I write to my team and company in email. I write code that needs to communicate to not only only my other team members, but myself when I come back to this code in the future Writing is a skill and blogging is a way to practice.

This is why I write. Your motive for blogging or not are your own. But I would recommend you occasionally write something.

No Comments

Checks Over $10,000 Require Two Signatures

work safe

When I get a check from SEP, there is a note and a blank line by Traci’s signature. It states that any check for more than $10,000 needs a second person to agree. While I’m sure there are some fraud prevention reasons this is a standard business practice, it serves a better purpose in our Demming based world view. We assume Traci is doing her best and has the truest of intentions,
but it sets a line on the sand for when she should get a collaborator. It is one thing for the IT staff to buy a new workstation for a couple grand. Once we are acquiring Oracle licenses it is nice to have a second person to say, “yes, there is no better way to achieve this goal other than to give Oracle everything in our bank account.”

We are reading Lean Architecture for Sep’s book club. It seems to be a good book because there is some agreement and some disagreement with its contents.

One such disagreement was over separating modules basd on teams. Why make such an artificial distinction? Isn’t it far too heavy to have to hand off a feature from a business layer team to a UI team to create a new feature? Yes. But that’s not what we are talking about. We are talking about distinctions of usage, not necissarily infrastructure. The graphs verses the import logic For example.

It was also mentioned that we have yet to talk about changing the architecture. The pithy answer is, of course, you change it every time you write any new code. The less pithy answer is any time another team is impacted by your code.

Crossing from one module to another is the trigger. Since it will cause changes on the code of another module, I should not make these design choices alone. It is a firewall against the inevitable descent in to the big ball of mud. It is the second signature line on the check.

No Comments

All I Want For Christmas

work safe


Blog BattleSEP is currently having a blog battle between some of the employees. This week’s challenge is the title – All I Want for Christmas.

All I want for Christmas is essential complexity.

As I move from project to project, the interesting part is not the code. I care about software architecture for the same reason Shaker carpenters worried about mortis and tenon joints. They help make the table look and act like a surface for making and eating nourishing meals. That’s it.

Shaker designs are simple and distilled from years of learning and sharing. But they did not ornament their work. Those fancy overlaps on the oval shaker boxes? They were the back. Not meant to be seen. They shunned inlay and other ornamentation.

There are two parts to software that make it hard. There is the actual work we are tying to do, the complexity that comes from health insurance regulations or keeping aircraft serviced. Then there is the stuff we do to ourselves. By being to clever, by trying to do more than we should. This second complexity is incidental to the work at hand.

So here is want I want. I want to have a program that looks like the business rules. It screams, “THIS IS MY UTILITY! THIS IS MY REASON FOR BEING!”

I want to be able to write this code. To craft this software. To make the hard parts of the user interface and data access code disappear unnoticed behind an app that just works, and code that is only on service of simplifying that work as sparingly as possible.

No Comments

Motion != Progress

work safe


Blog BattleSEP is currently having a blog battle between some of the employees. This week’s challenge is the title – Motion != Progress.

When given this topic, a lot of people are going to talk about how to ensure you’re making progress this week. I’m sure of it. At SEP we are deeply culturally committed to Agile and Lean philosophies. We like making a difference and writing software that matters. It doesn’t matter until someone uses it.

How to look busy and not do a damn thing

It is sometimes useful to look at the other side of a topic. So let us explore the fine art of sabotage and see if we can identify patterns of behavior that, even when not intentionally malicious, can have a huge negative impact on a project’s ability to close in on “shipping.” For each of these, there are legitimate reasons, which is what makes them effective. In the real world, don’t accuse people of sabotage.

This is based on a riff I did as a lighting talk because of an old OSS field guide used by the French Resistance.

  • Rules rule
    Make sure everyone follows all the rules all the time. If you work in a regulated industry, apply the most rigorous rules to projects that don’t need it. Add more cross and double checks. Constantly audit the status of those rules and stop all work until everyone is in compliance.
  • Make your voice heard
    If someone manages to finish their action items for a meeting, talk about them in detail. If someone is assigned an action item, that’s just a chance for you to talk some more. As for everything else, try restating it in your own words. If you are invited to a meeting, you should spend 1/2 of it talking.
  • Everyone matters
    Make sure others also speak. Especially if it seems like the meeting has wound down or met consensus. Pass the talking stick! Make them restate the problem in their own words. Start bringing in people from other groups. Reschedule if others aren’t going to make some meetings.
  • Details matter
    Never leave room for interpretation. Why let implemented exercise judgment? More importantly, if you optimize locally and create contradictory rules, it only means more meetings to sort it all out later.
  • Documentation is key
    Instead of a quick memo stating an expected impact, require a formal report with facts. Make them prove that the bad is actually bad. Time spent documenting is time spent. It isn’t good enough to be comprehensive, it must be edited then change the decisions to make sure the document is once again in need of a drastic rewrite.
  • No business like old business
    Since we are constantly changing our minds, we should reopen old decisions to make sure they are still relevant. Even stuff discussed last meeting. Bonus points for changing core functionality that’s been in place for months.
  • Mitigate all risks
    More time spent proving a risk or reducing a risk is more time spent. If it might annoy one user, it must be pursued. One person’s choice is another person’s risk.
  • Respect the hierarchy
    No project exists in a vacuum. Make sure you fit well with other teams and integrate with them. Spend time making sure they mesh well with you. Use their techniques even if they aren’t a fit for your team.
  • Don’t use it until it is done
    I cannot provide feedback until I can judge the whole thing. In addition, once you find a single problem you should stop review until that issue is fixed. You can’t provide useful feedback when you know it is broken. Don’t provide thorough feedback on issues. Call a meeting if they want more detail. Make sure the room is packed.
  • Redesign
    Sweeping architecture changes are a great way to call a bunch of people in to a room, make them write a bunch of code, and risk new issues. Try to avoid actually improving the architecture. If they can develop faster and easier once you’re done, you did it wrong.
  • Ask
    Is there a choice to be made? Make someone else responsible for it. Especially if they are out of the office and will take time to respond. Call a meeting if they are in town.
  • Don’t ask
    Assume. An assumption, especially a seemingly minor one that can go undetected for a while, can ripple out a bunch of issues, meetings, and rework.
  • When in doubt, call a meeting
    If everyone is busy, it can always wait till next week. Let meetings ramble.
  • Spread responsibility
    The perfect model for decision making is the UN Security council. Make 5 people responsible and give total veto power to three of them. It takes a while to build consensus and differing opinions leave all kinds of interpretation open for conflict.
  • Work on several projects at once
    No only does this make it harder for them to schedule you in a meeting, but you now have twice as much work to not be doing!
  • New Priorities
    It takes time for people to ramp up and become familiar with a new project or reorient to an old one. By constantly shifting team members from one project to another, you can keep driving the starved project behind schedule requiring a shift of a number of people off of a team that finally started being effective and on to one requiring ramp up time.

But seriously, don’t

Don’t assume sabotage. I’ve seen all of these in different projects at different times. Fortunately, no one was doing any of these with malicious intent. Usually, choices are made for the people involved on a project from several organizational levels beyond their control. It is very difficult to do your best when those doing the work aren’t in charge of the methodology.

In addition, a lot of these are perfectly valid in the right context. And you have to seek understanding of the whole story to get the real context.

No Comments
« Older Posts
Newer Posts »