Thanks for all this wonderful discussion.

I am very interested the Domain Driven Design approach of Eric Evans, for the main reason that it recognises (to me) the importance of human culture in some projects. The car analogy of Ihe is perfectly valid, but that is an engineering persective and more careful thought up front is capable of making the right specification, that is what structural engineers do all the time and very successfully.

The extra problem with some software is that it comes to be an extension of ourselves, particularly that which is used in everyday work. That is why free and open-source has not taken over completely the desktop environment, the economics of it suggest that it should, that word-processing and spreadsheet software is now a commodity. People get familiar with doing things a particular way and if one person doesn't know how to achieve a goal another person does, this is the view of culture as a shared mental map of the world and how it works. People find ways of working around problems in existing systems, then once those practices are repeated, they become ingrained, effectively at both a personal and organisational level

So I agree with Kurt about systems thinking, but the scope of the system being analysed includes aspects of basic human nature. This is where DDD seems to offer some assistance as it places all the analysis focus on the main feature of humans 'language'. By teasing out the meaning of words used by Domain Experts two things happen; firstly the developer/designer (the difference seems rather meaningless to me) comes to understand the domain intimately and this flows through into the domain model built into the software; secondly the assumptions, and possible inefficiences, that the Domain Experts themselves have made, to get their jobs done, come into sharp focus.

As a result of this second factor there seems to be real opportunities for a redesign of the whole system, software and ways of working (culture), one that is non-threatening to the actual workers as they are part of the team. Having to explain something to someone is a very effective way to analyse the fine detail of a domain.

My ideal is now one where a prototype system is built in real-time, with a business analyst, domain experts and a designer (typing code madly) all working together on it. This seems possible via a Naked Objects framework.

If you contrast this to the creation of a requirements request-for-tender document, in-house people are trying to redesign a system, but largely ignoring this cultural aspect. They are using blind-faith, hoping that what is provided will be accepted as an improvement. If there are problems at the end for any reason, such acceptance, once disappointed is very hard to re-establish. Agile is capable of handling culture as it involves users, but DDD goes further, or is useful in more complex scenarios perhaps.

One aspect of good leaders IMO is that they are capable almost intuitively of handling this human aspect of systems, not many technologists are, that is why they become technologists quite possibly, they gravitate to things that are more predictable, or, that they can tell to do something (figuratively) rather than having to persuade or convine that the old 'tried-and-true' methods can be replaced by something better.

In the analogy of redesigning a car, the thing that would stop such mistakes is market-research. In the case of custom software the 'human aspect' has to be incorporated in the process in some more effective way. This tendency for human nature to result in a 'herd mentality' is very strong in the IT industry itself i believe, so its not strange that it should be such an issue in the design of complex software.

The design of physical products is more amenable to an up-front requirements gathering and RFT type approach, as its more a case of defining the boundaries within which a solution must be provided. From that point the solution providers assume most of the risk I believe, that is reason tenders are so legalistic, to try to prevent loopholes. This does not work with software.

If anyone is interested in this I also have to recommend, not an IT text, but one by a primatologist, called 'The Age of Empathy' by Frans de Waal, where he draws parallels between discoveries from observations of primate society/culture and the human. We have complex language and use of symbols and tools, but underneath this visible layer are more basic universal behaviours/drives, that essentially make it possible to live together, largely via empathy.  I am also about to read 'The Social Life of Information' by Brown and Duguid, which I think will have more specific but similar insights.

So, perhaps good IT project managers have a good ability, perhaps a natural ability, to take these human factors into consideration. When telling people what they should do, in a top down way, is just not going to work.

Steve

On Sun, Dec 1, 2013 at 6:13 AM, Kurt Cagle <kurt.cagle@g...> wrote:

Thomas,

These are all good points (and distinctions). 

One thing that many of these failed projects have in common is that they are SYSTEMS, and for all that we title ourselves system architects the reality is that systems are hard because complex systems are interdependent. Reductionism will take you only so far, and then feedback begins to change the nature of the system, and this holds as true (perhaps even more true) for developing systems as it does for maintaining them.

In so much as the term has become pretty meaningless, any architect is, has to be, a systemic thinker. To the extent that these projects fail, most do not do so due to a single component failure. They fail because they are systemically poorly designed - integration can't happen without a lot impedance, there's a lack of consistency with regard to data models, there are no ways of routing around a failure point. I think a point in favor of agile is that it encourages systemic understanding of the problem domain and how any solution much integrate, but even that's not a panacea when there's a failure of architecture.

In many respects the political aspects of a project should be seen as a side-show. Pointy-haired managers will always be clueless, because for the most part, they lack the technical expertise to understand the systemic nature of the process. The procurement process will always seem byzantine and inherently unfair because money is involved, and everyone will want to water their fields first before letting the river move on to the next farmer. Yet at a technology level, you need to have a person (or people) who both understands the technical challenges and can nonetheless keep a systemic view of the problem in sight. I'm not sure this is leadership - an architect is not in general a leader or even a manager. It's something different, and I don't think we are, collectively, very good at it yet. 

Kurt Cagle
Invited Expert, XForms Working Group, W3C

Managing Editor, XMLToday.org

kurt.cagle@g...

443-837-8725

On Sat, Nov 30, 2013 at 7:11 AM, Thomas Passin <list1@t...> wrote:

On 11/28/2013 1:46 AM, Stephen Cameron wrote:

My theory is that the idea of issuing tenders, which is based on
defining requirements up front, is just plain wrong for complex software
systems, that are essentially part of the culture of an organisation.

The problem of writing requirements up front is that one often can't know all the important aspects of the problem until one gets well into trying to solve it.  Same with quoting a project - early on, you can't quote with precision unless the job is very similar to others you have done.  But a contractor has to know what it's bidding on in good detail, so you can't just leave lots of details until later.

The only solution is to get people with the expertise to do ALL the
work, including writing a specification if necessary, and to take ALL
the risk.

And right here is a good example of the confusion many people and agencies have, one that often causes no end of difficulties.  There are requirements, and there are specifications. And they are not the same. Requirements are what the end user/agency needs to accomplish, specifications are testable statements of behavior that, if passed, help to show that the system will accomplish the requirements.

They are not the same.  The builder can only design and build a system to meet a set of specifications - because they are what can be tested. The requirements - needs of the customer - ideally will mesh with the specifications (or the other way around).

Let's say, for example, that you are the buyer of a car to be developed, and one of your requirements is that the car has to be able to transport a family of four at least 400 miles between refueling stops.  Fine.  The vendor will have to say "well, a family of four, we will take that to mean two adults weighing an average of 160 lb apiece, and two teenage children, height 5 ft 6 inches, weight 130 lb (sorry, metric users!). "transport" we will take to mean drive at an average speed of 55.

Good.  Off we go to design and prototype.  We estimate the weight of the vehicle, then we can estimate the amount of fuel, and by iteration we converge on a workable design.

But now the customer comes back and says that the adult weights must be 170 lb, the teens will need more room, and oh, yes, the customer forgot about baggage.

OK, fine, maybe the changes can be accommodated, maybe they can't.  But the vendor can't now accept all the risk for failure.  The vendor quoted the job, but now the basis for the original quoted design has been changed.  Maybe the changes make for an overall system that's no longer feasible.  Maybe a complete redesign is called for.  Now there are lots of risks, and the vendor can't know how much to charge to account for them.

Though this is a very simplified example, it's realistic in the sense that this kind of change goes on all the time, and ordinary contracting practices just aren't up to handling it.  "The" government (whichever one you want to talk about) can be pretty bad about it, but not just government buyers.

_______________________________________________________________________

XML-DEV is a publicly archived, unmoderated list hosted by OASIS
to support XML implementation and development. To minimize
spam in the archives, you must subscribe before posting.

[Un]Subscribe/change address: http://www.oasis-open.org/mlmanage/
Or unsubscribe: xml-dev-unsubscribe@l....org
subscribe: xml-dev-subscribe@l....org
List archive: http://lists.xml.org/archives/xml-dev/
List Guidelines: http://www.oasis-open.org/maillists/guidelines.php