Software engineering

Software engineering is the technologies and practices that people use to create and maintain computer software, while emphasizing productivity and quality. In the year 2000, these technologies and practices encompass languages, databases, tools, platforms, libraries, standards, patterns, and processes.

Software engineering applications include email, embedded software, graphical user interfaces, office suites, operating systems, optimizing compilers, relational databases, robotics controllers, video games, and the world wide web. Other important applications include accounting, airline reservations, avionics, banking, and telephony. These applications embody social and economic value, in that they make people more productive, improve their quality of life, and enable them to do things that would otherwise be impossible.

Software engineers are the community of practitioners who create programs. In the year 2000, there were about 640,000 software engineers in the U.S. and another (1,200,000?) in Europe, Asia, and elsewhere. Steve McConnell says 40% have computer science degrees. Many others have science, engineering, business or other, non-technical degrees. They use diverse technologies and practices and work on diverse applications.

Considerable debate remains over what software engineering really is. The differences between software engineering and related fields of programming, computer science, and traditional engineering are murky at best. In recent years, software engineering has emerged from a subfield of computer science to stand on its own. Many of the similarities and differences are discussed below. The dispute with SE is on what these technologies, practices, and applications actually are and who precisely belongs to the community.

Origin of the term

The term software engineering originated in the early 1970s as researchers (often at the encouragement of management) attempted to apply rigorous mathematical analysis to computer programming. The perception at the time was that conventional engineering was carried out with a great degree of mathematical rigor, while computer programming was commonly seen as an iterative trial and error process. It's likely that neither of these perceptions were especially accurate, but the term software engineering was adopted to indicate an intent to make programming more rigorous. In several companies the title "programmer" was changed to "software engineer", at least for certain categories of programmers.

Importance of software engineering

In the USA, software drove 1/4 of all increase in GDP during the 1990s (about $90 billion per year), and 1/6 of all productivity growth (efficiency within GDP) during the late 1990s (about $33 billion per year).

Software engineering has changed the world's culture, particularly in countries whose people use computers. Email, the world-wide web, and instant messaging enable people to interact in new ways. Software lowers the cost of many important social goods and services, such as health-care and fire departments.

Current directions for software engineering

Agile processes are an important emerging practice. Agile processes are methods to manage software development so projects evolve with changing expectations. The older document-driven processes (like CMM and ISO 9000) may be fading in importance. Some persons believe that companies have exported many of the jobs that can be controlled by these processes. Related concepts are Lean software development and Extreme programming.

Aspect programming is also an important emerging programming technology. Aspects help programmers deal with ilities by providing tools to add or remove boilerplate code from many areas in a software project. Aspects describe how all objects or functions should behave in a particular circumstance. For example, aspects can add debugging, logging, or locking control into all objects of a particular type. Related concepts are Generative programming and Templates.

The future of software engineering was an important conference at the ICSE 2000, [1]. The FOSE project summarized the state of the art in 2000 and listed many problems to be solved over the next decade. The Feyerabend project is attempting to discover the future, [2].

Software engineering versus programming

Software engineers work on all sizes of applications: small and large.

Programming emphasizes coding to produce working software applications, independent of budget and schedule.

Software engineering tries to encompass software projects more completely, including budget and schedule. Software engineering recognizes that software development fits in a large business context with relationships to marketing, sales, production, installation, training, support, and operations. Software engineering emphasizes methods to construct large applications that individual programmers cannot write alone. Software engineering tries to come up with methods to write programs in a consistent way.

Software engineering versus computer science

Whether software development is more like science or engineering has been fiercely debated for many decades. Software development shares attributes of both and many software projects have elements of both, but distinctions exist.

Many believe that (metaphorically) software engineering is to computer science and information science as traditional engineering is to physics and chemistry.

While 40% of software engineers earn computer science degrees, they do not practice computer science every day, which is different than practicing software engineering every day.

Differences

Issue	Software Engineering	Computer Science
Ideal	Constructing software system for real-world use	Correct eternal truths about computability and algorithms with good runtime behavior
Goals	Working programs (like office suites and compilers)	Algorithms (like Shell sort) and abstract problems (like Travelling Salesman)
Budgets and Schedules	Projects (like the next upgrade) have fixed budgets and schedules.	Projects (like solving NP) are independent of budget and schedule.
Emphasis	Software engineering emphasizes applying skills and working programs that deliver value to users.	The theory of software emphasizes eternal truths, like the running time analysis, space analysis, and correctness of algorithms.
Change	Programs will evolve as user needs and expectations evolve, and as technologies and practices evolve.	When computer science problems are solved, the solution will never change.
Additional Skills	Domain knowledge	Mathematics
Notable Educators and Researchers	Barry Boehm, David Parnas, and Frederick P. Brooks	Edsger Dijkstra, Donald Knuth, Robert Tarjan, and Alan Turing
Notable Practitioners	John Backus, Dan Bricklin, Tim Berners-Lee, Linus Torvalds, Richard Stallman	Not applicable
Number of Practitioners in U.S.	640,000	25,000
Number of Practitioners in World	unknown	unknown

Software engineering versus traditional engineering

Some practitioners believe that they apply concepts of traditional engineering to software design and implementation. They believe this provides a structured, logical approach and subsequently, a stable final product. Other practitioners are inspired by traditional engineering, but believe that software problems need particular solutions. They believe that traditional engineering concepts may not apply, because software is fundamentally different than bridges and roads. For example, traditional engineers do not use compilers or linkers to build roads.

Software engineers aspire to build low-cost, reliable, safe software, which is much like what traditional engineers do.

Software engineers borrow many metaphors and techniques from traditional engineering disciplines: requirements analysis, quality control, and project management techniques.

Traditional engineers now use software tools to design and analyze systems, such as bridges and buildings. These new kinds of design and analysis resemble programming in many respects, because the work exists as electronic documents and goes through analysis, design, implementation, and testing phases, just like software.

Differences

Issue	Software Engineering	Traditional Engineering
Foundations	Software engineering is based on computer science, information science, and discrete math.	Traditional engineering is based on physics, chemistry, and calculus.
Cost	Compilers and computers are now cheap, so software engineering and consulting often cost more than 50% of a project. Even minor software engineering cost-overruns can cause a project to fail.	Construction and manufacturing costs are high, so traditional engineering may only cost 15% of a project. Even major engineering cost overruns may not affect a project's viability.
Management Status	Few software engineers manage anyone, so they are not viewed as managers, except by themselves.	Many traditional engineers manage construction, manufacturing, or maintenance crews, so they are all treated as managers.
Innovation	Software engineers apply new and untested elements in many software projects.	Though some projects have innovations, traditional engineers apply known and tested principles, and limit the untested innovations that goes into each product.
Replication	Replication is trivial, and most development effort goes into building new (unproven) or changing old designs and features.	Most development effort goes into replicating proven designs.
Number of Practioners in U.S. in 2000	640,000	1,100,000 total engineers 65,000 computer engineers

In the U.S., there are 10 times as many software engineers as computer engineers, and the software engineering community is about 60% as large as the traditional engineering community.

Software engineering is much younger than many traditional engineering disciplines such as civil engineering.

Some claim that software engineering is already as predicatable and reliable as many fields of engineering, such as space or biological engineering. Although large, reliable software systems can be and have been constructed, software projects that fail during construction or in service are still too common. However, failures in large traditional engineering systems, such as those preceding the disasters in Three Mile Island, Chernobyl, Bhopal Disaster, Space Shuttles Challenger and Columbia are also too common. Others argue that unlike in traditional engineering where practicioners analyze failures, find precise causes, and set up guidelines to avoid them in the future software engineers routinely fail to pinnpoint causes of failure or delay precisely enough to avoid repeats in the future.

The software crisis

Software engineering arose out of the so called software crisis of the 1960s, 1970s, and 1980s, when many software projects had bad endings. Many software projects ran over budget and schedule. Some projects caused property damage. A few projects caused loss of life. As software becomes more pervasive, we all recognize the need for better software. The software crisis was originally defined in terms of productivity, but evolved to emphasize quality.

The OS 360 operating system was a classic example. It is still used on the IBM 360 Series and its descendants. This decade-long project eventually produced a working system amongst the most complex software systems ever designed. OS 360 was one of the first very large (1000 programmer) software projects. Fred Brooks admits in Mythical Man Month that he made a multi-million dollar mistake when he managed the project.

Property Damage: Software defects can cause property damage. Poor software security allows hackers to steal identities, costing time, money and reputations. The explosion of a European Ariane rocket and other disasters spurred further developments in the field.

Life and Death: Defects in software have killed. Several embedded systems used in radiotherapy machines failed so catastrophically that they administered lethal doses of radiation to patients.

A current list of software problems and disasters is kept at Peter G. Neumann's Computer Risks column.

According to a study by the Standish Group, in 2000, only 28 percent of software projects could be classed as complete successes (meaning they were executed on time and on budget), while 23 percent failed outright (meaning that they were abandoned).

Silver bullets and no silver bullets

For decades, solving the software crisis was paramount. Seemingly, every new technology and practice from the 1970s to the 1990s was trumpeted as a silver bullet to solve the software crisis.

Technologies: Every new technology and practice for decades was touted as the solution to the software crisis: structured programming, object-oriented programming, process, CMM, UML, Ada, methodologies, and so on.

Practices: Some pundits argued that the software crisis was due to the lack of discipline of programmers. Many practitioners resist process strongly. Some believed that if formal engineering methodologies could be applied to software development, the production of software would become as predictable an industry as other branches of engineering. This led to work on a code of ethics, and professionalism. The issue is so pervasive that it cannot be deliberate. It must be a natural response to experience.

In 1987, Brooks published the famous paper "No Silver Bullet", arguing that no individual technology or practice can make a 10 fold improvement in productivity in 10 years. All of the technologies and practices have made incremental improvements. See his article

Historical notes

Women's Issues: In the 1940s, 1950s, and 1960s, software was a women's ghetto. Men preferred the higher prestige of hardware engineering roles. So, women filled the programming roles. Women like Grace Hopper were common.

Many unsung women wrote code prior to 1968. Today, few women work in software engineering. Women have largely moved into analysis and testing roles. Saying that this is sexual discrimination is too simple, because it related directly to individual identity. In this sense, software engineering is the masculinization of programming.

NATO: The NATO Science Committee sponsored two conferences on software engineering in 1968 (Garmisch, Germany) and 1969, which gave the field its initial boost. Many consider these conferences to be the start of the field. However, there has been a continuity of practice from the 1940s to the 2000s.

Cost: The cost of software versus hardware has changed substantially. When mainframes were expensive, software projects could be expensive. Because powerful PCs are cheap, software costs must become cheaper, in comparison.

Emerging from Programming: Prior to the mid-1990s, most software practitioners called themselves programmers or developers, regardless of their actual jobs. The term programmer had often been used as a pejorative term to refer to those who lacked the skills, education, or ethics to write quality software. Software practitioners began to describe themselves as software engineers to escape the stigma attached to "programmer". This caused a lot of confusion, because some saw no difference while others were trying to create a difference.

Criticisms of software engineering

Critics charge that some assumptions made during the process of implementing software engineering are inherently flawed. However, these criticisms may seem to apply to every human activity (Eg : fine arts or even quantum cryptography). Hence, a few believe that these criticisms do not hold much water. The following paragraphs detail many of the criticisms and some responses to them.

• Managing Expectations:
  • Criticism: - Managing the customer's expectations to something that can be built and delivered is the key to successful "software engineering" methodologies. Thus, the field resembles marketing, sociology, or voodoo, more than traditional engineering with its responsibilities to society at large and the perils of legal liability with failure to protect the public interest.
  • Response: Every profession manages expectations, including all forms of engineering. This may happen at different levels. Software Engineering focusses on the immediate requirements, whereas other engineering fields tend to solve problems that are typically long-ranged. Moreover, "responsibility to the society" means meeting the expectations of the general public, which can be considered a customer.

• Poor Requirements:
  • Criticism: - The requirements for most projects are incomplete or inconsistent. Some customers have little previous experience in writing requirements for a software project. Other customers really do not know what they want, and say "I'll know it when I see it" (IKIWISI). Even experienced customers who know exactly what they want may be unable to articulate their requirements. It has often been observed that the users almost always expect far more than they write in specifications. And, complete requirements may describe programs that have no computable or practical solutions. In contrast to this, other engineering sub-fields like Embedded systems are unique in that they are designed by other engineers, who can sometimes define interfaces completely.
  • Response:- One response to this objection is that one must avoid all unclear projects (but there might be no others left). Another response is agile development and rapid prototyping to clarify project goals as soon as possible.

• Rising Complexity:
  • Criticism: - Critics argue that the probability of failure increases with the size, scope and complexity of the project. Technologies and practices have improved through the 1970s, 1980s, and 1990s, but the complexity in requirements and user expectations have also increase fast enough to overwhelm the improvement in practices. So, the gap between what is expected and what is delivered has not improved.
  • Response: This is not the fault of practitioners. This is actually a measure of the success of practitioners, because demand normally follows the supply. So customers demanding more is an obvious sign of their belief that what they demand will be supplied (even if they peg the demand slightly above the measure of supply in order to enforce agility.)

• Ongoing Change:
  • Criticism: - Practitioners are eager to develop new technologies and practices and try new tools in every project. Some view this ongoing change as proof that older technologies and practices were failures.
  • Response: - One can view this ongoing change as proof that the field successfully learns and grows.

• Ongoing Failure:
  • Criticism: - Critics charge that incomplete or poorly designed systems are still too common. The early lessons of the field were not sufficient to prevent subsequent disasters.
  • Response: - No field that strives to do bigger and better projects has ever avoided all unitentional failures. Some identified compromises cannot be precluded too. Moreover, other engineering fields frequently resort to a balance of factors to achieve optimization, and hence this is not an isolated problem.

• Nothing New:
  • Criticism: - Critics argue that software engineering has not created anything on its own, it merely uses what computer scientists already know.
  • Response: - Practitioners developed many tools and practices (compilers, make, cvs, XP), on their own, out of simple need. These tools make use of the tenets of computer science, just like the other engineering fields that make use of pure sciences like maths and physics.

• Anyone Can Do It:
  • Criticism: - Many bright people from other fields (engineers, scientists, business people) write spreadsheet templates or calculator programs, and eventually switch to writing large applications, and believe that they do software engineering.
  • Response: - Software engineering is a skill that is refined through long practice. Software engineers are the ones who already have the education and experience, and they keep up with evolving technologies and practices. This is true of every skill.

• We Do Not Know What It Is:
  • Criticism: - One cannot categories software engineering in the usually accepted definition of engineering. It doesn't yield to the standard ways of categorization.
  • Response: - Software engineering is grounded in technologies and practices, applications, and the community of Software Engineering practitioners, who have quite successfully tackled this problem by exactly defining the term. It is time, the definition is accepted by the rest.

• Software as Executable Knowledge:
  • Criticism: - In Zepplins and Jet Planes [3], Philip Armour argues that software is executable knowledge, meaning that the development of this knowledge is a major part of software development. Knowledge is discovered in an creative process where trial and error, learning, and the ability to challenge one's assumptions are important. In this view, the potential benefit of software engineering is limited to making it easier for developers to trial ideas, to discover errors earlier, and to provide them with information about the state of a software system.

Methodologies

Software engineering methods span many disciplines, including project management, analysis, specification, design, coding, testing, and quality assurance. All of the methods guiding this field are collations of all of these fields.

Software design methods can be informally classified into thick and thin. Thick methods include a large amount of formal process paperwork and documentation.

The well-known thick methods include Cleanroom, ISO 9000, CMM and Rational Unified Process (RUP).

Thin methods eschew formal process paperwork and documentation. Well-known thin methodologies include Extreme Programming (XP) and Agile Processes.

Recently, some (like Karl Weigers) have argued for no more methodologies. Methodologies tend to list the contemporary technologies and practices and insist that everyone use them. This advice is obvious for those who have the opportunity to use contemporary technologies and practices, and useless for those who maintain legacy systems and must use legacy tools cannot do so, due to circumstance. So, methodologies are not specifically useful. And methodologies must be updated as technologies and practices evolve.

Processes and meta-processes

There is a growing body of software development organisations that are implementing process methodoligies. Many of them are in the defence industry, which in the U.S. requires a 'Rating' baised on 'Process models' to obtain contracts.

The Capability Maturity Model (CMM) grades organizations on how well they create software accrording to how they define and execute their processes. ISO 9000 describes standards for formally organizing processes with documentation.

ISO 15504 or SPICE, The software process life cycle is also gaining wide usage. This standard is aimed at setting out a clear model for process comparison. SPICE is used much like CMM and CMMI. It models processes to manage, control, guide and monitor software development. This model is then used to measure what a development organization or project team actually does during software development. This information is analyzed to identify weaknesses and drive improvement. It also identifies strengths that can be continued or integrated into common practice for that organization or team.

Six Sigma is a methodology that uses data and statistical analysis to measure and improve a company's operational performance. It works by identifying and eliminating "defects" in manufacturing and service-related processes. The maximum permissible defects are 3.4 per million opportunities.

Extreme programming, Agile processes, and Lean software development are full blown processes that take an incremental or evolutionary approach to software development.

Differences of opinion

Different methodologies advocate conflicting solutions. Proponents of different methodologies often get into heated debates over their merits.

Everyone agrees that we want better software. We disagree on priorities and approach, on what an individual should do first in a specific circumstance.

With an industry of 640,000 software engineers and 530,000 programmers in the USA, there should be room for many approaches. Subfields like consumer applications are very sensitive to time and cost. Subfields like military and medical applications are very sensitive to quality. All subfields mix these needs to varying degrees. There should be room for people to try different approaches.

Professional software engineering

In the mid-1990s, The National Society of Professional Engineers sued in all states to prohibit anyone from using the term software engineer as a noun or field of employment. They won in most states. Utah does not license professional engineers, so the point was moot there.

In response, the IEEE and ACM began a joint effort called JCESEP in 1993, which evolved into SWECC in 1998 to explore making software engineering into a profession. The professional movement uses the traditional engineering model. The ACM pulled out of SWECC in [1998?] objecting to its support for the Texas professionalization efforts. The IEEE continued to support making software engineering a branch of traditional engineering.

Donald Bagart of Texas became the first professional software engineer in the U.S. on September 4, 1998 or October 9, 1998. As of May 2002, Texas had issued 44 professional enginering licenses for software engineers.

The professional movement has been criticized for many reasons.

• Licensed software engineers must learn years of physics and chemistry to pass the exams, which is irrelevant to most software practitioners.
• Many (most?) computer science majors don't earn degrees in engineering schools. These engineers are unqualified to pass the exams.
• In Canada, most people who become professional software engineers actually studied computer or electrical engineering, even though these people already qualified to become professional engineers in their own fields. Thus, professionalism turned into another tool that traditional engineers use to prove that they are better than computer scientists and software engineers.

For more information, see:

• http://www.faqs.org/faqs/engineering/pe-eit-exam/ Apr 10, 2002
• http://www.sdmagazine.com/documents/s=746/sdm0003g/0003g.htm

The fields of data engineering, knowledge engineering, user interface engineering, and so on have similar concerns over engineering issues.

Ethics

Software engineering ethics is a large field. In some ways it began as an attempt to define bugs as unethical. However that is unrealistic.

• http://www.acm.org/serving/se/code.htm

Bill Joy argued that "better software" can only enable its privileged end users, make reality more power-pointy as opposed to more humane, and ultimately run away with itself so that "the future doesn't need us." He openly questioned the goals of software engineering in this respect, asking why it isn't trying to be more ethical rather than more efficient.

Lawrence Lessig argued that coding is more like law, in that it expresses a certain social ethic by deciding what to ignore/consider in making detailed decisions.

See also

• List of software engineering topics

It is also about picking up heaps of sexy birds!